Kafka
Kafka consumer transport ingests data from subscribed cluster topics. Events consumed are deserialised in to StreamEvents ready for pipeline processing either automatically or through a custom transformer implementation.
Configuration Guide
Add to the /etc/hosts file the address of the Kafka host
i.e. 127.0.0.1 KAFKA_BROKER
Example configuration
This basic example leverages the default setting resulting in events consumed from the quotes topic as StreamEvent Json object. This assumes the upstream data producer publishes the Joule StreamEvent object as a Json object.
Core Attributes
Available configuration parameters
Attribute | Description | Data Type | Required |
---|---|---|---|
name | Name of source stream | String | |
cluster address | Kafka cluster details. Maps to bootstrap.servers | http://<ip-address>:port | |
consumerGroupId | Consumer group to ensure an event is read only once. Maps to group.id | String | |
topics | Message topics to subscribe too | List of strings | |
consumerThreads | Number of consumer threads to use for event parsing | Integer Default : 1 | |
pollingTimeout | The maximum time to block | Long Default: 100ms | |
deserializer | Deserialization configuration | See Deserialization Attributes section | |
properties | Additional publisher properties to be applied to the Kafka publisher subsystem | Properties map |
Deserialization Attributes
This topic provides configuration parameters available object deserialization process.
Attribute | Description | Data Type | Required |
---|---|---|---|
transform | User provided implementation | Implementation of CustomTransformer | |
key deserializer | Domain class that maps to the partition key type. Property maps to key.serializer property | String Default: IntegerDeserializer | |
value deserializer | Domain class that deserializes incoming object to StreamEvent. Property maps to value.serializer property | String Default: StringDeserializer |
Deserializer example
OOTB deserializers
Custom Transformers
By implementing a CustomTransformer you can provide domain specific data types to ease integration from upstream applications.
Example
Java transformer implementation
This class will convert the passed object to an internal StreamEvent object.
Client library
Last updated