Kafka
Sends events to specified Kafka topics, enabling real-time data streams
Overview
The Kafka Publisher Transport enables publishing of processed events to a specified Kafka topic on a defined Kafka cluster. Events are serialised using either a default JSON serialiser (StreamEventJsonSerializer
) or a custom-defined transformer.
Configuration includes setting the Kafka cluster address in the /etc/hosts
file and defining topics and partition keys for message distribution. Key configuration attributes are prioritised by importance and cover serialisation options, allowing flexible data handling.
Out-of-the-box serialisers support common formats (JSON, Avro), while custom serialisers and transformers facilitate integration with downstream applications by converting events into domain-specific data types.
The Kafka Publisher setup supports scalable, resilient event processing workflows, suitable for chaining multiple data-processing stages.
The Kafka consumers relies on the client library org.apache.kafka:kafka-clients:2.7.0
Examples & DSL attributes
In this example, the Kafka Publisher sends events to the customers
topic on the Kafka broker (KAFKA_BROKER:9092
).
Events are partitioned by customerId
, ensuring that all events for the same customer go to the same partition for ordered processing.
Using default settings, events are serialized as JSON StreamEvent
objects, making them easily readable for downstream processes. This setup enables chaining multiple processing stages, useful for building scalable and resilient data workflows.
Add to the /etc/hosts
file the address of the Kafka host i.e. 127.0.0.1 KAFKA_BROKER
Attributes schema
Configuration parameters available for the InfluxDB publisher transport.
The parameters are organised by order of importance, ranked from high to low.
Serialisation attributes schema
Serialiser example
This example configures the Kafka Publisher to send events to the customers
topic on a local broker (localhost:9092
). Events are partitioned by customerId
, ensuring ordered processing per customer.
key serializer
IntegerSerializer
forcustomerId
.value serializer
StreamEventJsonSerializer
for JSON-formatted event data.
This setup organises events by "customer ID" and makes them easily readable for downstream processing.
OOTB serialisers
A flexible event serialisation model allows Joule processes to be chained together for complex use cases and easy downstream integration.
Joule includes built-in serialisers, compatible with Apache Kafka, and also supports custom serialisation through the Joule SDK.
Custom transformers
By implementing a custom transformer, you can create domain-specific data types to simplify integration with downstream applications.
Example
In this example, the Kafka Publisher sends events to the customers
topic on a local Kafka broker (localhost:9092
).
Events are partitioned by customerId
and serialised using custom serialisers:
key serializer
IntegerSerializer
for thecustomerId
(integer).value serializer
ObjectSerializer
for the event data.transformer The
CustomerTransformer
is used to convert event data into a domain-specificCustomer
object before serialisation.
This setup enables custom data transformations and ensures events are processed and stored efficiently.
Java transformer implementation
This class will convert an internal StreamEvent
object to a user defined Customer
data type.
Default producer properties
Last updated