Deserialisers

Consumed data is mapped to internal StreamEvents using a flexible deserialisation framework

Overview

Joule's deserialization framework enables seamless conversion of raw domain-specific data into StreamEvent objects for processing.

By leveraging built-in deserializers (i.e., JSON, CSV, AVRO) or custom parsers, it ensures compatibility with various data formats and workflows, allowing precise transformation tailored to application needs.

For advanced scenarios, the StreamEventParser API supports custom parsing when standard deserializers cannot handle unique formats or structures. This flexibility allows developers to map domain-specific objects into StreamEvent collections, ensuring smooth integration with Joule's processing pipeline.

Every connector type will have it preferred supported deserialisation method

This page serves as a guide for configuring data deserialisation, with examples and options for effective data integration in Joule.

This page includes the following:

1. Deserialiser DSL element Describes the deserialisation element, which defines how received data is parsed ready for Joule processing.

2. Kafka example Provides an example Kafka consumer configuration that deserialises incoming domain events to StreamEvents.

3. Custom parser API Covers the StreamEventParser interface for custom deserialisation when unique data parsing is required.

4. AVRO deserialiser Introduce how AVRO can be used to transform domain events to Joule StreamEvent events using a provided AVRO IDL schema file.

5. Available Implementations Learn how to use OOTB deserialisers for JSON and CSV event data.

Deserialiser DSL element

The deserialiser DSL element defines the deserialisation process for StreamEvents to prepare them for Joule processing.

It allows developers to specify how data should be transformed into StreamEvents using custom parsing logic, enabling precise handling of various data formats and structures tailored to specific application needs.

For a custom example on deserialisers

Example

This code snippet shows a basic setup for a Kafka consumer within Joule, where JSON StreamEvent string objects are converted toStreamEvent objects ready for pipeline processing.

kafkaConsumer:
   ...
   deserializer:
      parser: 
         json deserializer: {} 

Attributes schema

Attribute	Description	Type
parser	Parser interface to convert a Object to a collection of stream events	StreamEventParser Default: StreamEventJsonDeserializer
compressed	If passed payload is compressed	Boolean Default: false
batch	Is payload a batch as events	Boolean Default: false
properties	Specific properties required for custom deserialisation process	Properties

Parser

Parser interface to converts a received Object to a collection of stream events.

StreamEventParser API

For complex use cases whereby AVRO cannot be used a custom parser will be needed. In this case developers can provide a parsing solution using the StreamEventParser API.

Example

This example shows how a custom parser (QuoteToStreamEventParser) transforms Quote domain objects into StreamEvents that can be processed by Joule. Deserialisation in action.

kafkaConsumer:
  ... 
  deserializer:
    parser: 
      com.fractalworks.examples.banking.data.QuoteToStreamEventParser
    
    key deserializer: 
      org.apache.kafka.common.serialization.IntegerDeserializer
    
    value deserializer: 
      com.fractalworks.streams.transport.kafka.serializers.object.ObjectDeserializer

AVRO deserialisation

This deserialiser has extra setting to support ability to load the target schema. Currently only local schema files are supported with schema registry support on request.

The transformer automatically maps domain attributes to StreamEvent attributes using a provided AVRO schema IDL. Currently only local schema files are supported with schema registry support on request.

Integrate Joule to any existing system using already established data structures

Example

kafkaConsumer:
   ...
   deserializer:
      parser: 
         avro deserializer: 
            schema: /home/myapp/schema/customer.avro

Attributes schema

Attribute	Description	Type	Required
schema	Path and name of schema file	String
field mapping	Map source fields to map to a target StreamEvent field	Map<String,String>

Available Implementations

Joule provides several deserialiser implementations for different data formats, each requiring a full package namespace.

Key implementations include:

1. JSON deserialiser Parse StreamEvent JSON events into concrete objects.

2. CSV Deserialiser Parse CSV formatted data using options for custom headers and a type map to define data types like STRING and DOUBLE.

3. Object Deserialiser Parse StreamEvent binary events into concrete objects.

Format	DSL	Class
json	json deserialiser	StreamEventJsonDeserializer
csv	csv deserialiser	StreamEventCSVDeserializer
object	object deserialiser	StreamEventObjectDeserializer

JSON

This parser reads a converts a JSON string formatted StreamEvent to a StreamEvent object.

Attribute	Description	Type	Required
date format	Provide a custom date format. See JavaDocs for further data format options.	String Default: yyyy/MM/dd

Example

deserializer:
    parser: 
        json deserializer:
            date format: dd/MM/yyyy

CSV

This parser has extra optional setting to support ability to custom parsing through the use of defining a custom header.

Attribute	Description	Type	Required
type map	Map of fields to types. See supported types	Map<String,Type>
delimiter	Define a custom file delimiter	String Default: ","
date format	Provide a custom date format.	String Default: yyyy/MM/dd

Example

deserializer:
    parser: 
        csv deserializer:
            type map:
               symbol: STRING
               rate:   DOUBLE

Supported Types

Following types are supported

• DOUBLE, FLOAT, LONG, INTEGER, SHORT, BYTE, BOOLEAN, STRING, DATE

Object

This parser reads a converts a binary formatted StreamEvent to a StreamEvent object

Example

deserializer:
    parser: 
        object deserializer: {}