# Deserialisers

## Overview

Joule's deserialization framework enables seamless conversion of raw domain-specific data into `StreamEvent` objects for processing.

By leveraging built-in deserializers (i.e., JSON, CSV, AVRO) or custom parsers, it ensures compatibility with various data formats and workflows, allowing precise transformation tailored to application needs.

For advanced scenarios, the [StreamEventParser API](#streameventparser-api) supports custom parsing when standard deserializers cannot handle unique formats or structures. This flexibility allows developers to map domain-specific objects into `StreamEvent` collections, ensuring smooth integration with Joule's processing pipeline.

{% hint style="info" %}
Every connector type will have it preferred supported deserialisation method
{% endhint %}

This page serves as a guide for configuring data deserialisation, with examples and options for effective data integration in Joule.

This page includes the following:

1. <mark style="color:green;">**Deserialiser DSL element**</mark>\
   Describes the deserialisation element, which defines how received data is parsed ready for Joule processing.
2. <mark style="color:green;">**Kafka example**</mark>\
   Provides an example Kafka consumer configuration that deserialises incoming domain events to `StreamEvents`.
3. <mark style="color:green;">**Custom parser API**</mark>\
   Covers the `StreamEventParser` interface for custom deserialisation when unique data parsing is required.
4. <mark style="color:green;">**AVRO deserialiser**</mark>\
   Introduce how [AVRO](https://avro.apache.org/) can be used to transform domain events to Joule `StreamEvent` events using a provided AVRO IDL schema file.
5. <mark style="color:green;">**Available Implementations**</mark>\
   Learn how to use OOTB deserialisers for JSON and CSV event data.

## Deserialiser DSL element

The deserialiser DSL element defines the deserialisation process for `StreamEvent`s to prepare them for Joule processing.

It allows developers to specify how data should be transformed into `StreamEvent`s using custom parsing logic, enabling precise handling of various data formats and structures tailored to specific application needs.

{% hint style="info" %}
For a [custom example on deserialisers](https://docs.fractalworks.io/joule/components/connectors/serialisers/deserialisers/custom-parsing-example)
{% endhint %}

### Example

This code snippet shows a basic setup for a Kafka consumer within Joule, where JSON `StreamEvent` string objects are converted to`StreamEvent` objects ready for pipeline processing.

```yaml
kafkaConsumer:
   ...
   deserializer:
      parser: 
         json deserializer: {} 
```

### Attributes schema

<table><thead><tr><th width="155.33333333333331">Attribute</th><th width="334">Description</th><th>Type</th></tr></thead><tbody><tr><td>parser</td><td>Parser interface to convert a Object to a collection of stream events</td><td><p><code>StreamEventParser</code></p><p>Default: <code>StreamEventJsonDeserializer</code></p></td></tr><tr><td>compressed</td><td>If passed payload is compressed</td><td><p>Boolean</p><p>Default: false</p></td></tr><tr><td>batch</td><td>Is payload a batch as events</td><td><p>Boolean</p><p>Default: false</p></td></tr><tr><td>properties</td><td>Specific properties required for custom deserialisation process</td><td>Properties</td></tr></tbody></table>

## Parser

Parser interface to converts a received Object to a collection of stream events.&#x20;

### StreamEventParser API

For complex use cases whereby AVRO cannot be used a custom parser will be needed. In this case developers can provide a parsing solution using the `StreamEventParser` API.

#### Example

This example shows how a custom parser (`QuoteToStreamEventParser`) transforms `Quote` domain objects into `StreamEvents` that can be processed by Joule. Deserialisation in action.

```yaml
kafkaConsumer:
  ... 
  deserializer:
    parser: 
      com.fractalworks.examples.banking.data.QuoteToStreamEventParser
    
    key deserializer: 
      org.apache.kafka.common.serialization.IntegerDeserializer
    
    value deserializer: 
      com.fractalworks.streams.transport.kafka.serializers.object.ObjectDeserializer
```

### AVRO deserialisation

This deserialiser has extra setting to **support ability to load the target schema**. Currently only local schema files are supported with schema registry support on request.

The transformer automatically maps domain attributes to `StreamEvent` attributes using a provided AVRO schema IDL. Currently only local schema files are supported with schema registry support on request.

{% hint style="success" %}
Integrate Joule to any existing system using already established data structures
{% endhint %}

#### Example

```yaml
kafkaConsumer:
   ...
   deserializer:
      parser: 
         avro deserializer: 
            schema: /home/myapp/schema/customer.avro
```

#### Attributes schema

<table><thead><tr><th width="151.33333333333331">Attribute</th><th width="298">Description</th><th width="197">Type</th><th data-type="checkbox">Required</th></tr></thead><tbody><tr><td>schema</td><td>Path and name of schema file</td><td>String</td><td>true</td></tr><tr><td>field mapping</td><td>Map source fields to map to a target StreamEvent field</td><td>Map&#x3C;String,String></td><td>false</td></tr></tbody></table>

## Available Implementations

Joule provides several deserialiser implementations for different data formats, each requiring a full package namespace.

Key implementations include:

1. <mark style="color:green;">**JSON deserialiser**</mark>\
   Parse StreamEvent JSON events into concrete objects.
2. <mark style="color:green;">**CSV Deserialiser**</mark>\
   Parse CSV formatted data using options for custom headers and a type map to define data types like `STRING` and `DOUBLE`.
3. <mark style="color:green;">**Object Deserialiser**</mark>\
   Parse StreamEvent binary events into concrete objects.

<table><thead><tr><th width="151.33333333333331">Format</th><th>DSL</th><th>Class</th></tr></thead><tbody><tr><td>json</td><td>json deserialiser</td><td>StreamEventJsonDeserializer</td></tr><tr><td>csv</td><td>csv deserialiser</td><td>StreamEventCSVDeserializer</td></tr><tr><td>object</td><td>object deserialiser</td><td>StreamEventObjectDeserializer</td></tr></tbody></table>

### JSON

This parser reads a converts a JSON string formatted StreamEvent to a `StreamEvent` object.

<table><thead><tr><th width="151.33333333333331">Attribute</th><th width="298">Description</th><th width="197">Type</th><th data-type="checkbox">Required</th></tr></thead><tbody><tr><td>date format</td><td>Provide a custom date format. See  <a href="https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html">JavaDocs</a> for further data format options.</td><td>String<br>Default: yyyy/MM/dd</td><td>false</td></tr></tbody></table>

#### Example&#x20;

```yaml
deserializer:
    parser: 
        json deserializer:
            date format: dd/MM/yyyy
```

### CSV

This parser has extra optional setting to support ability to custom parsing through the use of defining a custom header.

<table><thead><tr><th width="151.33333333333331">Attribute</th><th width="298">Description</th><th width="197">Type</th><th data-type="checkbox">Required</th></tr></thead><tbody><tr><td>type map</td><td>Map of fields to types. See supported types</td><td>Map&#x3C;String,Type></td><td>true</td></tr><tr><td>delimiter</td><td>Define a custom file delimiter </td><td>String<br>Default: ","</td><td>false</td></tr><tr><td>date format</td><td>Provide a custom date format. </td><td>String<br>Default: yyyy/MM/dd</td><td>false</td></tr></tbody></table>

#### Example&#x20;

```yaml
deserializer:
    parser: 
        csv deserializer:
            type map:
               symbol: STRING
               rate:   DOUBLE
```

#### Supported Types

Following types are supported

* DOUBLE, FLOAT, LONG, INTEGER, SHORT, BYTE, BOOLEAN, STRING, DATE

### Object

This parser reads a converts a binary formatted `StreamEvent` to a `StreamEvent` object

#### Example&#x20;

```yaml
deserializer:
    parser: 
        object deserializer: {}
```