# Deserialisers

## Overview

Joule's deserialization framework enables seamless conversion of raw domain-specific data into `StreamEvent` objects for processing.

By leveraging built-in deserializers (i.e., JSON, CSV, AVRO) or custom parsers, it ensures compatibility with various data formats and workflows, allowing precise transformation tailored to application needs.

For advanced scenarios, the [StreamEventParser API](#streameventparser-api) supports custom parsing when standard deserializers cannot handle unique formats or structures. This flexibility allows developers to map domain-specific objects into `StreamEvent` collections, ensuring smooth integration with Joule's processing pipeline.

{% hint style="info" %}
Every connector type will have it preferred supported deserialisation method
{% endhint %}

This page serves as a guide for configuring data deserialisation, with examples and options for effective data integration in Joule.

This page includes the following:

1. <mark style="color:green;">**Deserialiser DSL element**</mark>\
   Describes the deserialisation element, which defines how received data is parsed ready for Joule processing.
2. <mark style="color:green;">**Kafka example**</mark>\
   Provides an example Kafka consumer configuration that deserialises incoming domain events to `StreamEvents`.
3. <mark style="color:green;">**Custom parser API**</mark>\
   Covers the `StreamEventParser` interface for custom deserialisation when unique data parsing is required.
4. <mark style="color:green;">**AVRO deserialiser**</mark>\
   Introduce how [AVRO](https://avro.apache.org/) can be used to transform domain events to Joule `StreamEvent` events using a provided AVRO IDL schema file.
5. <mark style="color:green;">**Available Implementations**</mark>\
   Learn how to use OOTB deserialisers for JSON and CSV event data.

## Deserialiser DSL element

The deserialiser DSL element defines the deserialisation process for `StreamEvent`s to prepare them for Joule processing.

It allows developers to specify how data should be transformed into `StreamEvent`s using custom parsing logic, enabling precise handling of various data formats and structures tailored to specific application needs.

{% hint style="info" %}
For a [custom example on deserialisers](/joule/components/connectors/serialisers/deserialisers/custom-parsing-example.md)
{% endhint %}

### Example

This code snippet shows a basic setup for a Kafka consumer within Joule, where JSON `StreamEvent` string objects are converted to`StreamEvent` objects ready for pipeline processing.

```yaml
kafkaConsumer:
   ...
   deserializer:
      parser: 
         json deserializer: {} 
```

### Attributes schema

<table><thead><tr><th width="155.33333333333331">Attribute</th><th width="334">Description</th><th>Type</th></tr></thead><tbody><tr><td>parser</td><td>Parser interface to convert a Object to a collection of stream events</td><td><p><code>StreamEventParser</code></p><p>Default: <code>StreamEventJsonDeserializer</code></p></td></tr><tr><td>compressed</td><td>If passed payload is compressed</td><td><p>Boolean</p><p>Default: false</p></td></tr><tr><td>batch</td><td>Is payload a batch as events</td><td><p>Boolean</p><p>Default: false</p></td></tr><tr><td>properties</td><td>Specific properties required for custom deserialisation process</td><td>Properties</td></tr></tbody></table>

## Parser

Parser interface to converts a received Object to a collection of stream events.&#x20;

### StreamEventParser API

For complex use cases whereby AVRO cannot be used a custom parser will be needed. In this case developers can provide a parsing solution using the `StreamEventParser` API.

#### Example

This example shows how a custom parser (`QuoteToStreamEventParser`) transforms `Quote` domain objects into `StreamEvents` that can be processed by Joule. Deserialisation in action.

```yaml
kafkaConsumer:
  ... 
  deserializer:
    parser: 
      com.fractalworks.examples.banking.data.QuoteToStreamEventParser
    
    key deserializer: 
      org.apache.kafka.common.serialization.IntegerDeserializer
    
    value deserializer: 
      com.fractalworks.streams.transport.kafka.serializers.object.ObjectDeserializer
```

### AVRO deserialisation

This deserialiser has extra setting to **support ability to load the target schema**. Currently only local schema files are supported with schema registry support on request.

The transformer automatically maps domain attributes to `StreamEvent` attributes using a provided AVRO schema IDL. Currently only local schema files are supported with schema registry support on request.

{% hint style="success" %}
Integrate Joule to any existing system using already established data structures
{% endhint %}

#### Example

```yaml
kafkaConsumer:
   ...
   deserializer:
      parser: 
         avro deserializer: 
            schema: /home/myapp/schema/customer.avro
```

#### Attributes schema

<table><thead><tr><th width="151.33333333333331">Attribute</th><th width="298">Description</th><th width="197">Type</th><th data-type="checkbox">Required</th></tr></thead><tbody><tr><td>schema</td><td>Path and name of schema file</td><td>String</td><td>true</td></tr><tr><td>field mapping</td><td>Map source fields to map to a target StreamEvent field</td><td>Map&#x3C;String,String></td><td>false</td></tr></tbody></table>

## Available Implementations

Joule provides several deserialiser implementations for different data formats, each requiring a full package namespace.

Key implementations include:

1. <mark style="color:green;">**JSON deserialiser**</mark>\
   Parse StreamEvent JSON events into concrete objects.
2. <mark style="color:green;">**CSV Deserialiser**</mark>\
   Parse CSV formatted data using options for custom headers and a type map to define data types like `STRING` and `DOUBLE`.
3. <mark style="color:green;">**Object Deserialiser**</mark>\
   Parse StreamEvent binary events into concrete objects.

<table><thead><tr><th width="151.33333333333331">Format</th><th>DSL</th><th>Class</th></tr></thead><tbody><tr><td>json</td><td>json deserialiser</td><td>StreamEventJsonDeserializer</td></tr><tr><td>csv</td><td>csv deserialiser</td><td>StreamEventCSVDeserializer</td></tr><tr><td>object</td><td>object deserialiser</td><td>StreamEventObjectDeserializer</td></tr></tbody></table>

### JSON

This parser reads a converts a JSON string formatted StreamEvent to a `StreamEvent` object.

<table><thead><tr><th width="151.33333333333331">Attribute</th><th width="298">Description</th><th width="197">Type</th><th data-type="checkbox">Required</th></tr></thead><tbody><tr><td>date format</td><td>Provide a custom date format. See  <a href="https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html">JavaDocs</a> for further data format options.</td><td>String<br>Default: yyyy/MM/dd</td><td>false</td></tr></tbody></table>

#### Example&#x20;

```yaml
deserializer:
    parser: 
        json deserializer:
            date format: dd/MM/yyyy
```

### CSV

This parser has extra optional setting to support ability to custom parsing through the use of defining a custom header.

<table><thead><tr><th width="151.33333333333331">Attribute</th><th width="298">Description</th><th width="197">Type</th><th data-type="checkbox">Required</th></tr></thead><tbody><tr><td>type map</td><td>Map of fields to types. See supported types</td><td>Map&#x3C;String,Type></td><td>true</td></tr><tr><td>delimiter</td><td>Define a custom file delimiter </td><td>String<br>Default: ","</td><td>false</td></tr><tr><td>date format</td><td>Provide a custom date format. </td><td>String<br>Default: yyyy/MM/dd</td><td>false</td></tr></tbody></table>

#### Example&#x20;

```yaml
deserializer:
    parser: 
        csv deserializer:
            type map:
               symbol: STRING
               rate:   DOUBLE
```

#### Supported Types

Following types are supported

* DOUBLE, FLOAT, LONG, INTEGER, SHORT, BYTE, BOOLEAN, STRING, DATE

### Object

This parser reads a converts a binary formatted `StreamEvent` to a `StreamEvent` object

#### Example&#x20;

```yaml
deserializer:
    parser: 
        object deserializer: {}
```


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.fractalworks.io/joule/components/connectors/serialisers/deserialisers.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.