> For the complete documentation index, see [llms.txt](https://docs.fractalworks.io/joule/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.fractalworks.io/joule/tutorials/advanced-tutorials/data-quality.md).

# Data quality

## Objective

We will create a simple missing data transformer that fills in missing values by either adding a default value or using the previous field's value.

<figure><img src="/files/SUgTAFUYihAlVqVLwn7M" alt=""><figcaption><p>Source: <a href="https://medium.com/@ayushmandurgapal/data-preprocessing-handling-missing-values-in-a-dataset-5140f77d2a47">Data Preprocessing — Handling Missing Values in a dataset</a></p></figcaption></figure>

### Prerequisites

To get started building a custom processor ensure you have your development environment configured.

Read the [environment setup](/joule/developer-guides/setting-up-developer-environment/environment-setup.md) documentation to get your environment ready to build.&#x20;

## Development steps <a href="#explaining-each-step" id="explaining-each-step"></a>

These instructions cover how to build, deploy a use the processor on to the Joule Platform.&#x20;

{% stepper %}
{% step %}

### Create project using the template <a href="#step-1-create-the-destination-using-the-template" id="step-1-create-the-destination-using-the-template"></a>

We have provided a project template project to quick start development. The project can be found [here](https://gitlab.com/joule-platform/fractalworks-project-templates). Clone the template project and copy relevant code and structure to your own project.

```bash
git clone git@gitlab.com:joule-platform/fractalworks-project-templates.git
```

{% hint style="info" %}
Joule uses Gradle to manage Java dependencies. To add dependencies for your processor, manage them in the `build.gradle` file inside your processors project directory.
{% endhint %}
{% endstep %}

{% step %}

### Implement missing value transformer

Processors differ from connectors as they do not require, currently, a specification and builder classes. So jump right in and create and name a class that reflects the processing function.

&#x20;Joule provides the core logic such as batching, cloning, linking of data stores, and a unique processor UUID for event change lineage.

Key areas of implementation:

* Define processor DSL namespace
* Initialize and apply methods
* Attribute setters and properties
* Add the class definition to plugins.properties&#x20;
* Deploy and apply to a Joule runtime environment

#### Code implementation

```java
package com.yourcompany.processor.transformers;

import com.fasterxml.jackson.annotation.JsonRootName;
import com.fasterxml.jackson.annotation.JsonProperty;

import com.fractalworks.streams.core.data.streams.Context;
import com.fractalworks.streams.core.data.streams.StreamEvent;
import com.fractalworks.streams.core.data.streams.Metric;

import java.util.HashMap;
import java.util.Map;

// JsonRootName value will be used in the use case definition
@JsonRootName(value = "missing value transformer")
public class CustomMissingValueTransformer extends AbstractProcessor {

    private Map<String,Object> previousValue = new HashMap<>();

    private String key;
    private String field;
    
    private Object defaultValue;
    
    public CustomMissingValueTransformer() {
        super();
    }

    @Override
    public void initialize(Properties prop) throws ProcessorException {
        super.initialize(prop);
        // Add specific initialisation code here        
    }

    /**
    * This is were your custom code is provided
    */
    @Override
    public StreamEvent apply(StreamEvent event, Context context) 
        throws StreamsException {
        
        var value = event.getValue(field);
        if(value == null){
            value = (previousValue.containsKey(key) 
                ? previousValue.getValue(previousValue): defaultValue;
            event.addValue(uuid,field,value); 
            if (logger.isInfoEnabled()) {
                logger.info("Updated missing {} value with {}",field,value);
            }              
        }
        previousValue.put(key, value);
        
        // JMX enabled metrics
        metrics.incrementMetric(Metric.PROCESSED);
        return event;
    }
    
    /**
    * Attribute setters and dsl property 
    */

    @JsonProperty(value = "unique key", required = true)
    public void setField(String key) {
        this.key = key;
    }
    
    @JsonProperty(value = "field", required = true)
    public void setField(String field) {
        this.field = field;
    } 
        
    @JsonProperty(value = "default value", required = true)
    public void setDefaultValue(Object defaultValue) {
        this.defaultValue = defaultValue;
    } 
}
```

**Note:** If you would like to perform batch processing override the below method.

```java
public MicroBatch apply(MicroBatch batch, Context context) throws StreamsException;
```

{% endstep %}

{% step %}

### Add to plugins.properties

For Joule to load and initialised the component the processor must be defined within the `plugins.properties` file under the `META-INF/services` directory.

Add the below line in the `plugins.properties` file:

```properties
com.yourcompany.processor.transformers.CustomMissingValueTransformer
```

{% endstep %}

{% step %}

### Build, test and package <a href="#step-1-create-the-destination-using-the-template" id="step-1-create-the-destination-using-the-template"></a>

The template project provides basic JUnit test to validate DSL. The project will execute these tests during the gradle build cycle and deploy to your local maven repository.&#x20;

```bash
gradle build publishToMavenLocal
```

{% endstep %}

{% step %}

### Deploy

Once your package has been successfully created you are ready to deploy to a Joule project.

The resulting jar from the build process needs to be copied to the `userlibs` directory under a Joule project directory. For example using the getting started project copy the file to `quickstart/userlibs` directory.&#x20;

```bash
cp build/libs/<your-processor>.jar <location>/userlibs
```

{% endstep %}

{% step %}

### Now apply to a stream

Lets say, sometimes we do not get a bid value which is needed to trigger an alert.  So overcome a division by zero we provide a default value and use previous values when needed.

```yaml
stream:
  name: nasdaq_major_banks_stream
  eventTimeType: EVENT_TIME
        
  processing unit:
    pipeline:
    # Filter events by major banks
    - filter:
        expression: "(typeof industry !== 'undefined' && 
                      industry == 'Major Banks')"
    
    # Apply 
    - missing value transformer:
        key: symbol
        field: bid
        default value: 1.0
    
  emit:
    select: symbol, bid, ask
    
    # Spread trigger
    having: "((bid - ask) / bid) > 0.015"
    
  group by:
  - symbol
```

Follow the same steps used in the [getting started](/joule/tutorials/getting-started.md) documentation to apply this script.
{% endstep %}
{% endstepper %}

## What we have learnt

As a first process we have covered a number of key features:

* <mark style="color:green;">**Build a simple transformer**</mark>\
  Used the provided template project to quick start development and add custom code within key processor methods.
* <mark style="color:green;">**Built the jar**</mark>\
  Used gradle build tool to build, test and deploy to local maven repo.
* <mark style="color:green;">**Deploy the jar to a Joule runtime environment**</mark>\
  Copied the Jar to an existing local Joule runtime environment&#x20;
* <mark style="color:green;">**Apply transformer within a use case**</mark>\
  Apply the transformer within a use case to provide consistent spread alerts in the event of missing data.