Field Tokenizer

Event enricher using a field tokenizer DSL that applies user defined tokenization logic to a specified event field.

Event enricher using a field tokenizer DSL that applies user defined tokenization logic to a specified event field. The resulting tokenized fields are added in to the source event before passing on to the next component.

Available plugins

tokenizer enricher:
  tokenizers:
    longlat : com.fractalworks.streams.examples.telco.enricher.LatitudeLongitudeDecoder

The above example will take the aggregated value of longitude and latitude and split it to independent fields and added to the StreamEvent object.

Attributes

Attribute	Description	Data Type	Required
tokenizers	Map of field and decoders to perform tokenization	Map<String, FieldTokenizer>

FieldTokenizer API

Tokenization is typically used in many data engineering processing pipelines to extract key data elements from aggregate values. Stream processing also using the same functionality to drive further processing requirements such as external data enrichments, decoding composite value in to component parts to drive complex logic paths etc. Joule provides an API to define a tokenizition process.

API

import java.util.Map;
import java.util.Optional;

/**
 * Field tokenizer that will provide a map of tokenized attributes from the passed object
 */
public interface FieldTokenizer {
    Optional<Map<String, Object>> decode(Object value);
}

Code example

Below is a very simple example of breaking out a field value using a comma to split the required values.

import com.fractalworks.streams.sdk.referenceData.FieldTokenizer;
import java.util.HashMap;
import java.util.Map;
import java.util.Optional;

public class LatitudeLongitudeDecoder implements FieldTokenizer {

    public LatitudeLongitudeDecoder() {}

    @Override
    public Optional<Map<String, Object>> decode(Object o) {
        if( o instanceof String) {
            String[] co = ((String) o).split(",");
            Map<String, Object> map = new HashMap<>();
            map.put("latitude", Float.parseFloat(co[1]));
            map.put("longitude", Float.parseFloat(co[0]));
            return Optional.of(map);
        }
        return Optional.empty();
    }
}

Addtional Information

The developer builds and deploys ObfuscationTypes in to the Joule platform using the provided SDK. Follow the instructions on how to build and deploy custom code using the Joule SDK.