Comment on page
Field Tokenizer
Event enricher using a field tokenizer DSL that applies user defined tokenization logic to a specified event field.
Event enricher using a field tokenizer DSL that applies user defined tokenization logic to a specified event field. The resulting tokenized fields are added in to the source event before passing on to the next component.
tokenizer enricher:
tokenizers:
longlat : com.fractalworks.streams.examples.telco.enricher.LatitudeLongitudeDecoder
The above example will take the aggregated value of longitude and latitude and split it to independent fields and added to the StreamEvent object.
Attribute | Description | Data Type | Required |
|---|---|---|---|
tokenizers | Map of field and decoders to perform tokenization | Map<String, FieldTokenizer> |
Tokenization is typically used in many data engineering processing pipelines to extract key data elements from aggregate values. Stream processing also using the same functionality to drive further processing requirements such as external data enrichments, decoding composite value in to component parts to drive complex logic paths etc. Joule provides an API to define a tokenizition process.
import java.util.Map;
import java.util.Optional;
/**
* Field tokenizer that will provide a map of tokenized attributes from the passed object
*/
public interface FieldTokenizer {
Optional<Map<String, Object>> decode(Object value);
}
Below is a very simple example of breaking out a field value using a comma to split the required values.
import com.fractalworks.streams.sdk.referenceData.FieldTokenizer;
import java.util.HashMap;
import java.util.Map;
import java.util.Optional;
public class LatitudeLongitudeDecoder implements FieldTokenizer {
public LatitudeLongitudeDecoder() {}
@Override
public Optional<Map<String, Object>> decode(Object o) {
if( o instanceof String) {
String[] co = ((String) o).split(",");
Map<String, Object> map = new HashMap<>();
map.put("latitude", Float.parseFloat(co[1]));
map.put("longitude", Float.parseFloat(co[0]));
return Optional.of(map);
}
return Optional.empty();
}
}
The developer builds and deploys
ObfuscationTypes in to the Joule platform using the provided SDK. Follow the instructions on how to build and deploy custom code using the Joule SDK. Last modified 7mo ago