Field Tokenizer
Event enricher using a field tokenizer DSL that applies user defined tokenization logic to a specified event field.
Event enricher using a field tokenizer DSL that applies user defined tokenization logic to a specified event field. The resulting tokenized fields are added in to the source event before passing on to the next component.
Available plugins
The above example will take the aggregated value of longitude and latitude and split it to independent fields and added to the StreamEvent object.
Attributes
Attribute | Description | Data Type | Required |
---|---|---|---|
tokenizers | Map of field and decoders to perform tokenization | Map<String, FieldTokenizer> |
FieldTokenizer API
Tokenization is typically used in many data engineering processing pipelines to extract key data elements from aggregate values. Stream processing also using the same functionality to drive further processing requirements such as external data enrichments, decoding composite value in to component parts to drive complex logic paths etc. Joule provides an API to define a tokenizition process.
API
Code example
Below is a very simple example of breaking out a field value using a comma to split the required values.
Addtional Information
The developer builds and deploys ObfuscationTypes
in to the Joule platform using the provided SDK. Follow the instructions on how to build and deploy custom code using the Joule SDK.
Last updated