# Field Tokeniser

## Objective

Split complex aggregated event attributes in to independent attributes using custom field tokeniser plugins

## Uses

The uses of this filter is very much depended upon context to which Joule is deployed within. As such it relies entirely upon custom implementations developed by business developers. 

Listed below are various uses for this type filter:

1. Split an address provided as a single string in to independent address components.
2. Get a device code from a mobile IMEI code.
3. Tokenise sentences ready for LLM processing

## Example & DSL attributes

This example take an aggregated value of longitude and latitude and **split it to independent fields** and added to the `StreamEvent` object. This uses the custom plugin `LatitudeLongitudeDecoder` provided in the telco project, see code below.

```yaml
tokenizer enricher:
  tokenizers:
    longlat : com.fractalworks.streams.examples.telco.enricher.LatitudeLongitudeDecoder
```

### Attributes schema

<table><thead><tr><th width="138">Attribute</th><th width="303">Description</th><th width="196">Data Type</th><th data-type="checkbox">Required</th></tr></thead><tbody><tr><td>tokenizers</td><td>Map of field and tokenizers. Tokenizers are provided by supporting business developers. </td><td>Map&#x3C;String, FieldTokenizer></td><td>true</td></tr></tbody></table>

## Plugin code

The plugin code extracts a latitude and longitude string, splits it by the comma, and converts each part into separate numeric event attributes.

A [FieldTokenizer API](/joule/developer-guides/builder-sdk/transformation-api/fieldtokenizer-api.md) is provided for developers to build and deploy custom implementation.&#x20;

```java
import com.fractalworks.streams.sdk.referenceData.FieldTokenizer;
import java.util.HashMap;
import java.util.Map;
import java.util.Optional;

public class LatitudeLongitudeDecoder implements FieldTokenizer {

    public LatitudeLongitudeDecoder() {}

    @Override
    public Optional<Map<String, Object>> decode(Object o) {
        if( o instanceof String) {
            String[] co = ((String) o).split(",");
            Map<String, Object> map = new HashMap<>();
            map.put("latitude", Float.parseFloat(co[1]));
            map.put("longitude", Float.parseFloat(co[0]));
            return Optional.of(map);
        }
        return Optional.empty();
    }
}
```


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.fractalworks.io/joule/components/processors/transformation/field-tokeniser.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.