Processor API
Processors form the core of the joule platform. Given the importance of Processors, an API has been provided to enable developers to build and extend the capabilities of the platform.
Last updated
Was this helpful?
Processors form the core of the joule platform. Given the importance of Processors, an API has been provided to enable developers to build and extend the capabilities of the platform.
Last updated
Was this helpful?
Create project using the template
Implement custom processor
Build, test and package
Deploy
We have provided a project template project to quick start development. The project can be found . Clone the template project and copy relevant code and structure to your own project
Processors differ from connectors as they do not require, currently, a specification and builder classes. So jump right in and create and name a class that reflects the processing function. Joule provides the core logic such as batching, cloning, linking of data stores, and a unique processor UUID for event change lineage.
Key areas of implementation
Define processor DSL namespace
Class definition to plugins.properties
Initialize and apply methods
Attribute setters
For Joule to load and initialised the component the processor must be defined within the plugins.properties
file under the META-INF/services
directory
initialize
and apply
methodsAll processor Initialize functions are called before any event processing. Add your custom initialisation logic e.g. priming expensive reference data attributes, calculations etc,.
Add your custom event processing logic within the apply function. Joule provides a single event to processors through micro batch dispatches, this includes event cloning.
To gain processor JMX telemetry add relevant metrics. Every platform component with defined metrics will be accessible within a JMX monitoring platform.
Note: If you would like to perform batch processing override the below method.
The template project provides basic JUnit test to validate DSL. The project will execute these tests during the gradle build cycle and deploy to your local maven repository.
Below is a simple data quality checker for a specific field using a default value in the event of it being missing.
Once your package has been successfully created you are ready to deploy to a Joule project. The resulting jar artefact needs to be placed in to the userlibs
directory in your Joule projects directory. See provided examples for further directions.