Configuration

Set up a contextual data-driven use case

Overview

Joule’s contextual data management is streamlined through a single configuration file, which allows easy customisation and management of the required data sources.

The configuration specifies one or more contextual data sources, with support for

1. MinIO for S3 object storage.

2. Apache Geode as a high-performance cached data source.

This ensures efficient access to static or slow-moving contextual data critical for enriching event processing.

Example

The following is an example configuration for integrating Apache Geode with Joule for external data access. For a detailed explanation on how to use this powerful feature, read the Geode connector documentation.

This setup allows Joule to leverage Geode's low-latency caching capabilities for accessing essential contextual data such as market indices and holidays.

reference data:
  name: banking market data 
  data sources:
    - geode stores:
        name: us markets
        connection:
          locator address: 192.168.86.39
          locator port: 41111
        stores:
          nasdaqIndexCompanies:
            region: nasdaq-companies
            keyClass : java.lang.String
            gii: true
          holidays:
            region: us-holidays
            keyClass : java.lang.Integer

Application example

In Joule, contextual data is integrated into events via the enricher processor, which attaches the relevant contextual data objects to the StreamEvent.

This configuration uses the nasdaqIndex contextual data to enrich the event's companyInformation field by looking up company data based on the symbol key.

Following example showcases how contextual data is applied within the processing pipeline:

enricher:
  fields:
    companyInformation:
      key: symbol
      using: nasdaqIndex
      
    stores:
       nasdaqIndex:
          store name: nasdaqIndexCompanies

Attributes schema

Attribute	Description	Data Type	Required
name	Contextual data store namespace	String
data sources	List of data sources to connect and bind in to the Joule processor	List of connector configurations