Architecture

Understand how Joule integrates to contextual data solutions

Overview

Joule’s data source interface allows processors to access and utilise contextual data seamlessly across the platform.

Upon startup, the Joule runtime connects to each configured data source, assigning each source a logical name and making it available to all processors.

This setup is managed through a single configuration file, which specifies the necessary contextual data stores (see Configuration section).

Architecture

To maintain high performance in scenarios with large event throughputs, it is essential to choose the right data source implementation.

Poorly optimised data sources can hinder the efficiency of the processing pipeline, particularly during high-frequency data access

To mitigate performance issues and reduce the load from out-of-process I/O operations, Joule recommends caching contextual data locally within the process, especially in high-read environments.

This approach optimises data retrieval speed and enhances the overall processing efficiency, enabling smoother handling of real-time events.

High level architecture

The Joule solution enables external tools to plugin to the Joule processing using the Reference Data Interface.

Streaming Prediction example