Contextual data

Enrich real-time streams with contextual data

Overview

For Joule applications, contextual (or reference) data is crucial for enabling advanced and insightful stream processing. By seamlessly integrating contextual data with real-time events, the system delivers enriched processing outcomes and better informed insights.

Within Joule, contextual data comprises both static and dynamic data:

1. Static data Includes reference information, such as customer contracts, product SKUs, or start-of-day FX rates, machine learning predictive models etc.

2. Dynamic data Consists of real-time metrics that drive KPI computations, machine learning (ML) engineered features, analytics, and other performance-based measures.

Some example applications include:

1. Geospatial location analytics Leveraging well-known areas of interest for real-time geospatial insights.

2. Real-time customer promotions Tailored promotions based on historical buying profiles and real-time customer behaviour.

3. Multi-currency dynamic pricing Adapting pricing strategies based on live exchange rates and market conditions.

4. Real-time spending alerts Powered by pre-engineered ML features to track and alert users of unusual spending patterns.

5. Predictive capacity demand forecasting Using live metrics to forecast future resource or capacity needs with high accuracy.

These capabilities enable Joule to not only process data in real time but also drive actionable insights that influence business decisions.

Key features of Joule's contextual data capability

1. Event enrichment Adds static information (i.e., geographic or device data) to live events for richer analysis.

2. Transformation and validation Transforms and validates event attributes using predefined patterns or rules.

3. Real-time predictive analytics Utilises engineered features for machine learning models.

4. Machine Learning models Load and replace machine learning models

5. KPI computation Generates real-time performance indicators based on enriched event data.

6. Local caching Contextual data is treated as "slow-moving" and often cached locally within processing nodes to optimise performance.

Joule's contextual data in action

Joule provides every processor with implementation interfaces to access and apply contextual data within its processing scope.

The selection of a contextual data store depends on the specific access and usage requirements of each processor. For scenarios involving high-throughput event processing, an integrated, enterprise-grade, low-latency caching solution is available. For processors where models, rules, or mathematical constants require periodic updates, the MinIO S3-compatible solution is provided.

Contextual data is treated differently to event driven data, it is considered slow-moving

For further information on how Joule integrates contextual data review the following documents:

Architecture

Understand how contextual data integrates within Joule

Configuration

Set up a contextual data-driven use case

Types of source implementations

Joule integrates with various data sources to support contextual data within stream processing, these are:

Minio S3

Multi-cloud S3-compatible interface for static contextual data (e.g., ML models)

Apache Geode

High-performance caching platform for static and slow-moving contextual data