Enrichment

Enrich events using linked reference data, metrics and analytics

Overview

In advanced streaming scenarios, it's often necessary to add extra contextual data to improve processing accuracy and decision-making.

By using low-latency data stores, streaming events can be quickly enriched with up-to-date information, ensuring fast and responsive data retrieval.

Enrichment is a common practice across industries and Joule makes it simpler.

Stream & batch processed data can be combined with Joule.

Use cases

Here are some use cases of how it can be applied in stream processing:

1. Customer data

   Adding information such as data bundle plans or credit ratings.

2. Company metrics

   Enriching streams with market ratings or previous day stock sensitivities.

3. Technical operations

   Including data like cell tower signal coverage or machine utilisation limits.

4. Geofencing

   Enriching streams with static geofences for places of interest.

5. Dynamic triggers

   Incorporating real-time business triggers like SLAs or stock price levels.

6. Domain-specific metrics

   Adding general KPIs or metrics for domain-level insights.

Enrichment options

Contextual data, usually stored in data marts or warehouses changes slowly. Integrating it into a high-throughput streaming platform can cause processing delays.

A stream processing platform is only as fast as its slowest process.

These processing delays are often due to the I/O overhead from the network and storage layers.

To tackle these challenges, Joule offers a set of optimised solutions.

Metrics enrichment

Enrich events with pre-calculated metrics to facilitate further pipeline processing

In-motion contextual data

Use linked low latency data stores to enrich events with fresh reference data

Static contextual data

Enrich events with static reference data