Dynamic contextual data
Use fast, connected data stores to quickly enrich in-motion events with up-to-date context
Joule provides an Apache Geode connector with an embedded client cache.
Objective
Enriching event data with the latest contextual information is crucial for obtaining real-time business insights.
Since this data often resides in data marts or warehouses and changes slowly, integrating it into a high-speed streaming platform can lead to delays, primarily due to the pressure on network and storage systems. Joule addresses this challenge by incorporating an embedded cache within its architecture.
Upon startup, developers can specify the necessary datasets and customise the cache's functionality. This is an enterprise-level solution which supports advanced use cases, enabling real-time insights and rapid decision-making.
Two types of enrichment strategies:
Key based enrichment Use a event attribute as a key to retrieve the value object from the key-value store.
Query based enrichment A flexible approach to retrieve the target value using an object query.
Uses
There are various uses for this filter such as:
In retail integrate real-time sales data with inventory levels using key-based enrichment to ensure that stock levels are updated instantly, allowing for accurate inventory tracking and efficient restocking processes.
In healthcare Utilise query-based enrichment to enrich patient event data with the latest contextual information, such as medication history and allergies, ensuring healthcare providers have comprehensive insights for timely decision-making.
In e-commerce Utilise key-based enrichment to enhance user interactions with contextual product recommendations based on browsing history and past purchases, optimising the shopping experience and increasing conversion rates.
Example & DSL attributes
The enricher allows users to enhance events with multiple data elements through advanced mapping.
The code defines an enricher that enhances events with data from two stores mobiledevices
and mobilecontracts
:
deviceManufacturer Retrieves
deviceManufacturer
andyear_released
usingtac
as the key from themobiledevices
store.modelDetails Retrieves all details about the model as an object using
tac
as the key from themobiledevices
store.contractedDataBundle Uses an SQL query to fetch data from
dataBundleStore
based on theimsi
field, retrieving all attributes.
Key based enrichment
Using the key-based lookup approach requires defining a target key-value store. Joule provides an Apache Geode connector with an embedded client cache.
Return contextual data as a linked map of key-value pairs
Provide return contextual data as a linked object
Attributes schema
Attribute | Description | Data Type |
---|---|---|
by key | Uses the value within the passed event as a lookup key on the linked store | String |
Query based enrichment
Similarly, the query-based approach requires defining a target key-value store.
Attributes schema
Attribute | Description | Data Type |
---|---|---|
by query | Uses the provided query to enrich the event | String |
OQL Note
Joule utilises the Apache Geode platform for dynamic data solutions, offering an enterprise-grade approach that is well-understood and thoroughly tested.
Geode employs a query syntax based on Object Query Language (OQL). While OQL shares many similarities with SQL, it has key differences; for instance, OQL does not support all SQL features like aggregates but allows
Querying complex object graphs
Object attributes
Invoking object methods
Last updated