Dynamic contextual data

Use fast, connected data stores to quickly enrich in-motion events with up-to-date context

Joule provides an Apache Geode connector with an embedded client cache.

Objective

Enriching event data with the latest contextual information is crucial for obtaining real-time business insights.

Since this data often resides in data marts or warehouses and changes slowly, integrating it into a high-speed streaming platform can lead to delays, primarily due to the pressure on network and storage systems. Joule addresses this challenge by incorporating an embedded cache within its architecture.

Upon startup, developers can specify the necessary datasets and customise the cache's functionality. This is an enterprise-level solution which supports advanced use cases, enabling real-time insights and rapid decision-making.

Two types of enrichment strategies:

1. Key based enrichment Use a event attribute as a key to retrieve the value object from the key-value store.

2. Query based enrichment A flexible approach to retrieve the target value using an object query.

Uses

There are various uses for this filter such as:

1. In retail integrate real-time sales data with inventory levels using key-based enrichment to ensure that stock levels are updated instantly, allowing for accurate inventory tracking and efficient restocking processes.

2. In healthcare Utilise query-based enrichment to enrich patient event data with the latest contextual information, such as medication history and allergies, ensuring healthcare providers have comprehensive insights for timely decision-making.

3. In e-commerce Utilise key-based enrichment to enhance user interactions with contextual product recommendations based on browsing history and past purchases, optimising the shopping experience and increasing conversion rates.

Example & DSL attributes

The enricher allows users to enhance events with multiple data elements through advanced mapping.

The code defines an enricher that enhances events with data from two stores mobiledevices and mobilecontracts:

1. deviceManufacturer Retrieves deviceManufacturer and year_released using tac as the key from the mobiledevices store.

2. modelDetails Retrieves all details about the model as an object using tac as the key from the mobiledevices store.

3. contractedDataBundle Uses an SQL query to fetch data from dataBundleStore based on the imsi field, retrieving all attributes.

enricher:
  fields:      
    deviceManufacturer:
      by key: tac
      with values: [deviceManufacturer, year_released]
      using: deviceStore

    modelDetails:
      by key: tac
      as object: true   
      using: deviceStore

    contractedDataBundle:
      by query:  "select * from /userBundle where imsi = ?"
      query fields: [imsi]
      all attributes: true
      using: dataBundleStore

  stores:
    deviceStore:
      store name: mobiledevices

    dataBundleStore:
      store name: mobilecontracts

Key based enrichment

Using the key-based lookup approach requires defining a target key-value store. Joule provides an Apache Geode connector with an embedded client cache.

Return contextual data as a linked map of key-value pairs

deviceManufacturer:
    by key: tac
    with values: [deviceManufacturer, year_released]
    using: deviceStore

Provide return contextual data as a linked object

modelDetails:
  by key: tac
  as object: true   
  using: deviceStore

Attributes schema

Attribute	Description	Data Type
by key	Uses the value within the passed event as a lookup key on the linked store	String

Query based enrichment

Similarly, the query-based approach requires defining a target key-value store.

contractedDataBundle:
    by query:  "select * from /userBundle where imsi = ?"
    query fields: [imsi]
    all attributes: true
    using: dataBundleStore

Attributes schema

Attribute	Description	Data Type
by query	Uses the provided query to enrich the event	String

OQL Note

Joule utilises the Apache Geode platform for dynamic data solutions, offering an enterprise-grade approach that is well-understood and thoroughly tested.

Geode employs a query syntax based on Object Query Language (OQL). While OQL shares many similarities with SQL, it has key differences; for instance, OQL does not support all SQL features like aggregates but allows

• Querying complex object graphs

• Object attributes

• Invoking object methods