Joule
  • Welcome to Joule's Docs
  • Why Joule?
    • Joule capabilities
  • What is Joule?
    • Key features
    • The tech stack
  • Use case enablement
    • Use case building framework
  • Concepts
    • Core concepts
    • Low code development
    • Unified execution engine
    • Batch and stream processing
    • Continuous metrics
    • Key Joule data types
      • StreamEvent object
      • Contextual data
      • GeoNode
  • Tutorials
    • Getting started
    • Build your first use case
    • Stream sliding window quote analytics
    • Advanced tutorials
      • Custom missing value processor
      • Stateless Bollinger band analytics
      • IoT device control
  • FAQ
  • Glossary
  • Components
    • Pipelines
      • Use case anatomy
      • Data priming
        • Types of import
      • Processing unit
      • Group by
      • Emit computed events
      • Telemetry auditing
    • Processors
      • Common attributes
      • Filters
        • By type
        • By expression
        • Send on delta
        • Remove attributes
        • Drop all events
      • Enrichment
        • Key concepts
          • Anatomy of enrichment DSL
          • Banking example
        • Metrics
        • Dynamic contextual data
          • Caching architecture
        • Static contextual data
      • Transformation
        • Field Tokeniser
        • Obfuscation
          • Encryption
          • Masking
          • Bucketing
          • Redaction
      • Triggers
        • Change Data Capture
        • Business rules
      • Stream join
        • Inner stream joins
        • Outer stream joins
        • Join attributes & policy
      • Event tap
        • Anatomy of a Tap
        • SQL Queries
    • Analytics
      • Analytic tools
        • User defined analytics
          • Streaming analytics example
          • User defined analytics
          • User defined scripts
          • User defined functions
            • Average function library
        • Window analytics
          • Tumbling window
          • Sliding window
          • Aggregate functions
        • Analytic functions
          • Stateful
            • Exponential moving average
            • Rolling Sum
          • Stateless
            • Normalisation
              • Absolute max
              • Min max
              • Standardisation
              • Mean
              • Log
              • Z-Score
            • Scaling
              • Unit scale
              • Robust Scale
            • Statistics
              • Statistic summaries
              • Weighted moving average
              • Simple moving average
              • Count
            • General
              • Euclidean
        • Advanced analytics
          • Geospatial
            • Entity geo tracker
            • Geofence occupancy trigger
            • Geo search
            • IP address resolver
            • Reverse geocoding
            • Spatial Index
          • HyperLogLog
          • Distinct counter
      • ML inferencing
        • Feature engineering
          • Scripting
          • Scaling
          • Transform
        • Online predictive analytics
        • Model audit
        • Model management
      • Metrics engine
        • Create metrics
        • Apply metrics
        • Manage metrics
        • Priming metrics
    • Contextual data
      • Architecture
      • Configuration
      • MinIO S3
      • Apache Geode
    • Connectors
      • Sources
        • Kafka
          • Ingestion
        • RabbitMQ
          • Further RabbitMQ configurations
        • MQTT
          • Topic wildcards
          • Session management
          • Last Will and Testament
        • Rest endpoints
        • MinIO S3
        • File watcher
      • Sinks
        • Kafka
        • RabbitMQ
          • Further configurations
        • MQTT
          • Persistent messaging
          • Last Will and Testament
        • SQL databases
        • InfluxDB
        • MongoDB
        • Geode
        • WebSocket endpoint
        • MinIO S3
        • File transport
        • Slack
        • Email
      • Serialisers
        • Serialisation
          • Custom transform example
          • Formatters
        • Deserialisers
          • Custom parsing example
    • Observability
      • Enabling JMX for Joule
      • Meters
      • Metrics API
  • DEVELOPER GUIDES
    • Setting up developer environment
      • Environment setup
      • Build and deploy
      • Install Joule
        • Install Docker demo environment
        • Install with Docker
        • Install from source
        • Install Joule examples
    • Joulectl CLI
    • API Endpoints
      • Mangement API
        • Use case
        • Pipelines
        • Data connectors
        • Contextual data
      • Data access API
        • Query
        • Upload
        • WebSocket
      • SQL support
    • Builder SDK
      • Connector API
        • Sources
          • StreamEventParser API
        • Sinks
          • CustomTransformer API
      • Processor API
      • Analytics API
        • Create custom metrics
        • Define analytics
        • Windows API
        • SQL queries
      • Transformation API
        • Obfuscation API
        • FieldTokenizer API
      • File processing
      • Data types
        • StreamEvent
        • ReferenceDataObject
        • GeoNode
    • System configuration
      • System properties
  • Deployment strategies
    • Deployment Overview
    • Single Node
    • Cluster
    • GuardianDB
    • Packaging
      • Containers
      • Bare metal
  • Product updates
    • Public Roadmap
    • Release Notes
      • v1.2.0 Join Streams with stateful analytics
      • v1.1.0 Streaming analytics enhancements
      • v1.0.4 Predictive stream processing
      • v1.0.3 Contextual SQL based metrics
    • Change history
Powered by GitBook
On this page
  • Objective
  • Uses
  • Example & DSL attributes
  • Key based enrichment
  • Attributes schema
  • Query based enrichment
  • Attributes schema
  • OQL Note

Was this helpful?

  1. Components
  2. Processors
  3. Enrichment

Dynamic contextual data

Use fast, connected data stores to quickly enrich in-motion events with up-to-date context

PreviousMetricsNextCaching architecture

Last updated 6 months ago

Was this helpful?

Joule provides an connector with an embedded client cache.

Objective

Enriching event data with the latest contextual information is crucial for obtaining real-time business insights.

Since this data often resides in data marts or warehouses and changes slowly, integrating it into a high-speed streaming platform can lead to delays, primarily due to the pressure on network and storage systems. Joule addresses this challenge by incorporating an embedded cache within its architecture.

Upon startup, developers can specify the necessary datasets and customise the cache's functionality. This is an enterprise-level solution which supports advanced use cases, enabling real-time insights and rapid decision-making.

Two types of enrichment strategies:

  1. Key based enrichment Use a event attribute as a key to retrieve the value object from the key-value store.

  2. Query based enrichment A flexible approach to retrieve the target value using an object query.

Uses

There are various uses for this filter such as:

  1. In retail integrate real-time sales data with inventory levels using key-based enrichment to ensure that stock levels are updated instantly, allowing for accurate inventory tracking and efficient restocking processes.

  2. In healthcare Utilise query-based enrichment to enrich patient event data with the latest contextual information, such as medication history and allergies, ensuring healthcare providers have comprehensive insights for timely decision-making.

  3. In e-commerce Utilise key-based enrichment to enhance user interactions with contextual product recommendations based on browsing history and past purchases, optimising the shopping experience and increasing conversion rates.

Example & DSL attributes

The enricher allows users to enhance events with multiple data elements through advanced mapping.

The code defines an enricher that enhances events with data from two stores mobiledevices and mobilecontracts:

  1. deviceManufacturer Retrieves deviceManufacturer and year_released using tac as the key from the mobiledevices store.

  2. modelDetails Retrieves all details about the model as an object using tac as the key from the mobiledevices store.

  3. contractedDataBundle Uses an SQL query to fetch data from dataBundleStore based on the imsi field, retrieving all attributes.

enricher:
  fields:      
    deviceManufacturer:
      by key: tac
      with values: [deviceManufacturer, year_released]
      using: deviceStore

    modelDetails:
      by key: tac
      as object: true   
      using: deviceStore

    contractedDataBundle:
      by query:  "select * from /userBundle where imsi = ?"
      query fields: [imsi]
      all attributes: true
      using: dataBundleStore

  stores:
    deviceStore:
      store name: mobiledevices

    dataBundleStore:
      store name: mobilecontracts

Key based enrichment

Return contextual data as a linked map of key-value pairs

deviceManufacturer:
    by key: tac
    with values: [deviceManufacturer, year_released]
    using: deviceStore

Provide return contextual data as a linked object

modelDetails:
  by key: tac
  as object: true   
  using: deviceStore

Attributes schema

Attribute
Description
Data Type

by key

Uses the value within the passed event as a lookup key on the linked store

String

Query based enrichment

Similarly, the query-based approach requires defining a target key-value store.

contractedDataBundle:
    by query:  "select * from /userBundle where imsi = ?"
    query fields: [imsi]
    all attributes: true
    using: dataBundleStore

Attributes schema

Attribute
Description
Data Type

by query

Uses the provided query to enrich the event

String

OQL Note

Joule utilises the Apache Geode platform for dynamic data solutions, offering an enterprise-grade approach that is well-understood and thoroughly tested.

Geode employs a query syntax based on Object Query Language (OQL). While OQL shares many similarities with SQL, it has key differences; for instance, OQL does not support all SQL features like aggregates but allows

  • Querying complex object graphs

  • Object attributes

  • Invoking object methods

Using the key-based lookup approach requires defining a target key-value store. Joule provides an connector with an embedded client cache.

Apache Geode
Apache Geode