Joule
  • Welcome to Joule's Docs
  • Why Joule?
    • Joule capabilities
  • What is Joule?
    • Key features
    • The tech stack
  • Use case enablement
    • Use case building framework
  • Concepts
    • Core concepts
    • Low code development
    • Unified execution engine
    • Batch and stream processing
    • Continuous metrics
    • Key Joule data types
      • StreamEvent object
      • Contextual data
      • GeoNode
  • Tutorials
    • Getting started
    • Build your first use case
    • Stream sliding window quote analytics
    • Advanced tutorials
      • Custom missing value processor
      • Stateless Bollinger band analytics
      • IoT device control
  • FAQ
  • Glossary
  • Components
    • Pipelines
      • Use case anatomy
      • Data priming
        • Types of import
      • Processing unit
      • Group by
      • Emit computed events
      • Telemetry auditing
    • Processors
      • Common attributes
      • Filters
        • By type
        • By expression
        • Send on delta
        • Remove attributes
        • Drop all events
      • Enrichment
        • Key concepts
          • Anatomy of enrichment DSL
          • Banking example
        • Metrics
        • Dynamic contextual data
          • Caching architecture
        • Static contextual data
      • Transformation
        • Field Tokeniser
        • Obfuscation
          • Encryption
          • Masking
          • Bucketing
          • Redaction
      • Triggers
        • Change Data Capture
        • Business rules
      • Stream join
        • Inner stream joins
        • Outer stream joins
        • Join attributes & policy
      • Event tap
        • Anatomy of a Tap
        • SQL Queries
    • Analytics
      • Analytic tools
        • User defined analytics
          • Streaming analytics example
          • User defined analytics
          • User defined scripts
          • User defined functions
            • Average function library
        • Window analytics
          • Tumbling window
          • Sliding window
          • Aggregate functions
        • Analytic functions
          • Stateful
            • Exponential moving average
            • Rolling Sum
          • Stateless
            • Normalisation
              • Absolute max
              • Min max
              • Standardisation
              • Mean
              • Log
              • Z-Score
            • Scaling
              • Unit scale
              • Robust Scale
            • Statistics
              • Statistic summaries
              • Weighted moving average
              • Simple moving average
              • Count
            • General
              • Euclidean
        • Advanced analytics
          • Geospatial
            • Entity geo tracker
            • Geofence occupancy trigger
            • Geo search
            • IP address resolver
            • Reverse geocoding
            • Spatial Index
          • HyperLogLog
          • Distinct counter
      • ML inferencing
        • Feature engineering
          • Scripting
          • Scaling
          • Transform
        • Online predictive analytics
        • Model audit
        • Model management
      • Metrics engine
        • Create metrics
        • Apply metrics
        • Manage metrics
        • Priming metrics
    • Contextual data
      • Architecture
      • Configuration
      • MinIO S3
      • Apache Geode
    • Connectors
      • Sources
        • Kafka
          • Ingestion
        • RabbitMQ
          • Further RabbitMQ configurations
        • MQTT
          • Topic wildcards
          • Session management
          • Last Will and Testament
        • Rest endpoints
        • MinIO S3
        • File watcher
      • Sinks
        • Kafka
        • RabbitMQ
          • Further configurations
        • MQTT
          • Persistent messaging
          • Last Will and Testament
        • SQL databases
        • InfluxDB
        • MongoDB
        • Geode
        • WebSocket endpoint
        • MinIO S3
        • File transport
        • Slack
        • Email
      • Serialisers
        • Serialisation
          • Custom transform example
          • Formatters
        • Deserialisers
          • Custom parsing example
    • Observability
      • Enabling JMX for Joule
      • Meters
      • Metrics API
  • DEVELOPER GUIDES
    • Setting up developer environment
      • Environment setup
      • Build and deploy
      • Install Joule
        • Install Docker demo environment
        • Install with Docker
        • Install from source
        • Install Joule examples
    • Joulectl CLI
    • API Endpoints
      • Mangement API
        • Use case
        • Pipelines
        • Data connectors
        • Contextual data
      • Data access API
        • Query
        • Upload
        • WebSocket
      • SQL support
    • Builder SDK
      • Connector API
        • Sources
          • StreamEventParser API
        • Sinks
          • CustomTransformer API
      • Processor API
      • Analytics API
        • Create custom metrics
        • Define analytics
        • Windows API
        • SQL queries
      • Transformation API
        • Obfuscation API
        • FieldTokenizer API
      • File processing
      • Data types
        • StreamEvent
        • ReferenceDataObject
        • GeoNode
    • System configuration
      • System properties
  • Deployment strategies
    • Deployment Overview
    • Single Node
    • Cluster
    • GuardianDB
    • Packaging
      • Containers
      • Bare metal
  • Product updates
    • Public Roadmap
    • Release Notes
      • v1.2.0 Join Streams with stateful analytics
      • v1.1.0 Streaming analytics enhancements
      • v1.0.4 Predictive stream processing
      • v1.0.3 Contextual SQL based metrics
    • Change history
Powered by GitBook
On this page
  • Overview
  • Example & DSL attributes
  • Stores explanation
  • Enrichment example
  • Attributes schema

Was this helpful?

  1. Components
  2. Contextual data

Configuration

Set up a contextual data-driven use case

PreviousArchitectureNextMinIO S3

Last updated 6 months ago

Was this helpful?

Overview

Joule’s contextual data management is streamlined through an unified configuration file that defines all relevant data sources, ensuring consistency and applicability across all deployed use cases.

The configuration specifies one or more contextual data sources, with current support for:

  1. Disk access for S3 object storage.

  2. Memory access as a high-performance cached data source.

This file needs to be deployed before any use case deployments

Example & DSL attributes

The following is an example configuration for integrating Apache Geode with Joule as an external data source. For detailed guidance on leveraging this robust enterprise feature, refer to the .

This configuration specifies key data locations for nasdaqIndexCompanies and holidays. Although these data points may not change frequently, they may have high read demands, making them ideal candidates for hosting within Joule’s in-memory space.

contextual data:
  name: banking market data 
  data sources:
    - geode stores:
        name: us markets
        connection:
          locator address: 192.168.86.39
          locator port: 41111
        stores:
          nasdaqIndexCompanies:
            region: nasdaq-companies
            keyClass : java.lang.String
            gii: true
          holidays:
            region: us-holidays
            keyClass : java.lang.Integer

Stores explanation

A key element of the configuration’s DSL is the stores syntax. In essence, the stores definition binds a logical store name to a specific implementation, with properties tailored to the underlying technology. For instance, an S3 implementation will have different configuration properties compared to a distributed caching solution.

This can be better illustrated through an example where company information needs to be added to each event, and an ML model requires updates during the Joule process’s runtime. In this case, the configurations for nasdaqIndexCompanies and predictors differ, reflecting their distinct underlying technologies and usage patterns.

contextual data:
  name: banking market data 
  data sources:
    - geode stores:
        ...
        stores:
          nasdaqIndexCompanies:
            region: nasdaq-companies
            keyClass : java.lang.String
            gii: true
            
    - minio stores:
        ...
        stores:
          predictors:
            bucketId: models
            initial version Id: 12345
            download dir: /home/joule/nasdaq-companies-model/tmp

Enrichment example

In Joule, contextual data is applied to events through the enricher processor, which attaches the relevant contextual data objects to each StreamEvent.

The example below enriches the event's companyInformation attribute using the nasdaqCompanies contextual data, by performing a lookup based on the symbol key to retrieve relevant company data.

enricher:
  fields:
    companyInformation:
      key: symbol
      using: nasdaqCompanies
      
    stores:
       nasdaqCompanies:
          store name: nasdaqIndexCompanies

Attributes schema

Attribute
Description
Data Type
Required

name

Contextual data store namespace

String

data sources

List of data sources to connect and bind in to the Joule processor

List of connector configurations

Read the document for full feature information and further examples

MinIO
Apache Geode
Geode connector documentation
enrichment