Joule
  • Welcome to Joule's Docs
  • Why Joule?
    • Joule capabilities
  • What is Joule?
    • Key features
    • The tech stack
  • Use case enablement
    • Use case building framework
  • Concepts
    • Core concepts
    • Low code development
    • Unified execution engine
    • Batch and stream processing
    • Continuous metrics
    • Key Joule data types
      • StreamEvent object
      • Contextual data
      • GeoNode
  • Tutorials
    • Getting started
    • Build your first use case
    • Stream sliding window quote analytics
    • Advanced tutorials
      • Custom missing value processor
      • Stateless Bollinger band analytics
      • IoT device control
  • FAQ
  • Glossary
  • Components
    • Pipelines
      • Use case anatomy
      • Data priming
        • Types of import
      • Processing unit
      • Group by
      • Emit computed events
      • Telemetry auditing
    • Processors
      • Common attributes
      • Filters
        • By type
        • By expression
        • Send on delta
        • Remove attributes
        • Drop all events
      • Enrichment
        • Key concepts
          • Anatomy of enrichment DSL
          • Banking example
        • Metrics
        • Dynamic contextual data
          • Caching architecture
        • Static contextual data
      • Transformation
        • Field Tokeniser
        • Obfuscation
          • Encryption
          • Masking
          • Bucketing
          • Redaction
      • Triggers
        • Change Data Capture
        • Business rules
      • Stream join
        • Inner stream joins
        • Outer stream joins
        • Join attributes & policy
      • Event tap
        • Anatomy of a Tap
        • SQL Queries
    • Analytics
      • Analytic tools
        • User defined analytics
          • Streaming analytics example
          • User defined analytics
          • User defined scripts
          • User defined functions
            • Average function library
        • Window analytics
          • Tumbling window
          • Sliding window
          • Aggregate functions
        • Analytic functions
          • Stateful
            • Exponential moving average
            • Rolling Sum
          • Stateless
            • Normalisation
              • Absolute max
              • Min max
              • Standardisation
              • Mean
              • Log
              • Z-Score
            • Scaling
              • Unit scale
              • Robust Scale
            • Statistics
              • Statistic summaries
              • Weighted moving average
              • Simple moving average
              • Count
            • General
              • Euclidean
        • Advanced analytics
          • Geospatial
            • Entity geo tracker
            • Geofence occupancy trigger
            • Geo search
            • IP address resolver
            • Reverse geocoding
            • Spatial Index
          • HyperLogLog
          • Distinct counter
      • ML inferencing
        • Feature engineering
          • Scripting
          • Scaling
          • Transform
        • Online predictive analytics
        • Model audit
        • Model management
      • Metrics engine
        • Create metrics
        • Apply metrics
        • Manage metrics
        • Priming metrics
    • Contextual data
      • Architecture
      • Configuration
      • MinIO S3
      • Apache Geode
    • Connectors
      • Sources
        • Kafka
          • Ingestion
        • RabbitMQ
          • Further RabbitMQ configurations
        • MQTT
          • Topic wildcards
          • Session management
          • Last Will and Testament
        • Rest endpoints
        • MinIO S3
        • File watcher
      • Sinks
        • Kafka
        • RabbitMQ
          • Further configurations
        • MQTT
          • Persistent messaging
          • Last Will and Testament
        • SQL databases
        • InfluxDB
        • MongoDB
        • Geode
        • WebSocket endpoint
        • MinIO S3
        • File transport
        • Slack
        • Email
      • Serialisers
        • Serialisation
          • Custom transform example
          • Formatters
        • Deserialisers
          • Custom parsing example
    • Observability
      • Enabling JMX for Joule
      • Meters
      • Metrics API
  • DEVELOPER GUIDES
    • Setting up developer environment
      • Environment setup
      • Build and deploy
      • Install Joule
        • Install Docker demo environment
        • Install with Docker
        • Install from source
        • Install Joule examples
    • Joulectl CLI
    • API Endpoints
      • Mangement API
        • Use case
        • Pipelines
        • Data connectors
        • Contextual data
      • Data access API
        • Query
        • Upload
        • WebSocket
      • SQL support
    • Builder SDK
      • Connector API
        • Sources
          • StreamEventParser API
        • Sinks
          • CustomTransformer API
      • Processor API
      • Analytics API
        • Create custom metrics
        • Define analytics
        • Windows API
        • SQL queries
      • Transformation API
        • Obfuscation API
        • FieldTokenizer API
      • File processing
      • Data types
        • StreamEvent
        • ReferenceDataObject
        • GeoNode
    • System configuration
      • System properties
  • Deployment strategies
    • Deployment Overview
    • Single Node
    • Cluster
    • GuardianDB
    • Packaging
      • Containers
      • Bare metal
  • Product updates
    • Public Roadmap
    • Release Notes
      • v1.2.0 Join Streams with stateful analytics
      • v1.1.0 Streaming analytics enhancements
      • v1.0.4 Predictive stream processing
      • v1.0.3 Contextual SQL based metrics
    • Change history
Powered by GitBook
On this page
  • Overview
  • Example
  • Pipeline
  • Example
  • Metrics engine
  • Example
  • policy
  • Metrics computations

Was this helpful?

  1. Components
  2. Pipelines

Processing unit

Quickly build custom business use cases

PreviousTypes of importNextGroup by

Last updated 6 months ago

Was this helpful?

Overview

At its simplest form, a processing unit provides two key functions; a stream processing pipeline and a metrics engine that generate metrics based on a time schedule policy. This forms the core of data handling in Joule.

The processing unit is composed of two main components:

  1. A configurable series of processors that perform transformations, filters, and aggregations on event data. Pipelines are built from Joule’s set of core processors and allow users to construct sophisticated processing workflows.

  2. This engine calculates complex metrics on a defined schedule using SQL-based queries. Metrics are generated and updated at regular intervals, allowing for real-time insights into data.

Example

stream:
  ...
  processing unit:              
      pipeline:
        # Here comes the list of processors

      metrics engine:
        # Here comes the metrics declarations            

Pipeline

A pipeline is a sequence of processors that compute functions on events. Joule provides out-of-the-box a set of core processors to enable you to build useful use cases, learn more about the available processors and analytic tools.

Example

This example defines a pipeline that filters, groups, and aggregates events:

  1. Filter Filters events that match where symbol != 'A'.

  2. Tumbling time window Groups events into 5-second windows.

  3. Aggregations Calculates MIN and MAX values for ask and bid fields, grouped by symbol.

  4. Event emission Outputs one event per symbol with aggregated values at the end of each window.

stream:
  ...
  pipeline:
    - filter:
        expression: "symbol != 'A'"
    - time window:
        emitting type: tumblingQuoteAnalytics
        aggregate functions:
          MIN: [ask, bid]
          MAX: [ask, bid]
        policy:
          type: tumblingTime
          window size: 5000

Metrics engine

The metrics engine provides the ability to compute complex metrics based upon SQL queries, the embedded SQL engine provides.

This feature which is enabled by default

Example

stream:
  ...
  metrics engine:
      policy:
        timeUnit: MINUTES
        frequency: 1
        startup delay: 2
  
      foreach metric compute:
        metrics:
          - name: BidMovingAverage
            metric key: symbol
            table definition: standardQuoteAnalyticsStream.BidMovingAverage (symbol VARCHAR, avg_bid_min FLOAT, avg_bid_avg FLOAT,avg_bid_max FLOAT)                          
            query:
              SELECT symbol, 
              AVG(bid_MAX) AS mt_avg_bid_min,
              AVG(bid_MAX) AS mt_avg_bid_max
              OVER (
              PARTITION BY symbol ORDER BY eventTime ASC
              RANGE BETWEEN INTERVAL 1 SECONDS PRECEDING
              AND INTERVAL 1 SECONDS FOLLOWING)
              AS 'bid max Moving Average'
              FROM quotesStream.tumblingQuoteAnalytics_View
              ORDER BY 1;
            truncate on start: true
            compaction:
              frequency: 8
              timeUnit: HOURS

policy

The policy block defines the schedule for executing metrics calculations and managing data within the metrics engine.

The following table describes each attribute.

Attribute
Description
Required
Default
Supported values

policy

Defines the scheduling policy for metric calculations, setting the timing and intervals for updates

N/A

N/A

timeUnit

Sets the time unit used for the frequency and startup delay intervals

MINUTES

SECONDS, MINUTES, HOURS

frequency

Specifies how often metric calculations are performed

1 Minute

Any positive integer in timeUnit

startup delay

Delay before the initial metric calculation

5 Minutes

Any positive integer in timeUnit

Metrics computations

The foreach metric compute syntax defines a metric table, computation, management and assigns it to a named metric family.

The following table describes each attribute.

Attributes
Description
Required
Default
Supported values

name

A unique identifier for the metric; also referred to as the metrics family name

N/A

N/A

metric key

Generates optimised metric queries for user lookups and management functions

N/A

N/A

table definition

Defines the SQL table for storing and accessing metrics, including the schema as part of the table name

N/A

N/A

query

An ANSI SQL query executed periodically, with results inserted into the defined metric table

N/A

N/A

Management

Default management processes on startup for efficient memory and housekeeping. Enabled hourly

N/A

N/A

truncate on start

Truncates the table on startup if set to true

true

true, false

compaction

Removes outdated metrics according to a set period, ensuring efficient use of storage

hourly

N/A

frequency

Defines the interval between metric compaction processes

hourly

N/A

timeUnits

Specifies the time units for frequency, supporting HOURS and MINUTES

HOURS

SECONDS, MINUTES, HOURS

Stream processors
Metrics engine

Processors

Processors are the core of the Joule platform, each performing a specific task. These create use case when linked together

Analytics tools

Define math expressions or provide as a file using Joule supported languages and APIs