Joule
  • Welcome to Joule's Docs
  • Why Joule?
    • Joule capabilities
  • What is Joule?
    • Key features
    • The tech stack
  • Use case enablement
    • Use case building framework
  • Concepts
    • Core concepts
    • Low code development
    • Unified execution engine
    • Batch and stream processing
    • Continuous metrics
    • Key Joule data types
      • StreamEvent object
      • Contextual data
      • GeoNode
  • Tutorials
    • Getting started
    • Build your first use case
    • Stream sliding window quote analytics
    • Advanced tutorials
      • Custom missing value processor
      • Stateless Bollinger band analytics
      • IoT device control
  • FAQ
  • Glossary
  • Components
    • Pipelines
      • Use case anatomy
      • Data priming
        • Types of import
      • Processing unit
      • Group by
      • Emit computed events
      • Telemetry auditing
    • Processors
      • Common attributes
      • Filters
        • By type
        • By expression
        • Send on delta
        • Remove attributes
        • Drop all events
      • Enrichment
        • Key concepts
          • Anatomy of enrichment DSL
          • Banking example
        • Metrics
        • Dynamic contextual data
          • Caching architecture
        • Static contextual data
      • Transformation
        • Field Tokeniser
        • Obfuscation
          • Encryption
          • Masking
          • Bucketing
          • Redaction
      • Triggers
        • Change Data Capture
        • Business rules
      • Stream join
        • Inner stream joins
        • Outer stream joins
        • Join attributes & policy
      • Event tap
        • Anatomy of a Tap
        • SQL Queries
    • Analytics
      • Analytic tools
        • User defined analytics
          • Streaming analytics example
          • User defined analytics
          • User defined scripts
          • User defined functions
            • Average function library
        • Window analytics
          • Tumbling window
          • Sliding window
          • Aggregate functions
        • Analytic functions
          • Stateful
            • Exponential moving average
            • Rolling Sum
          • Stateless
            • Normalisation
              • Absolute max
              • Min max
              • Standardisation
              • Mean
              • Log
              • Z-Score
            • Scaling
              • Unit scale
              • Robust Scale
            • Statistics
              • Statistic summaries
              • Weighted moving average
              • Simple moving average
              • Count
            • General
              • Euclidean
        • Advanced analytics
          • Geospatial
            • Entity geo tracker
            • Geofence occupancy trigger
            • Geo search
            • IP address resolver
            • Reverse geocoding
            • Spatial Index
          • HyperLogLog
          • Distinct counter
      • ML inferencing
        • Feature engineering
          • Scripting
          • Scaling
          • Transform
        • Online predictive analytics
        • Model audit
        • Model management
      • Metrics engine
        • Create metrics
        • Apply metrics
        • Manage metrics
        • Priming metrics
    • Contextual data
      • Architecture
      • Configuration
      • MinIO S3
      • Apache Geode
    • Connectors
      • Sources
        • Kafka
          • Ingestion
        • RabbitMQ
          • Further RabbitMQ configurations
        • MQTT
          • Topic wildcards
          • Session management
          • Last Will and Testament
        • Rest endpoints
        • MinIO S3
        • File watcher
      • Sinks
        • Kafka
        • RabbitMQ
          • Further configurations
        • MQTT
          • Persistent messaging
          • Last Will and Testament
        • SQL databases
        • InfluxDB
        • MongoDB
        • Geode
        • WebSocket endpoint
        • MinIO S3
        • File transport
        • Slack
        • Email
      • Serialisers
        • Serialisation
          • Custom transform example
          • Formatters
        • Deserialisers
          • Custom parsing example
    • Observability
      • Enabling JMX for Joule
      • Meters
      • Metrics API
  • DEVELOPER GUIDES
    • Setting up developer environment
      • Environment setup
      • Build and deploy
      • Install Joule
        • Install Docker demo environment
        • Install with Docker
        • Install from source
        • Install Joule examples
    • Joulectl CLI
    • API Endpoints
      • Mangement API
        • Use case
        • Pipelines
        • Data connectors
        • Contextual data
      • Data access API
        • Query
        • Upload
        • WebSocket
      • SQL support
    • Builder SDK
      • Connector API
        • Sources
          • StreamEventParser API
        • Sinks
          • CustomTransformer API
      • Processor API
      • Analytics API
        • Create custom metrics
        • Define analytics
        • Windows API
        • SQL queries
      • Transformation API
        • Obfuscation API
        • FieldTokenizer API
      • File processing
      • Data types
        • StreamEvent
        • ReferenceDataObject
        • GeoNode
    • System configuration
      • System properties
  • Deployment strategies
    • Deployment Overview
    • Single Node
    • Cluster
    • GuardianDB
    • Packaging
      • Containers
      • Bare metal
  • Product updates
    • Public Roadmap
    • Release Notes
      • v1.2.0 Join Streams with stateful analytics
      • v1.1.0 Streaming analytics enhancements
      • v1.0.4 Predictive stream processing
      • v1.0.3 Contextual SQL based metrics
    • Change history
Powered by GitBook
On this page
  • Objective
  • Processor architecture
  • Execution models
  • Expression only
  • Expression and state
  • Expression and script
  • Variables
  • Stateful variables
  • Attributes schema
  • Execution attributes schema
  • Stateful attributes schema

Was this helpful?

  1. Components
  2. Analytics
  3. Analytic tools
  4. User defined analytics

User defined analytics

Declarative stateful math expressions evaluated in real-time

PreviousStreaming analytics exampleNextUser defined scripts

Last updated 1 month ago

Was this helpful?

Objective

The analytics processor provides a rich set of features that enable developers to define and execute analytics. This feature is idea for executing an event based analytic expression using attributes present within the event, linked reference data and passed variables.

Currently only Javascript ECMAScript 2024 and Python 3.11 are supported

Processor architecture

The key analytic processor objective is to provide analytic deployment flexibility for the developer. The ambition is to enable a quick test, learn and refine cycle that reduces time to production deployment.

Execution models

This section covers methods for executing expressions in stream processing, focusing on three key approaches:

  1. Expression only Execute independent event-based calculations using simple mathematical formulas by defining core attributes.

  2. Expression and state Incorporate constants and manage state to initialise calculations and update them with new data.

  3. Expression and script Leverage pre-existing scripts alongside expressions for more complex calculations.

See User Defined Scripts documentation for script based execution models.

Expression only

When you only want to execute an expression that can be defined as a mathematical formula use this method to execute independent event based calculations provide the expression along with the required core attributes.

analytic:
  expression: (bid - ask) / scalar
  assign to: new_event_variable

Expression and state

This example demonstrates how you would use provided constants for a calculation. This same method can be used to prime a calculation with a starting value for the initial calculation and then being it replaced with updated values by using the same assign to variable.

See Stateful variables for further documentation.

analytic:
  expression: "(ask + (bid - ask) / 2.0) / scaled_price"
  assign to: scaled_price
  variables:
      scaled_price : 120.21
  stateful:
      assign to: scaled_price
      memory capacity: 1

Expression and script

Use this option when you have pre-existing scripts which you want to leverage within a stream processing context.

analytic:
  script: ./scripts/js/preExistingFunctions.js
  expression: bid - squareRoot(`${ask}` / `${bid}`)
  assign to: new_event_variable

Javascript function

The example will apply the bid and ask event attributes to the expression.

Note the current implementation requires the script to be provided using the js extension due to the way the expression is defined.

export function squareRoot(num) {
    if (num < 0) return NaN;
    return Math.sqrt(num);
}
import math

def square_root(num):
    if num < 0:
        return float('nan')  # Return NaN for negative input
    return math.sqrt(num)  # Return the square root for non-negative input

Variables

Variables are provided as a map of key and numerical values.

Variables and the event are passed in to the execution context per event, including stateful variables. This example demonstrates how to apply a constant as a scaling factor for a calculation.

analytic:  
  expression: (highest_price - lowest_price) / scaling_factor
  assign to: scaled_price
  variables:
      scaling_factor : 120.21 

Stateful variables

Stateful variables provide a key function whereby the previous value(s) can be used within the current calculation context.

The example primes the first calculation with a starting value for the initial calculation and thereafter replaced with the computed values from the stateful memory. This is achieved using the same assign to variable set within the variables section.

analytic: 
  expression: (ask + (bid - ask) / 2.0) / Math.avg(scaled_price)
  assign to: scaled_price
  variables:
      scaled_price : 100.00
  stateful:
      assign to: scaled_price
      memory capacity: 25  

Attributes schema

The follow attributes are used for expression and scripting.

Attribute
Description
Data Type
Required

assign to

Assignment variable for the evaluated expression result

String Default: result

variables

Map of constants and seed values for the expression to use. This is an optional variable.

Map<String, Number>

stateful

Store previous computed values for next computation cycle

assignment datatype

Data type to cast too for the assignment variable. See scripting supported data types section

DataType Default: Double

Execution attributes schema

These attributes define what is need to execute analytical functions over a stream events.

Attribute
Description
Data Type

language

Language runtime to use to execute required execution definitions. Currently Javascript (javascript or js) and Python (python or py) are supported

String Default: js

expression

Math expression without an assignment variable. Required if method has not been provided.

String

script

Path of the script to loaded within the Joule processing context

String

function

Function to execute within the provided script using a StreamEvent as a parameter

String

expand all

Add all StreamEvent attributes to the processing context as independ variables

Boolean Default: true

virtualEnv

Python only setting which set the virtual environment path to execute with the script of python expression within.

String

Stateful attributes schema

Store previous computed values for next computation cycle. Honours the groupby definition. The previous computed value can be used on the next calculation.

Attribute
Description
Data Type
Required

assign to

Assignment variable for the result

String

memory capacity

Number of rolling elements to store within a FIFO array

Integer Default: 10

attach memory

Attach a copy of the array contents to the computed event

Boolean Default: false

See

stateful section
Processor overview