Joule
  • Welcome to Joule's Docs
  • Why Joule?
    • Joule capabilities
  • What is Joule?
    • Key features
    • The tech stack
  • Use case enablement
    • Use case building framework
  • Concepts
    • Core concepts
    • Low code development
    • Unified execution engine
    • Batch and stream processing
    • Continuous metrics
    • Key Joule data types
      • StreamEvent object
      • Contextual data
      • GeoNode
  • Tutorials
    • Getting started
    • Build your first use case
    • Stream sliding window quote analytics
    • Advanced tutorials
      • Custom missing value processor
      • Stateless Bollinger band analytics
      • IoT device control
  • FAQ
  • Glossary
  • Components
    • Pipelines
      • Use case anatomy
      • Data priming
        • Types of import
      • Processing unit
      • Group by
      • Emit computed events
      • Telemetry auditing
    • Processors
      • Common attributes
      • Filters
        • By type
        • By expression
        • Send on delta
        • Remove attributes
        • Drop all events
      • Enrichment
        • Key concepts
          • Anatomy of enrichment DSL
          • Banking example
        • Metrics
        • Dynamic contextual data
          • Caching architecture
        • Static contextual data
      • Transformation
        • Field Tokeniser
        • Obfuscation
          • Encryption
          • Masking
          • Bucketing
          • Redaction
      • Triggers
        • Change Data Capture
        • Business rules
      • Stream join
        • Inner stream joins
        • Outer stream joins
        • Join attributes & policy
      • Event tap
        • Anatomy of a Tap
        • SQL Queries
    • Analytics
      • Analytic tools
        • User defined analytics
          • Streaming analytics example
          • User defined analytics
          • User defined scripts
          • User defined functions
            • Average function library
        • Window analytics
          • Tumbling window
          • Sliding window
          • Aggregate functions
        • Analytic functions
          • Stateful
            • Exponential moving average
            • Rolling Sum
          • Stateless
            • Normalisation
              • Absolute max
              • Min max
              • Standardisation
              • Mean
              • Log
              • Z-Score
            • Scaling
              • Unit scale
              • Robust Scale
            • Statistics
              • Statistic summaries
              • Weighted moving average
              • Simple moving average
              • Count
            • General
              • Euclidean
        • Advanced analytics
          • Geospatial
            • Entity geo tracker
            • Geofence occupancy trigger
            • Geo search
            • IP address resolver
            • Reverse geocoding
            • Spatial Index
          • HyperLogLog
          • Distinct counter
      • ML inferencing
        • Feature engineering
          • Scripting
          • Scaling
          • Transform
        • Online predictive analytics
        • Model audit
        • Model management
      • Metrics engine
        • Create metrics
        • Apply metrics
        • Manage metrics
        • Priming metrics
    • Contextual data
      • Architecture
      • Configuration
      • MinIO S3
      • Apache Geode
    • Connectors
      • Sources
        • Kafka
          • Ingestion
        • RabbitMQ
          • Further RabbitMQ configurations
        • MQTT
          • Topic wildcards
          • Session management
          • Last Will and Testament
        • Rest endpoints
        • MinIO S3
        • File watcher
      • Sinks
        • Kafka
        • RabbitMQ
          • Further configurations
        • MQTT
          • Persistent messaging
          • Last Will and Testament
        • SQL databases
        • InfluxDB
        • MongoDB
        • Geode
        • WebSocket endpoint
        • MinIO S3
        • File transport
        • Slack
        • Email
      • Serialisers
        • Serialisation
          • Custom transform example
          • Formatters
        • Deserialisers
          • Custom parsing example
    • Observability
      • Enabling JMX for Joule
      • Meters
      • Metrics API
  • DEVELOPER GUIDES
    • Setting up developer environment
      • Environment setup
      • Build and deploy
      • Install Joule
        • Install Docker demo environment
        • Install with Docker
        • Install from source
        • Install Joule examples
    • Joulectl CLI
    • API Endpoints
      • Mangement API
        • Use case
        • Pipelines
        • Data connectors
        • Contextual data
      • Data access API
        • Query
        • Upload
        • WebSocket
      • SQL support
    • Builder SDK
      • Connector API
        • Sources
          • StreamEventParser API
        • Sinks
          • CustomTransformer API
      • Processor API
      • Analytics API
        • Create custom metrics
        • Define analytics
        • Windows API
        • SQL queries
      • Transformation API
        • Obfuscation API
        • FieldTokenizer API
      • File processing
      • Data types
        • StreamEvent
        • ReferenceDataObject
        • GeoNode
    • System configuration
      • System properties
  • Deployment strategies
    • Deployment Overview
    • Single Node
    • Cluster
    • GuardianDB
    • Packaging
      • Containers
      • Bare metal
  • Product updates
    • Public Roadmap
    • Release Notes
      • v1.2.0 Join Streams with stateful analytics
      • v1.1.0 Streaming analytics enhancements
      • v1.0.4 Predictive stream processing
      • v1.0.3 Contextual SQL based metrics
    • Change history
Powered by GitBook
On this page
  • Objective
  • Execution models
  • Script and Expression
  • Script and function
  • Attributes schema

Was this helpful?

  1. Components
  2. Analytics
  3. Analytic tools
  4. User defined analytics

User defined scripts

Leverage pre-existing analytics scripts within a streaming context

PreviousUser defined analyticsNextUser defined functions

Last updated 11 days ago

Was this helpful?

Objective

The analytics processor feature set is extended to host external scripts within the Joule processing context. Enabling existing scripting assets to be leveraged within a streaming context.

Apply external analytical scripts within a real-time streaming context.

Currently only Javascript ECMAScript 2024 and Python 3.11 are supported

Execution models

This guide explores different execution models for integrating scripts and functions within a stream processing environment.

We will discuss how to use pre-existing JavaScript or Python scripts, custom functions, and expressions to perform complex analytics and calculations on event data.

Script and Expression

Use this option when you have pre-existing scripts which you want to leverage within a stream processing context.

scripting:
  script: ./scripts/js/myCustomFunctions.js
  expression: bid - squareRoot(`${ask}` / `${bid}`)
  assign to: new_event_variable

User defined function

This example applies the bid and ask event attributes to the expression.

Note, the current implementation requires the script to be provided using the js extension due to the way the expression is defined.

All other variables are provided as global within the execution context.

export function squareRoot(num) {
    if (num < 0) return NaN;
    return Math.sqrt(num);
}
import math

def square_root(num):
    if num < 0:
        return float('nan')
    return math.sqrt(num)

Script and function

This option builds upon the previous example, but uses a method which is supplied with a StreamEvent which can further call imported scripts.

All other variables are provided as global within the execution context.

scripting:
  script: ./scripts/js/usedDefinedCustomModuleFunctions.js
  function: totalCompensationPackage
  assign to: total_compensation

User defined function

The event is automatically passed as a method parameter per execution call.

This method uses the Javascript modules which are defined using the mjs file extension.

export function totalCompensationPackage(event) {
    return event.getValue("salary") +
           event.getValue("bonus") + 
           event.getValue("benefits");
}
def total_compensation_package(event):
    return (event.get("salary", 0) +
            event.get("bonus", 0) +
            event.get("benefits", 0))

Attributes schema

Attribute
Description
Data Type
Required

language

Language runtime to use to execute required execution definitions. Currently Javascript (javascript or js) and Python (python or py) are supported

String Default: js

expression

Math expression without an assignment variable. Required if method has not been provided.

String

script

Path of the script to load within the Joule processing context

String

function

Function to execute within the provided script using a StreamEvent as a parameter

String

expand all

Add all StreamEvent attributes to the processing context as independ variables

Boolean Default: true

virtualEnv

Python only setting which set the virtual environment path to execute with the script of python expression within.

String

Learn how to incorporate existing JavaScript files and use expressions that reference event attributes, ideal when you want to calculate new variables on-the-fly.

Understand how to define custom functions that process entire events, allowing for deeper calculations by accessing multiple event attributes.

The same attributes described in User defined analytics are applicable, see .

documentation
Script and expression
Script and function
Graphic data flow of a functioning UDF