User defined scripts

Leverage pre-existing analytics scripts within a streaming context

Objective

The analytics processor feature set is extended to host external scripts within the Joule processing context. Enabling existing scripting assets to be leveraged within a streaming context.

Apply external analytical scripts within a real-time streaming context.

Currently Javascript ECMAScript 2024, graalpy, Python 3 and 3.11 are supported

Graphic data flow of a functioning UDF

Execution models

This guide explores different execution models for integrating scripts and functions within a stream processing environment.

We will discuss how to use pre-existing JavaScript or Python scripts, custom functions, and expressions to perform complex analytics and calculations on event data.

1. Script and expression Learn how to incorporate existing JavaScript files and use expressions that reference event attributes, ideal when you want to calculate new variables on-the-fly.

2. Script and function Understand how to define custom functions that process entire events, allowing for deeper calculations by accessing multiple event attributes.

Script and Expression

Use this option when you have pre-existing scripts which you want to leverage within a stream processing context.

scripting:
  script: ./scripts/js/myCustomFunctions.js
  expression: bid - squareRoot(`${ask}` / `${bid}`)
  assign to: new_event_variable

User defined function

This example applies the bid and ask event attributes to the expression.

Note, the current implementation requires the script to be provided using the js extension due to the way the expression is defined.

All other variables are provided as global within the execution context.

Javascript

export function squareRoot(num) {
    if (num < 0) return NaN;
    return Math.sqrt(num);
}

Python

import math

def squareRoot(num):
    if num < 0:
        return float('nan')
    return math.sqrt(num)

Script and function

This option builds upon the previous example, but uses a method which is supplied with a StreamEvent which can further call imported scripts.

All other variables are provided as global within the execution context.

scripting:
  script: ./scripts/js/usedDefinedCustomFunctions.js
  language: javascript
  function: totalCompensationPackage
  assign to: total_compensation

User defined function

The event is automatically passed as a method parameter per execution call.

This method uses the Javascript modules which are defined using the mjs file extension.

Javascript

export function totalCompensationPackage(event) {
    return event.getValue("salary") +
           event.getValue("bonus") + 
           event.getValue("benefits");
}

Python

def totalCompensationPackage(event):
    return (event.get("salary", 0) +
            event.get("bonus", 0) +
            event.get("benefits", 0))

Attributes schema

The same attributes described in User defined analytics are applicable, see documentation.

Attribute	Description	Data Type	Required
language	Language runtime to use to execute required execution definitions. Currently Javascript (javascript or js) and Python (python or py) are supported	String Default: js
expression	Math expression without an assignment variable. Required if method has not been provided.	String
script	Path of the script to load within the Joule processing context	String
function	Function to execute within the provided script using a StreamEvent as a parameter	String
expand all	Add all StreamEvent attributes to the processing context as independ variables	Boolean Default: true
options	Provide language specific GraalVm engine options. See GraalVM Options section	Map<String, String> Default: Language specific

GraalVM Options

To change the behaviour of the GraalVm engine language specific options can be passed in using DSl syntax

Javascript

Example

scripting:
  script: ./scripts/js/usedDefinedCustomFunctions.js
  language: javascript
  function: totalCompensationPackage
  assign to: total_compensation
  options:
    js.strict: true
    js.nashorn-compat: true
    js.ecmascript-version: 2024
    js.esm-eval-returns-exports: true

Default options

If no options are provided the following options are applied. For a complete list of available options review the GraalVM Js documentation.

Key	Value
js.esm-eval-returns-exports	true
js.nashorn-compat	true
js.ecmascript-version	2024

Python

Example

scripting: 
  script: ./scripts/python/usedDefinedCustomFunctions.py
  language: python
  function: totalCompensationPackage
  assign to: total_compensation
  options:
    python.PythonPath: ./org.graalvm.python.vfs/venv
    python.Executable: ./org.graalvm.python.vfs/venv/bin/
    python.ForceImportSite: true

Default options

If no options are provided the following options are applied.

Key	Value
python.ForceImportSite	true

Note

Please contact Fractalworks to help you leverage this feature. These resources will need to be understand to fully leverage this capability:

• GraalPy documentation

• Package compatibility