User Defined Analytics

Declarative stateful math expressions evaluated in real-time


Overview

The analytics processor provides a rich set of features that enable developers to define and execute analytics. This feature is idea for executing an event based analytic expression using attributes present within the event, linked reference data and passed variables.

Currently only Javascript and Python expressions are supported

Key features

  • Expression evaluation and execution

  • Scripting and function execution

  • Stateful analytics

  • Analytic result memory

Processor architecture

The key analytic processor objective is to provide analytic deployment flexibility for the developer. The ambition is to enable a quick test, learn and refine cycle that reduces time to production deployment.


Execution models

  • Expression only

  • Script and expression

See User Defined Scripts documentation for the script based execution models.

Expression only

When you only want to execute an expression that can be defined as a mathematical formula use this method to execute independent event based calculations. provide the expression along with the required core attributes, see below.

analytic:
  expression: (bid - ask) / scalar
  assign to: new_event_variable

Expression and State

This example demonstrates how you would use provided constants for a calculation. This same method can be used to prime a calculation with a starting value for the initial calculation and then being it replaced with updated values by using the same assign to variable.

analytic:
  expression: "(ask + (bid - ask) / 2.0) / scaled_price"
  assign to: scaled_price
  variables:
      scaled_price : 120.21
  stateful:
      assign to: scaled_price
      memory capacity: 1

See Stateful variables for further documentation.

Expression and script

Use this option when you have pre-existing scripts which you want to leverage within a stream processing context.

analytic:
  script: ./scripts/js/preExistingFunctions.js
  expression: bid - squareRoot(`${ask}` / `${bid}`)
  assign to: new_event_variable

Javascript function

export function squareRoot(num) {
    if (num < 0) return NaN;
    return Math.sqrt(num);
}

The above example would apply the bid and ask event attributes to the expression. Note the current implementation requires the script to be provided using the js extension due to the way the above expression is defined.


Variables

Variables and the event are passed in to the execution context per event, including stateful variables, see below. This example demonstrates how you would use a constant as a scaling factor for a calculation.

analytic:  
  expression: (highest_price - lowest_price) / scaling_factor
  assign to: scaled_price
  variables:
      scaling_factor : 120.21 

Variables are provided as a map of key and numerical values.

Stateful variables

Stateful variables provide a key function whereby the previous value/s can be used within the current calculation context.

analytic: 
  expression: (ask + (bid - ask) / 2.0) / Math.avg(scaled_price)
  assign to: scaled_price
  variables:
      scaled_price : 100.00
  stateful:
      assign to: scaled_price
      memory capacity: 25  

The above example primes the first calculation with a starting value for the initial calculation and thereafter replaced with the computed values from the stateful memory. This is achieved using the same assign to variable set within the variables section.


DSL Attributes

The following section details the DSL attributes supported.

Core Attributes

The follow attributes are used for expression and scripting.

Execution Attributes

These attributes define what is need to execute analytical functions over a stream events.

Stateful Attributes

Store previous computed values for next computation cycle. Honours the groupby definition. The previous computed value can be used on the next calculation.

Last updated