Streaming analytics example

Example on how to run streaming analytics

What we will learn in this example?

In this article, we will learn how to combine results from two separate analytic computations to generate an alert signal. We will break down each stage of the pipeline, from data enrichment to complex analytics and end with custom alerting rules.

Here’s what each stage covers:

1. Pre-Processing How to enrich events by pulling signal thresholds from a database and associating them with each event.

2. User-defined function (UDF) How to apply sliding window analytics with an exponential moving average (EMA) function, which calculates trends over time for specific fields.

3. Aggregate analytics How to merge analytics results using JavaScript expressions to create difference calculations, such as the Moving Average Convergence Divergence (MACD) signal.

4. Alerting Rules How to set up alerting rules based on calculated thresholds and emit alert events when conditions are met.

By the end, we will understand how to set up a streamlined alerting process using a mix of database enrichment, custom functions and real-time rule evaluation.

Overview

This example demonstrates how two independent computed analytic pipeline results can be combined to generate an alerting signal.

Sreaming analytics pipeline

Stream processing application

Stage 1 - Pre-processing

Enrich event with signal thresholds using the internal in-memory database.

- enricher:
    fields:
      company_signals:
        by query: "select * from reference_data.nasdaq_companies where Symbol = ?"
        query fields: [ symbol ]
        with values: [ bid_signal_threshold, ask_signal_threshold ]
        using: JouleDB

Stage 2 - User defined function

Perform complex analytics using event based sliding windows and a stateful analytic user defined function (exponential mean average).

- user defined function:
  ema:
    parameters:
      ema_factor: 0.33333
  fields: [ bid,ask ]
  event history: 12

- user defined function:
    ema:
      parameters:
        ema_factor: 0.33333
    fields: [ bid,ask ]
    event history: 26

Stage 3 - Aggregate analytic

Combine the results of the stateful analytics in to two basic difference calculations using Javascript expressions.

- analytic:
    expression: ema12_ask - ema26_ask
    assign to: macd_ask_signal

- analytic:
    expression: ema12_bid - ema26_bid
    assign to: macd_bid_signal

Stage 4 - Alerting Rules

Finally once the calculations have been completed perform the final predicate using the having expression to send an alerting event to a downstream system

emit:
  select: "symbol, macd_bid_signal, macd_ask_signal"
  having: "macd_bid_signal <= bid_signal_threshold or macd_ask_signal >= ask_signal_threshold"