Metrics engine

In-memory SQL compliant metrics engine for advance analytics solutions

Overview

Staying informed with near real-time business metrics allows organisations to make swift, decisive actions that enhance success. Joule’s SQL-compliant metrics engine facilitates this by generating real-time analytics and storing metrics through SQL expressions, supporting KPIs, alerting and predictive insights.

By providing a framework for proactive management, Joule enables organisations to focus on clear, measurable goals and make data-driven decisions. The engine computes metrics using event data and runtime policies, empowering users to analyse and react to data effectively as it streams.

Use cases

Here are some use cases of how metrics it can be applied in stream processing:

1. Business metrics monitoring Track intra-day business metrics such as number of orders placed in an hour.

2. Proactive service failure alerting Track key failure indicators to raise early failure alerting.

3. Data quality metrics Monitor data quality in near real-time.

4. ML feature engineering Use calculated metrics as feature vector components.

5. KPI analytics Apply metrics to form intra-day KPIs. For example Conversion rate uses the number of conversions and total audience are running metrics to form the result.

Key features

• ANSI SQL compliant metrics calculation definition

• Apply calculated metrics within processors

• Pre-computed metrics import

• Runtime policy

• Metric management

Powered by the DuckDB In-memory column database

Example

The following example computes a single family of metrics BidMovingAverage at set time intervals and saved to an in-memory standardQuoteAnalyticsStream.BidMovingAverage SQL table.

The emit definition performs a query lookup against the table and returns the avg_bid_max for each matching symbol.

processing unit:
  metrics engine:
    runtime policy:
      frequency: 1
      startup delay: 2
      time unit: MINUTES

    foreach metric compute:
      metrics:
        - name: BidMovingAverage
          metric key: symbol
          table definition: standardQuoteAnalyticsStream.BidMovingAverage 
          (symbol VARCHAR, avg_bid_min FLOAT, avg_bid_avg FLOAT,avg_bid_max FLOAT)
          query:
            SELECT symbol,
            MIN(bid) AS 'avg_bid_min',
            AVG(bid) AS 'avg_bid_avg',
            MAX(bid) AS 'avg_bid_max'
            FROM standardQuoteAnalyticsStream.quote
            WHERE
            ingestTime >= date_trunc('minutes',now() - INTERVAL 2 MINUTES) AND 
            ingestTime <= date_trunc('minutes',now())
            GROUP BY symbol
            ORDER BY 1;
          truncate on start: true
          compaction policy:
            frequency: 8
            time unit: HOURS

emit:
  select: "symbol, BidMovingAverage.avg_bid_max;WHERE symbol=${symbol} 'avg_bid_max'"

group by:
  - symbol

Available metric engine options

In the following articles you can further explore how to interact with the metrics engine.

Create a metric

Define time based metrics using captured events

Apply metrics

Apply metrics within stream processing functions

Manage metrics

Apply runtime compaction policies on calculated metrics

Priming metrics

Prime metrics with existing values on process startup