Create metrics

Define a set of time based metrics that are generated using captured and stored streamed events

Objective

Metrics are defined as individual units referred to as a metric family. A metric family is an SQL query that defines metric calculations computed within a continuous scheduled cycle, making them available to all processors.

Here are some important areas to understand when creating metrics:

  • The term metrics family refers the the set of metrics created under the logical name provided.

  • Access to a specific set of metrics is performed by a key look up.

  • Metric storage is provided by SQL tables and therefore must match the query result.

  • Metrics are indexed by key.

  • Queries must follow the DuckDB SQL dialect, see DuckDB documentation for a comprehensive guide.

In addition to the following example, you can read the tutorial on how to create your own metric.

Tutorial creation of custom metrics

Example

This example configures a list of metrics to be computed.

metrics engine:
  ...
  foreach metric compute:
    metrics:
      #
      # Metric family: BidMovingAverage
      #
      - name: BidMovingAverage
        #
        # Metric definition
        #
        metric key: symbol
        table definition: standardQuoteAnalyticsStream.BidMovingAverage 
        (symbol VARCHAR, avg_bid_min FLOAT, avg_bid_avg FLOAT,avg_bid_max FLOAT)
        query:
          SELECT symbol,
            MIN(bid) AS 'avg_bid_min',
            AVG(bid) AS 'avg_bid_avg',
            MAX(bid) AS 'avg_bid_max'
          FROM standardQuoteAnalyticsStream.quote
          WHERE
            ingestTime >= date_trunc('minutes',now() - INTERVAL 2 MINUTES) AND 
            ingestTime <= date_trunc('minutes',now())
          GROUP BY symbol
          ORDER BY 1;
        
        #
        # Truncate BidMovingAverage on startup
        #
        truncate on start: true
        
        #
        # Manage metric storage
        #
        compaction policy:
          frequency: 8
          time unit: HOURS

Attributes schema

Each metric is defined using the below metrics.

AttributeDescriptionData TypeRequired

name

Unique metric family name

String

metric key

Unique key for the metric to be used to optimise query generation and processing

String

query

Metrics ANSI SQL query. See DuckDB SQL documentation

String

table definition

SQL table definition for the resulting metrics

String

truncate on start

Truncate metric data on restart. Note if you import metrics using the initialisation DSL element you will need to set this to false.

See manage metrics for detailed information

Boolean Default: true

compaction policy

Manage the metric storage.

See manage metrics for detailed information

Last updated