Create metrics

Define a set of time based metrics that are generated using captured and stored streamed events

Objective

Metrics are defined as individual units referred to as a metric family. A metric family is an SQL query that defines metric calculations computed within a continuous scheduled cycle, making them available to all processors.

One or more metric families can be configured, each has its own management policy.

Here are some important areas to understand when creating metrics:

• The term metrics family refers the the set of metrics created under the logical name provided.

• Access to a specific set of metrics is performed by a key look up.

• Metric storage is provided by SQL tables and therefore must match the query result.

• Metrics are indexed by key.

• Queries must follow the DuckDB SQL dialect, see DuckDB documentation for a comprehensive guide.

In addition to the following example, you can read the tutorial on how to create your own metric.

Tutorial creation of custom metrics

Example

This example configures a list of metrics to be computed.

metrics engine:
  ...
  foreach metric compute:
    metrics:
      #
      # Metric family: BidMovingAverage
      #
      - name: BidMovingAverage
        #
        # Metric definition
        #
        metric key: symbol
        table definition: standardQuoteAnalyticsStream.BidMovingAverage 
        (symbol VARCHAR, avg_bid_min FLOAT, avg_bid_avg FLOAT,avg_bid_max FLOAT)
        query:
          SELECT symbol,
            MIN(bid) AS 'avg_bid_min',
            AVG(bid) AS 'avg_bid_avg',
            MAX(bid) AS 'avg_bid_max'
          FROM standardQuoteAnalyticsStream.quote
          WHERE
            ingestTime >= date_trunc('minutes',now() - INTERVAL 2 MINUTES) AND 
            ingestTime <= date_trunc('minutes',now())
          GROUP BY symbol
          ORDER BY 1;
        
        #
        # Truncate BidMovingAverage on startup
        #
        truncate on start: true
        
        #
        # Manage metric storage
        #
        compaction policy:
          frequency: 8
          time unit: HOURS

Attributes schema

Each metric is defined using the below metrics.

Attribute	Description	Data Type	Required
name	Unique metric family name	String
metric key	Unique key for the metric to be used to optimise query generation and processing	String
query	Metrics ANSI SQL query. See DuckDB SQL documentation	String
table definition	SQL table definition for the resulting metrics	String
truncate on start	Truncate metric data on restart. Note if you import metrics using the initialisation DSL element you will need to set this to false. See manage metrics for detailed information	Boolean Default: true
compaction policy	Manage the metric storage. See manage metrics for detailed information