Joule
Search
K
Comment on page

Metrics Engine

Joule provides a SQL compliant metrics engine. The engine computes SQL query based metrics using a runtime policy using the events stored within auto generated event tables.

Example

Below computes a single family of metrics BidMovingAverage at set time intervals and saved to an in-memory standardQuoteAnalyticsStream.BidMovingAverage SQL table. The emit definition performs a query lookup against eh table and returns the avg_bid_max for each matching symbol.
processing unit:
metrics engine:
runtime policy:
frequency: 1
startup delay: 2
time unit: MINUTES
foreach metric compute:
metrics:
- name: BidMovingAverage
metric key: symbol
table definition: standardQuoteAnalyticsStream.BidMovingAverage (symbol VARCHAR, avg_bid_min FLOAT, avg_bid_avg FLOAT,avg_bid_max FLOAT)
query:
SELECT symbol,
MIN(bid) AS 'avg_bid_min',
AVG(bid) AS 'avg_bid_avg',
MAX(bid) AS 'avg_bid_max'
FROM standardQuoteAnalyticsStream.quote
WHERE
ingestTime >= date_trunc('minutes',now() - INTERVAL 2 MINUTES) AND ingestTime <= date_trunc('minutes',now())
GROUP BY symbol
ORDER BY 1;
truncate on start: true
compaction policy:
frequency: 8
time unit: HOURS
emit:
select: "symbol, BidMovingAverage.avg_bid_max;WHERE symbol=${symbol} 'avg_bid_max'"
group by:
- symbol

Runtime Policy

This section configures when to start and how frequently metrics are cacluated. Below the metrics computation will start after 2 minutes on process startup and thereafter every 1 minute.
runtime policy:
frequency: 1
startup delay: 2
time unit: MINUTES

Attributes

Attribute
Description
Data Type
Required
frequency
Frequency metrics are computed. Minimum 15 second computation cycles
Long
Default: 1
startup delay
First compute cycle delay.
Long
Default: 5
time unit
Time unit used to set the scheduled processing policy
Supported units: SECONDS, MINUTES, HOURS
TimeUnit
Default: MINUTES

Compute

This section configures a list of metrics to be computed. One or metric families can be configured. Each metric configuration has there own management policies
foreach metric compute:
metrics:
- name: BidMovingAverage
metric key: symbol
table definition: standardQuoteAnalyticsStream.BidMovingAverage (symbol VARCHAR, avg_bid_min FLOAT, avg_bid_avg FLOAT,avg_bid_max FLOAT)
query:
SELECT symbol,
MIN(bid) AS 'avg_bid_min',
AVG(bid) AS 'avg_bid_avg',
MAX(bid) AS 'avg_bid_max'
FROM standardQuoteAnalyticsStream.quote
WHERE
ingestTime >= date_trunc('minutes',now() - INTERVAL 2 MINUTES) AND ingestTime <= date_trunc('minutes',now())
GROUP BY symbol
ORDER BY 1;
truncate on start: true
compaction policy:
frequency: 8
time unit: HOURS

Attributes

Attribute
Description
Data Type
Required
name
Unique metric name
String
metric key
Key for the metric
String
query
Metrics query
SQL query
table definition
SQL table for the resulting metrics
SQL table definition

Metric Management Attributes

Each metric defintiion includes management attributes to control how the metric table is managed throughout the Joule process lifetime. Since Joule uses an in-memory database the size of table needs to be managed

Example

truncate on start: true
compaction policy:
frequency: 8
time unit: HOURS
Attribute
Description
Data Type
Required
truncate on start
Truncate the mtric tables on startup
Boolean
Default: true
compaction policy
frequency
Frequency table compactions will occur
long
Default: 1
time unit
Time unit used to set the scheduled companion policy
Supported units: MINUTES and HOURS
TimeUnit
Default: HOURS

Metrics Access

Metrics can be presented on the final output stage within the emit select statement using a well formed metric query structure.

Declaratively

Users defined a metric query within the select statement with the results being added to the output event. The below example performs a metric lookup based upon the passed symbol value.

Example

emit:
select: "symbol, BidMovingAverage.avg_bid_max;WHERE symbol=${symbol} 'avg_bid_max'"

Query Structure

The declarative query structure is formed of three required components and an optional override output alias.
<metric-family>.<metrics>;<predicate> '<output-alias>'

metric-family

This is the logical name of the group of metrics, which should correspond to a metric name declared within the compute section.

metrics

A comma delimited list of metrics or asterisk. Any defined metric needs to correspond the query projection attributes.

predicate

Matching and filtering statement using standard SQL. This is used to extract only those records that fulfill a specified condition.
-- Simple lookup using a
WHERE symbol='IBM'
-- Parameter replace using the corresponding the event object symbol value
WHERE symbol=${symbol}
-- Apply additonal filtering criteria
WHERE symbol=${symbol} AND avg_bid_max > 120.88

Examples

-- Get all metrics from the BidMovingAverage metrics family for IBM
BidMovingAverage.*;WHERE symbol='IBM'
-- Get the avg_bid_max metric from the BidMovingAverage metrics family all symbols matching the reference ${symbol} variable
BidMovingAverage.avg_bid_max;WHERE symbol=${symbol}
-- Same as above with the ability to define a custom name for the resulting field
BidMovingAverage.avg_bid_max;WHERE symbol=${symbol} 'avg_bid_max'
-- Get the avg_bid_max and avg_bid_min metric from the BidMovingAverage metrics family
BidMovingAverage.avg_bid_max,avg_bid_min;WHERE symbol=${symbol}

Query Interface

Metrics can be used within custom processors my using the Metric Query API, see API documentation for details on how to leverage this capability.