Comment on page
Metrics Engine
Joule provides a SQL compliant metrics engine. The engine computes SQL query based metrics using a runtime policy using the events stored within auto generated event tables.
Below computes a single family of metrics
BidMovingAverage at set time intervals and saved to an in-memory standardQuoteAnalyticsStream.BidMovingAverage SQL table. The emit definition performs a query lookup against eh table and returns the avg_bid_max for each matching symbol.processing unit:
metrics engine:
runtime policy:
frequency: 1
startup delay: 2
time unit: MINUTES
foreach metric compute:
metrics:
- name: BidMovingAverage
metric key: symbol
table definition: standardQuoteAnalyticsStream.BidMovingAverage (symbol VARCHAR, avg_bid_min FLOAT, avg_bid_avg FLOAT,avg_bid_max FLOAT)
query:
SELECT symbol,
MIN(bid) AS 'avg_bid_min',
AVG(bid) AS 'avg_bid_avg',
MAX(bid) AS 'avg_bid_max'
FROM standardQuoteAnalyticsStream.quote
WHERE
ingestTime >= date_trunc('minutes',now() - INTERVAL 2 MINUTES) AND ingestTime <= date_trunc('minutes',now())
GROUP BY symbol
ORDER BY 1;
truncate on start: true
compaction policy:
frequency: 8
time unit: HOURS
emit:
select: "symbol, BidMovingAverage.avg_bid_max;WHERE symbol=${symbol} 'avg_bid_max'"
group by:
- symbol
This section configures when to start and how frequently metrics are cacluated. Below the metrics computation will start after 2 minutes on process startup and thereafter every 1 minute.
runtime policy:
frequency: 1
startup delay: 2
time unit: MINUTES
Attribute | Description | Data Type | Required |
|---|---|---|---|
frequency | Frequency metrics are computed. Minimum 15 second computation cycles | Long Default: 1 | |
startup delay | First compute cycle delay. | Long Default: 5 | |
time unit | Time unit used to set the scheduled processing policy Supported units: SECONDS, MINUTES, HOURS | TimeUnit Default: MINUTES |
This section configures a list of metrics to be computed. One or metric families can be configured. Each metric configuration has there own management policies
foreach metric compute:
metrics:
- name: BidMovingAverage
metric key: symbol
table definition: standardQuoteAnalyticsStream.BidMovingAverage (symbol VARCHAR, avg_bid_min FLOAT, avg_bid_avg FLOAT,avg_bid_max FLOAT)
query:
SELECT symbol,
MIN(bid) AS 'avg_bid_min',
AVG(bid) AS 'avg_bid_avg',
MAX(bid) AS 'avg_bid_max'
FROM standardQuoteAnalyticsStream.quote
WHERE
ingestTime >= date_trunc('minutes',now() - INTERVAL 2 MINUTES) AND ingestTime <= date_trunc('minutes',now())
GROUP BY symbol
ORDER BY 1;
truncate on start: true
compaction policy:
frequency: 8
time unit: HOURS
Attribute | Description | Data Type | Required |
|---|---|---|---|
name | Unique metric name | String | |
metric key | Key for the metric | String | |
query | Metrics query | SQL query | |
table definition | SQL table for the resulting metrics | SQL table definition |
Each metric defintiion includes management attributes to control how the metric table is managed throughout the Joule process lifetime. Since Joule uses an in-memory database the size of table needs to be managed
truncate on start: true
compaction policy:
frequency: 8
time unit: HOURS
Attribute | Description | Data Type | Required |
|---|---|---|---|
truncate on start | Truncate the mtric tables on startup | Boolean Default: true | |
compaction policy | | | |
frequency | Frequency table compactions will occur | long Default: 1 | |
time unit | Time unit used to set the scheduled companion policy Supported units: MINUTES and HOURS | TimeUnit Default: HOURS |
Metrics can be presented on the final output stage within the
emit select statement using a well formed metric query structure.Users defined a metric query within the select statement with the results being added to the output event. The below example performs a metric lookup based upon the passed
symbol value. emit:
select: "symbol, BidMovingAverage.avg_bid_max;WHERE symbol=${symbol} 'avg_bid_max'"
The declarative query structure is formed of three required components and an optional override output alias.
<metric-family>.<metrics>;<predicate> '<output-alias>'
This is the logical name of the group of metrics, which should correspond to a metric name declared within the compute section.
A comma delimited list of metrics or asterisk. Any defined metric needs to correspond the query projection attributes.
Matching and filtering statement using standard SQL. This is used to extract only those records that fulfill a specified condition.
-- Simple lookup using a
WHERE symbol='IBM'
-- Parameter replace using the corresponding the event object symbol value
WHERE symbol=${symbol}
-- Apply additonal filtering criteria
WHERE symbol=${symbol} AND avg_bid_max > 120.88
-- Get all metrics from the BidMovingAverage metrics family for IBM
BidMovingAverage.*;WHERE symbol='IBM'
-- Get the avg_bid_max metric from the BidMovingAverage metrics family all symbols matching the reference ${symbol} variable
BidMovingAverage.avg_bid_max;WHERE symbol=${symbol}
-- Same as above with the ability to define a custom name for the resulting field
BidMovingAverage.avg_bid_max;WHERE symbol=${symbol} 'avg_bid_max'
-- Get the avg_bid_max and avg_bid_min metric from the BidMovingAverage metrics family
BidMovingAverage.avg_bid_max,avg_bid_min;WHERE symbol=${symbol}
Metrics can be used within custom processors my using the Metric Query API, see API documentation for details on how to leverage this capability.
Last modified 8mo ago