The metrics engine can be leveraged to provide greater insights over a larger data set. These metrics can be used within custom processors to drive complex use case
This use case provides a simple demonstration on how metrics generated by the internal metrics engine can be emitted to connected publisher sinks using the select projection process.
The pipeline configures the metrics computation cycle of 1 minute, metrics to be generated, filtering and the select expression which emits the group by metrics to the connected publishing sink.
stream:name:standardQuoteAnalyticsStreamenabled:trueeventTimeType:EVENT_TIMEsources: [ nasdaq_quotes_stream ]processing unit:metrics engine:runtime policy:frequency:1startup delay:1time unit:MINUTESforeach metric compute:metrics: -name:BidMovingAveragemetric key:symboltable definition:bid_moving_averages(symbol VARCHAR, avg_bid_min FLOAT,avg_bid_avg FLOAT,avg_bid_max FLOAT,createdTimestamp TIMESTAMP)query:SELECT symbol,MIN(bid) AS 'avg_bid_min',AVG(bid) AS 'avg_bid_avg',MAX(bid) AS 'avg_bid_max'FROM quotes.nasdaqWHEREingestTime >= epoch_ms(date_trunc('minutes',now() - INTERVAL 3 MINUTES)) AND ingestTime <= epoch_ms(now())GROUP BY symbolORDER BY 1;truncate on start:truecompaction policy:frequency:8time unit:HOURSpipeline: - tap:target schema:quotesflush frequency:5index:unique:falsefields: - symbolemit:select:"symbol, BidMovingAverage.avg_bid_max;WHERE symbol=${symbol} 'avg_bid_max'"group by: - symbol