Use cases overview

At the core of the platform is the use case. A use case is defined by user who understand the required computation and output criteria needed by the business

Use case example

Below is a fully use case that performs a basic window aggregation computation using a metrics engine and tumbling window before publishing a filter stream of events to a connected publisher.

  stream:
    name: tumblingWindowQuoteStream
    eventTimeType: EVENT_TIME
    sources:
      - nasdaq_quotes_stream
  
    initialisation:
      sql import:
        schema: fxrates
        parquet:
          - table: quote
            asView: false
            files: [ 'fxrates.parquet' ]
            drop table: true
            index:
              fields: [ 'ccy' ]
              unique: false
              
    processing unit:
      metrics engine:
          policy:
            timeUnit: MINUTES
            frequency: 1
            startup delay: 2
    
          foreach metric compute:
            metrics:
              - name: BidMovingAverage
                metric key: symbol
                table definition: standardQuoteAnalyticsStream.BidMovingAverage (symbol VARCHAR, avg_bid_min FLOAT, avg_bid_avg FLOAT,avg_bid_max FLOAT)                          
                query:
                  SELECT symbol, 
                  AVG(bid_MAX) AS mt_avg_bid_min,
                  AVG(bid_MAX) AS mt_avg_bid_max
                  OVER (
                  PARTITION BY symbol ORDER BY eventTime ASC
                  RANGE BETWEEN INTERVAL 1 SECONDS PRECEDING
                  AND INTERVAL 1 SECONDS FOLLOWING)
                  AS 'bid max Moving Average'
                  FROM quotesStream.tumblingQuoteAnalytics_View
                  ORDER BY 1;
                truncate on start: true
                compaction:
                  frequency: 8
                  timeUnit: HOURS
                
      pipeline:
        - filter:
            expression: "symbol != 'A'"
        - timeWindow:
            emittingEventType: tumblingQuoteAnalytics
            aggregateFunctions:
              MIN: [ask, bid]
              MAX: [ask, bid]
            policy:
              type: tumblingTime
              windowSize: 5000
  
    emit:
      eventType: windowQuoteEvent
      select: "symbol, ask_MIN, ask_MAX, bid_MIN, bid_MAX, BidMovingAverage.avg_bid_max;WHERE symbol=${symbol}"
      having: "symbol !='A'"
  
    group by:
      - symbol

DSL Elements

Initialisation

Joule provide the ability to prime the system with static reference data at startup. Read the initialisation documentation for further information.

Procession Unit

The main objective of the platform was to build use cases easliy and clearly. To understand how to build use case read the processing unit documentation.

Emit

Ability to select required output feilds and to perform a final filter before publishing events to downstream systems. Read the emit documentation for further information.

Group by

Provides the ability to group similair events to perform a computation in order to reduce the overall downstream output. Read the group by documentation for more information.