Joule
Search
⌃K

Key Features

Build, deploy and scale use cases fast

Modern use case development platform

Joule has been developed to leverage mature and emerging technologies through the application of clean interfaces via the Joule SDK.

Low Code

At its core, Joule adheres to the design principle of delivering a low-code use case platform that fosters rapid development iterations for impactful business outcomes. Packaged with a dedicated use case language, DSL, and a suite of reusable assets, Joule empowers developers to commence building immediately after installation.

Key language elements

  • Flexible event subscriptions and publishing
  • Stream event processing pipeline
  • Custom SQL Metrics definition
  • Extendability through custom components
  • Mainstream product integrations

Example

stream:
name: standardQuoteAnalyticsStream
enabled: true
validFrom: 2020-01-01
validTo: 2025-12-31
eventTimeType: EVENT_TIME
sources: [ nasdaq_quotes_stream ]
​
initialisation:
sql import:
schema: standardQuoteAnalyticsStream
parquet:
-
table: quote
asView: false
files: ['quotes*.parquet']
index:
fields: ['symbol']
unique: false
​
processing unit:
metrics engine:
runtime policy:
frequency: 1
startup delay: 2
time unit: MINUTES
​
foreach metric compute:
metrics:
-
name: BidMovingAverage
metric key: symbol
table definition: standardQuoteAnalyticsStream.BidMovingAverage (symbol VARCHAR, avg_bid_min FLOAT, avg_bid_avg FLOAT,avg_bid_max FLOAT)
query:
SELECT symbol,
MIN(bid) AS 'avg_bid_min',
AVG(bid) AS 'avg_bid_avg',
MAX(bid) AS 'avg_bid_max'
FROM standardQuoteAnalyticsStream.quote
WHERE
ingestTime >= date_trunc('minutes',now() - INTERVAL 2 MINUTES) AND ingestTime <= date_trunc('minutes',now())
GROUP BY symbol
ORDER BY 1;
truncate on start: true
compaction policy:
frequency: 8
time unit: HOURS
​
pipeline:
- filter:
expression: "symbol != 'A'"
​
emit:
select: "symbol, BidMovingAverage.avg_bid_max;WHERE symbol=${symbol} 'avg_bid_max'"
​
group by:
- symbol
​

Stream Processing

Event processing is executed through the definition of a processor's pipeline. Events undergo sequential processing utilising a micro-batch methodology, a technique employed to boost processing throughput while optimising the utilisation of underlying hardware capabilities.
​
Filtering
Filtering event based using a configurable criteria or a an expression. Example use cases is customer opt-out, missing data elements, out of range etc,.
​
Enrichment
Enrichment of streaming events with a embedded low-latency data caching solution
​
Transformation
Event field tokenisation, encryption, masking, bucketing and redaction
​
Triggers
Real-time alerts and event triggers using rule based processing and delta CDC processing
​
Event Tap
Tap events directly in to an in-memory database to enable on/off line processing
​
Scripting
Execute external scripts or defined expression within the use case DSL. using supported scripting languages ( Node.js, JavaScript, Python)
​
New processors are constantly added to the platform. Please contact fractalworks for an updated list.

Analytics

Joule provides three flexible methods to build analytical insights. Each method is describe below.
Analytics
Machine Learning
Metrics Engine
The integration of streaming analytics serves as a pivotal feature, empowering the evolution of sophisticated use case development, including applications like geospatial analytics for marketing, business analytics, and feature preparation for machine learning predictions.

Key Features

  • Tumbling and sliding windows
  • Standard statistical functions
  • Custom analytic functions
  • Geospatial analytics (Geo Tracker, Geofence occupancy, spatial index)

Example

time window:
emitting type: tumblingQuoteAnalytics
aggregate functions:
FIRST: [ask]
LAST: [ ask ]
MIN: [ ask ]
MAX: [ bid ]
SUM: [ volume ]
MEAN: [ volatility ]
VARIANCE: [ volatility ]
STDEV: [ volatility ]
policy:
type: tumblingTime
window size: 5000
Utilising machine learning models enables the delivery of streaming predictions, offering versatility for applications in both start-of-day batch processing and real-time prediction services to bolster intra-day business operations.

Key Features

  • Predictive Model Markup Language (PMML)
  • In-place model refresh / replacement
  • Model versioning control
  • Prediction and model auditing
  • Feature engineering (binning, scaling, log transform, and scripting)

Example

pmml predictor:
name: irisScorer
model: ./models/iris_rf.pmml
response field: flowerPrediction
audit configuration:
target schema: ml_audit
queue capacity: 5000
flush frequency: 300
Joule offers a comprehensive global metrics engine designed to facilitate the development of intricate use cases. Developers can articulate metrics using standard ANSI SQL and specify their execution frequency using the Joule DSL. These defined metrics are seamlessly integrable into the defined streaming pipeline.

Key Features

  • ANSI SQL based metrics
  • Runtime policy to schedule metric generation
  • Metrics accessible within pipeline processes and projection
  • Export RestAPI
  • In-memory column database

Example

metrics engine:
runtime policy:
frequency: 1
startup delay: 2
time unit: MINUTES
​
foreach metric compute:
metrics:
-
name: BidMovingAverage
metric key: symbol
table definition: standardQuoteAnalyticsStream.BidMovingAverage (symbol VARCHAR, avg_bid_min FLOAT, avg_bid_avg FLOAT,avg_bid_max FLOAT)
query:
SELECT symbol,
MIN(bid) AS 'avg_bid_min',
AVG(bid) AS 'avg_bid_avg',
MAX(bid) AS 'avg_bid_max'
FROM standardQuoteAnalyticsStream.quote
WHERE
ingestTime >= date_trunc('minutes',now() - INTERVAL 2 MINUTES) AND ingestTime <= date_trunc('minutes',now())
GROUP BY symbol
ORDER BY 1;
truncate on start: true
compaction policy:
frequency: 8
time unit: HOURS
​

Observability

Each processing component in Joule furnishes a standard set of metrics, offering users insights into he number of events received, processed, discarded, and failed. Furthermore, with the SQL engine enabled, both raw and processed events are stored, making them queryable and exportable for enhanced analytical capabilities.

Integrations

Straight out of the box, Joule provides a standardised set of data integrations, facilitating a quick start to data consumption and publication processes.

Flexible Deployment

Joule has been designed to be platform agnostic, offering seamless deployment options whether you choose a local, on-premise, or cloud-based environment. Joule is packaged as a Docker container for simplified deployment configurations or as a standalone binary, providing flexibility to meet diverse deployment needs.

SDK

A Java SDK for developers is supplied to extend platform capabilities, enabling the customisation and enhancement of processors and data transports.
​