# Key features

## Modern use case development platform

As a user of Joule, your main focus will be **building use cases** that address point-in-time or strategic business needs - there is no need to continually rebuild technology assets. **Just reuse existing** components to kick start the process and add new assets when needed.

Joule supports turning your needs in to deployable runtime assets.​

<figure><img src="https://3062398388-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FUU6FZlV07ZD90OzbzGww%2Fuploads%2FI5ffz6gMA8FSUFRVypUH%2FJoule%20-%20Architecture.jpg?alt=media&#x26;token=a4277f2d-020b-4062-a3a3-e469cfdea999" alt=""><figcaption><p>Joule high level architecture</p></figcaption></figure>

### Key features include

1. <mark style="color:green;">**Low code development platform**</mark>\
   Effortlessly create use cases with Joule DSL, enabling swift development of versatile stream processing pipelines.
2. <mark style="color:green;">**Analytics enablement**</mark>\
   Harness ML model support, auditing, geospatial capabilities, streaming window analytics, SQL metrics engine and triggers for enhanced functionality.
3. <mark style="color:green;">**Stream processors**</mark>\
   Enrich, encrypt and mask data effortlessly with out-of-the-box processors and supported scripting languages (Node.js, JavaScript, Python).
4. <mark style="color:green;">**Data connectors**</mark>\
   Effortlessly consume and stream consistent data with out-of-the-box connectors, including Kafka, MQTT, data lakes, NoSQL, web sockets, OpenAPI and more.
5. <mark style="color:green;">**Contextual data**</mark>\
   Enrich streaming events with dynamic, static and slow-moving contextual data using embedded local caching.
6. <mark style="color:green;">**APIs & SDK**</mark>\
   An extendable API is available to empower developers in building custom components.
7. <mark style="color:green;">**Observability**</mark>\
   All components have processing metric counters which can be monitored using external solutions
8. <mark style="color:green;">**Flexible deployment**</mark>\
   Joule has been designed to be platform agnostic, offering seamless deployment options whether you choose a local, on-premise or cloud-based environment.

In the following section, we will further explore each feature.

***

## Low code

At its core, Joule adheres to the design principle of delivering a low-code use case platform that **fosters rapid development iterations** for impactful business outcomes.

Packaged with a dedicated use case language and a suite of reusable assets Joule empowers developers to **commence building immediately after installation**.

### Key language elements

* Flexible event subscriptions and publishing&#x20;
* Stream event processing pipeline
* Custom SQL Metrics definition&#x20;
* Extendability through custom components
* Mainstream product integrations

### Example

{% code fullWidth="false" %}

```yaml
stream:
  name: standardQuoteAnalyticsStream
  enabled: true
  eventTimeType: EVENT_TIME
  sources: [ nasdaq_quotes_stream ]

  initialisation:
    sql import:
      schema: standardQuoteAnalyticsStream
      parquet:
        - table: quote
          asView: false
          files: ['quotes*.parquet']
          index:
            fields: ['symbol']
            unique: false

  processing unit:
    metrics engine:
      runtime policy:
        frequency: 1
        startup delay: 2
        time unit: MINUTES

      foreach metric compute:
        metrics:
          - name: BidMovingAverage
            metric key: symbol
            table definition: standardQuoteAnalyticsStream.BidMovingAverage (symbol VARCHAR, avg_bid_min FLOAT, avg_bid_avg FLOAT,avg_bid_max FLOAT)
            query:
              SELECT symbol,
              MIN(bid) AS 'avg_bid_min',
              AVG(bid) AS 'avg_bid_avg',
              MAX(bid) AS 'avg_bid_max'
              FROM standardQuoteAnalyticsStream.quote
              WHERE
              ingestTime >= date_trunc('minutes',now() - INTERVAL 2 MINUTES) AND ingestTime <= date_trunc('minutes',now())
              GROUP BY symbol
              ORDER BY 1;
            truncate on start: true
            compaction policy:
              frequency: 8
              time unit: HOURS

    pipeline:
      - filter:
          expression: "symbol != 'A'"

  emit:
    select: "symbol, BidMovingAverage.avg_bid_max;WHERE symbol=${symbol} 'avg_bid_max'"

  group by:
    - symbol
```

{% endcode %}

## Analytics enablement

Joule provides three flexible methods to build analytical insights. Each method is describe below.&#x20;

{% tabs %}
{% tab title="Analytics" %}
The integration of streaming analytics serves as a pivotal feature, empowering the evolution of sophisticated use case development, including applications like geospatial analytics for marketing, business analytics and feature preparation for machine learning predictions.

### Key features

* Tumbling and sliding windows
* Standard statistical functions
* Custom analytic functions&#x20;
* Geospatial analytics (Geo Tracker, Geofence occupancy, spatial index)

### Example

```yaml
time window:
  emitting type: tumblingQuoteAnalytics
  aggregate functions:
    FIRST: [ask]
    LAST: [ ask ]
    MIN: [ ask ]
    MAX: [ bid ]
    SUM: [ volume ]
    MEAN: [ volatility ]
    VARIANCE: [ volatility ]
    STDEV: [ volatility ]
  policy:
    type: tumblingTime
    window size: 5000
```

{% endtab %}

{% tab title="Machine learning" %}
Utilising machine learning models enables the delivery of streaming predictions, offering versatility for applications in both start-of-day batch processing and real-time prediction services to bolster intra-day business operations.

### Key features

* Predictive Model Markup Language (PMML)&#x20;
* In-place model refresh / replacement&#x20;
* Model versioning control
* Prediction and model auditing&#x20;
* Feature engineering (binning, scaling, log transform, and scripting)

### Example

```yaml
pmml predictor:
  name: irisScorer
  model: ./models/iris_rf.pmml
  response field: flowerPrediction
  
  audit configuration:
    target schema: ml_audit
    queue capacity: 5000
    flush frequency: 300
```

{% endtab %}

{% tab title="Metrics engine" %}
Joule offers a comprehensive global metrics engine designed to facilitate the development of intricate use cases. Developers can articulate metrics using standard ANSI SQL and specify their execution frequency using the Joule DSL. These defined metrics are seamlessly integrable into the defined streaming pipeline.

### Key features

* ANSI SQL based metrics
* Runtime policy to schedule metric generation
* Metrics accessible within pipeline processes and projection&#x20;
* Export RestAPI &#x20;
* In-memory column database

### Example

```yaml
metrics engine:
    runtime policy:
      frequency: 1
      startup delay: 2
      time unit: MINUTES

    foreach metric compute:
      metrics:
        -
          name: BidMovingAverage
          metric key: symbol
          table definition: standardQuoteAnalyticsStream.BidMovingAverage (symbol VARCHAR, avg_bid_min FLOAT, avg_bid_avg FLOAT,avg_bid_max FLOAT)
          query:
            SELECT symbol,
            MIN(bid) AS 'avg_bid_min',
            AVG(bid) AS 'avg_bid_avg',
            MAX(bid) AS 'avg_bid_max'
            FROM standardQuoteAnalyticsStream.quote
            WHERE
            ingestTime >= date_trunc('minutes',now() - INTERVAL 2 MINUTES) AND ingestTime <= date_trunc('minutes',now())
            GROUP BY symbol
            ORDER BY 1;
          truncate on start: true
          compaction policy:
            frequency: 8
            time unit: HOURS
```

{% endtab %}
{% endtabs %}

## Stream processors

Event processing is executed through the definition of a processor's pipeline.

Events undergo **sequential processing utilising a micro-batch methodology**, a technique employed to boost processing throughput while optimising the utilisation of underlying hardware capabilities.

{% content-ref url="../components/processors" %}
[processors](https://docs.fractalworks.io/joule/components/processors)
{% endcontent-ref %}

Because Processors are a critical component, we have listed them below.

<table data-view="cards"><thead><tr><th></th><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><mark style="color:orange;"><strong>Filtering</strong></mark></td><td>Filtering event based using a configurable criteria or a an expression. Example use cases is customer opt-out, missing data elements, out of range etc</td><td></td><td><a href="../components/processors/filters">filters</a></td></tr><tr><td><mark style="color:orange;"><strong>Enrichment</strong></mark></td><td>Enrichment of streaming events with a embedded  low-latency data caching solution</td><td></td><td><a href="../components/processors/enrichment">enrichment</a></td></tr><tr><td><mark style="color:orange;"><strong>Transformation</strong></mark></td><td>Event field tokenisation, encryption, masking, bucketing and redaction </td><td></td><td><a href="../components/processors/transformation">transformation</a></td></tr><tr><td><mark style="color:orange;"><strong>Triggers</strong></mark></td><td>Real-time alerts and event triggers using rule based processing and delta CDC processing</td><td></td><td><a href="../components/processors/triggers">triggers</a></td></tr><tr><td><mark style="color:orange;"><strong>Event tap</strong></mark></td><td>Tap events directly in to an in-memory database to enable on / off line processing</td><td></td><td><a href="../components/processors/event-tap">event-tap</a></td></tr><tr><td><mark style="color:orange;"><strong>Scripting</strong></mark></td><td>Execute external scripts or defined expression within the use case DSL using supported scripting languages (Node.js, JavaScript, Python)</td><td></td><td><a href="../components/analytics/ml-inferencing/feature-engineering/scripting">scripting</a></td></tr><tr><td><mark style="color:orange;"><strong>Metrics</strong></mark></td><td>A SQL compliant metrics engine which computes scheduled metrics</td><td></td><td><a href="../components/analytics/metrics-engine">metrics-engine</a></td></tr><tr><td><mark style="color:orange;"><strong>Machine learning</strong></mark></td><td>Leverage streaming online predictions to drive insights to action</td><td></td><td><a href="../components/analytics/ml-inferencing">ml-inferencing</a></td></tr><tr><td><mark style="color:orange;"><strong>Analytics</strong></mark></td><td>Streaming analytics using event windows, expressions, scripts and much more</td><td></td><td><a href="../components/analytics/analytic-tools">analytic-tools</a></td></tr></tbody></table>

{% hint style="info" %}
New processors are constantly added to the platform. Please [contact Fractalworks](https://www.fractalworks.io/contact) for an updated list
{% endhint %}

## Data connectors

Data connectors, sources and sinks are a key component within the Joule low-code ecosystem that **consume and distribute data**.

{% content-ref url="../components/connectors" %}
[connectors](https://docs.fractalworks.io/joule/components/connectors)
{% endcontent-ref %}

## Contextual data

For Joule applications, contextual data is crucial for enabling advanced and insightful stream processing.

By seamlessly **integrating contextual data with real-time events**, the system delivers enriched processing outcomes and better informed insights.

{% content-ref url="../components/contextual-data" %}
[contextual-data](https://docs.fractalworks.io/joule/components/contextual-data)
{% endcontent-ref %}

## APIs & SDK

A Java SDK for developers is supplied to extend platform capabilities, enabling the customisation and enhancement of processors and data transports.

<mark style="color:green;">**APIs**</mark>

Rest base APIs to access key Joule functions

* Data access APIs
* Deployment management APIs

{% content-ref url="../developer-guides/api-endpoints" %}
[api-endpoints](https://docs.fractalworks.io/joule/developer-guides/api-endpoints)
{% endcontent-ref %}

<mark style="color:green;">**SDK**</mark>

Flexible SDK to enable platform extensibility&#x20;

* User defined analytics
* Processors
* Data connectors

{% content-ref url="../developer-guides/builder-sdk" %}
[builder-sdk](https://docs.fractalworks.io/joule/developer-guides/builder-sdk)
{% endcontent-ref %}

## Observability

Each processing component in Joule furnishes a standard set of metrics, **offering insights** to users into the number of events received, processed, discarded and failed.

Furthermore, with the SQL engine enabled, both raw and processed events are stored, making them queryable and exportable for enhanced analytical capabilities.

{% content-ref url="../components/observability" %}
[observability](https://docs.fractalworks.io/joule/components/observability)
{% endcontent-ref %}

## Flexible Deployment

Joule has been designed to be platform agnostic, offering seamless deployment options whether you choose a local, on-premise or cloud-based environment.

Joule is **packaged as a Docker container** for simplified deployment configurations or as a standalone binary, providing flexibility to meet diverse deployment needs.