# Unified execution engine

## What will we learn in this article?

This article explores the unified execution engine in data processing, covering how it **addresses challenges** in real-time and batch data processing within a single, integrated execution engine that is Joule.

We will gain insight into Joule’s architecture and how it **manages continuous and periodic** data without distinction between batch and streaming. 

Joule solves how a unified approach overcomes issues with traditional data processing frameworks by eliminating the requirement for data to be *complete* at time of ingestion by introducing components like [stream joins](https://docs.fractalworks.io/joule/components/processors/stream-join), [filters](https://docs.fractalworks.io/joule/components/processors/filters), [enrichments](https://docs.fractalworks.io/joule/components/processors/enrichment), [windows](https://docs.fractalworks.io/joule/components/analytics/analytic-tools/window-analytics) and [transformation](https://docs.fractalworks.io/joule/components/processors/transformation).

## What is a unified execution engine?

A unified execution engine processes both unbounded (continuous) and bounded (finite) data without requiring developers to differentiate between them. Because we will not cover what unbounded and bounded data is, [this article gives an overview](https://www.thomashenson.com/bound-vs-unbound-data-in-real-time-analytics/) of the concepts. 

Ideally we would like to have insights as soon as the initial event has occurred. However, this is entirely dependent upon the frequency the data is presented to the processing engine and the actual use case needs. 

Low latency event feeds generally produce a faster time to insight whereas bounded data deliver snapshots that reflect point in time view. Therefore when processing both types of data we need to balance the expectation of an ideal state versus the actual requirements of reality.

<mark style="color:green;">**In an ideal world**</mark>

> Processing = event creation

Processing would happen instantly as **events are created**.

<mark style="color:green;">**In reality**</mark>

> Processing ≠ event creation

Event processing **must wait for new events** to enter the pipeline before it can start generating actions to act upon.

{% hint style="success" %}
Joule **does not differentiate** between bounded and unbounded data
{% endhint %}

### Adapting to continuous data with a flexible unified model

Unlike traditional data processing frameworks, which assume data will eventually become complete; a unified model **operates on the assumption** that new data may always arrive.

This approach enables flexibility by **not tying data infrastructure** to specific execution engines and by providing **consistency across both unbounded and bounded datasets**.

Joule's processors would then allow developers to specify **when to emit the output results** for a given period of time, enabling responsive processing even in continuous workflows.

i.e., with Joule, you can spin up **multiple processors**, each with its own scheduler, to run independently within the same environment. Unlike traditional setups where a single scheduler manages all tasks, Joule enables **separate, decoupled processing** for different use cases.

## Understanding data processing with Joule

{% hint style="info" %}
Joule **does not differentiate** between batch or stream processing\
\
|For a more in-depth description of how Joule treats data processing please follow[batch-and-stream-processing](https://docs.fractalworks.io/joule/concepts/batch-and-stream-processing "mention").
{% endhint %}

### Batch processing

Batch processing in a unified engine handles large, finite datasets processed periodically. Joule manages batch data by **applying a micro batching** method which appears as a stream of events internally.

Joule uses the latest [Apache Arrow](https://arrow.apache.org/docs/) processing techniques to enable large file handling while managing memory efficiently.&#x20;

With unified execution, **batch jobs can also include real-time triggers**, allowing them to operate seamlessly **alongside streaming data**.

### Stream processing

Stream processing operates on unbounded datasets, offering near-real-time analytics as new data continuously arrives. Joule uses [DuckDB](https://duckdb.org/) as an internal data storage, enabling **high-performance analytics**.

This approach **allows streaming data to flow without waiting for all data to arrive**, overcoming delays caused by continuous event inflow.

## How does this work in Joule?

Joule **treats** batch and streaming **data consistently**. Allowing seamless processing across different data sources and formats.

Joule’s architecture supports modular pipelines, real-time observability and extensibility. This makes Joule adaptable to diverse data processing demands.

<table data-view="cards"><thead><tr><th></th><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><mark style="color:orange;"><strong>Processors</strong></mark></td><td>Processors are the core of the Joule platform, each performing a specific task. These create use case when linked together</td><td></td><td><a href="../components/processors">processors</a></td></tr><tr><td><mark style="color:orange;"><strong>Analytics</strong></mark></td><td>Analytics form the core platform feature that facilitates insight to value</td><td></td><td><a href="../components/analytics">analytics</a></td></tr><tr><td><mark style="color:orange;"><strong>Connectors</strong></mark></td><td>Integrate to external systems to consume events and publish insights</td><td></td><td><a href="../components/connectors">connectors</a></td></tr></tbody></table>

### Unified engine architecture

<figure><img src="https://3062398388-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FUU6FZlV07ZD90OzbzGww%2Fuploads%2Frc8DQZ0Lj9us9fUmeRT8%2Fimage.png?alt=media&#x26;token=53e426e4-fa48-4803-b0fb-5991d020216e" alt=""><figcaption><p>Unified execution engine being Joule</p></figcaption></figure>