> For the complete documentation index, see [llms.txt](https://docs.fractalworks.io/joule/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.fractalworks.io/joule/components/contextual-data/architecture.md).

# Architecture

## Overview

Joule’s data source interface allows processors to **access and utilise** contextual data seamlessly across the platform.

Upon startup, the Joule runtime **connects to each configured data source**, assigning each source a logical name and making it available to all processors.

This setup is managed through a **single configuration file**, which specifies the necessary contextual data stores ([see Configuration section](/joule/components/contextual-data/configuration.md)).

## Architecture

To maintain high performance in scenarios with **large event throughputs**, it is essential to choose the right data source implementation.

{% hint style="warning" %}
Poorly optimised data sources can hinder the **efficiency of the processing pipeline**, particularly during high-frequency data access
{% endhint %}

To **mitigate performance issues and reduce the load** from out-of-process I/O operations, Joule recommends caching contextual data locally within the process, especially in high-read environments.

This approach **optimises data retrieval speed and enhances the overall processing efficiency**, enabling smoother handling of real-time events.

### High level architecture

The Joule solution enables external tools to plugin to the Joule processing using the **Reference Data Interface**. 

<figure><img src="/files/oxvgEWgEadcEbu4dyrXJ" alt=""><figcaption><p>Streaming Prediction example</p></figcaption></figure>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.fractalworks.io/joule/components/contextual-data/architecture.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.