# Sources

## Overview

Sources provide a flexible data processing framework, enabling users to **quickly set up and run use cases** by transforming raw data into a structured format suitable for pipeline processing.

With built-in support for multiple data ingestion methods, Joule allows you to efficiently bring in data from various sources, converting it into `StreamEvent` objects that **flow through the platform** until the publishing stage.

[Fractalworks](https://www.fractalworks.io/) has also developed **ready-to-use connectors** to speed up development.

{% hint style="info" %}
Build your own connectors using the [**Connector API**](/joule/developer-guides/builder-sdk/connector-api.md) and quick start templates
{% endhint %}

## Available connector options

Incoming events are deserialised into `StreamEvents` objects for pipeline processing. This can be done automatically or via a custom transformer, depending on your needs.

Supported ingestion methods include:

{% tabs %}
{% tab title="Streaming" %}
These guides provide configurations for Joule integrations using Kafka, RabbitMQ and MQTT. Enable flexible data ingestion through standard messaging platforms. 

<table data-card-size="large" data-view="cards"><thead><tr><th></th><th></th><th data-hidden></th><th data-hidden data-card-cover data-type="files"></th><th data-hidden data-card-target data-type="content-ref"></th><th data-hidden data-type="rating" data-max="5"></th><th data-hidden></th><th data-hidden></th></tr></thead><tbody><tr><td><mark style="color:orange;"><strong>Kafka</strong></mark></td><td><p>Standard Kafka consumers transport ingests data from subscribed cluster topics</p><p></p></td><td></td><td></td><td><a href="/pages/g5TeMEOuFj6mgCNOvTn7">/pages/g5TeMEOuFj6mgCNOvTn7</a></td><td>null</td><td></td><td></td></tr><tr><td><mark style="color:orange;"><strong>RabbitMQ</strong></mark></td><td>AMQP messaging solution ideal for IoT and client/server use cases</td><td></td><td></td><td><a href="/pages/s4v5Tymvk7LN5G34Bidb">/pages/s4v5Tymvk7LN5G34Bidb</a></td><td>null</td><td></td><td></td></tr><tr><td><mark style="color:orange;"><strong>MQTT</strong></mark></td><td><p>Lightweight messaging protocol ideal for IoT use cases</p><p></p></td><td></td><td></td><td><a href="/pages/bfvsdDxfB6IwCZuaxLZ9">/pages/bfvsdDxfB6IwCZuaxLZ9</a></td><td>null</td><td></td><td></td></tr></tbody></table>
{% endtab %}

{% tab title="HTTP" %}
This guide outlines two endpoint types for Joule integration: a file-based endpoint for efficient batch processing of large files and a stream-based endpoint for real-time chaining of Joule processes to streamline complex workflows.

<table data-card-size="large" data-view="cards"><thead><tr><th></th><th></th><th data-hidden></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><mark style="color:orange;"><strong>Rest endpoints</strong></mark></td><td>RESTful consumer endpoints designed to facilitate seamless integration</td><td></td><td><a href="/pages/ydyxcKmpbswnrAiFjhL2">/pages/ydyxcKmpbswnrAiFjhL2</a></td></tr></tbody></table>
{% endtab %}

{% tab title="File" %}
These guides covers high-performance file processing using MinIO's S3-compatible storage and event-driven batch processing with an automated file watcher.

<table data-card-size="large" data-view="cards"><thead><tr><th></th><th></th><th data-hidden></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><mark style="color:orange;"><strong>MinIO S3</strong></mark></td><td>MinIO file consumer using S3 object level access to cloud or local hosted buckets </td><td></td><td><a href="/pages/zQyiXSTFGWT9PQr9Irst">/pages/zQyiXSTFGWT9PQr9Irst</a></td></tr><tr><td><mark style="color:orange;"><strong>File watcher</strong></mark></td><td>Process large event files using stream processing</td><td></td><td><a href="/pages/wV996jVkPHvNeSeazS4H">/pages/wV996jVkPHvNeSeazS4H</a></td></tr></tbody></table>
{% endtab %}
{% endtabs %}


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.fractalworks.io/joule/components/connectors/sources.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.