# Sources

## Overview

Sources provide a flexible data processing framework, enabling users to **quickly set up and run use cases** by transforming raw data into a structured format suitable for pipeline processing.

With built-in support for multiple data ingestion methods, Joule allows you to efficiently bring in data from various sources, converting it into `StreamEvent` objects that **flow through the platform** until the publishing stage.

[Fractalworks](https://www.fractalworks.io/) has also developed **ready-to-use connectors** to speed up development.

{% hint style="info" %}
Build your own connectors using the [**Connector API**](https://docs.fractalworks.io/joule/developer-guides/builder-sdk/connector-api) and quick start templates
{% endhint %}

## Available connector options

Incoming events are deserialised into `StreamEvents` objects for pipeline processing. This can be done automatically or via a custom transformer, depending on your needs.

Supported ingestion methods include:

{% tabs %}
{% tab title="Streaming" %}
These guides provide configurations for Joule integrations using Kafka, RabbitMQ and MQTT. Enable flexible data ingestion through standard messaging platforms. 

<table data-card-size="large" data-view="cards"><thead><tr><th></th><th></th><th data-hidden></th><th data-hidden data-card-cover data-type="files"></th><th data-hidden data-card-target data-type="content-ref"></th><th data-hidden data-type="rating" data-max="5"></th><th data-hidden></th><th data-hidden></th></tr></thead><tbody><tr><td><mark style="color:orange;"><strong>Kafka</strong></mark></td><td><p>Standard Kafka consumers transport ingests data from subscribed cluster topics</p><p></p></td><td></td><td></td><td><a href="sources/kafka">kafka</a></td><td>null</td><td></td><td></td></tr><tr><td><mark style="color:orange;"><strong>RabbitMQ</strong></mark></td><td>AMQP messaging solution ideal for IoT and client/server use cases</td><td></td><td></td><td><a href="sources/rabbitmq">rabbitmq</a></td><td>null</td><td></td><td></td></tr><tr><td><mark style="color:orange;"><strong>MQTT</strong></mark></td><td><p>Lightweight messaging protocol ideal for IoT use cases</p><p></p></td><td></td><td></td><td><a href="sources/mqtt">mqtt</a></td><td>null</td><td></td><td></td></tr></tbody></table>
{% endtab %}

{% tab title="HTTP" %}
This guide outlines two endpoint types for Joule integration: a file-based endpoint for efficient batch processing of large files and a stream-based endpoint for real-time chaining of Joule processes to streamline complex workflows.

<table data-card-size="large" data-view="cards"><thead><tr><th></th><th></th><th data-hidden></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><mark style="color:orange;"><strong>Rest endpoints</strong></mark></td><td>RESTful consumer endpoints designed to facilitate seamless integration</td><td></td><td><a href="sources/rest-endpoints">rest-endpoints</a></td></tr></tbody></table>
{% endtab %}

{% tab title="File" %}
These guides covers high-performance file processing using MinIO's S3-compatible storage and event-driven batch processing with an automated file watcher.

<table data-card-size="large" data-view="cards"><thead><tr><th></th><th></th><th data-hidden></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><mark style="color:orange;"><strong>MinIO S3</strong></mark></td><td>MinIO file consumer using S3 object level access to cloud or local hosted buckets </td><td></td><td><a href="sources/minio-s3">minio-s3</a></td></tr><tr><td><mark style="color:orange;"><strong>File watcher</strong></mark></td><td>Process large event files using stream processing</td><td></td><td><a href="sources/file-watcher">file-watcher</a></td></tr></tbody></table>
{% endtab %}
{% endtabs %}