Sources

Connect to key event and file data sources

Overview

Sources provide a flexible data processing framework, enabling users to quickly set up and run use cases by transforming raw data into a structured format suitable for pipeline processing.

With built-in support for multiple data ingestion methods, Joule allows you to efficiently bring in data from various sources, converting it into StreamEvent objects that flow through the platform until the publishing stage.

Fractalworks has also developed ready-to-use connectors to speed up development.

Build your own connectors using the Connector API and quick start templates

Available connector options

Incoming events are deserialised into StreamEvents objects for pipeline processing. This can be done automatically or via a custom transformer, depending on your needs.

Supported ingestion methods include:

Streaming

These guides provide configurations for Joule integrations using Kafka, RabbitMQ and MQTT. Enable flexible data ingestion through standard messaging platforms.

Kafka

Standard Kafka consumers transport ingests data from subscribed cluster topics

RabbitMQ

AMQP messaging solution ideal for IoT and client/server use cases

MQTT

Lightweight messaging protocol ideal for IoT use cases

HTTP

This guide outlines two endpoint types for Joule integration: a file-based endpoint for efficient batch processing of large files and a stream-based endpoint for real-time chaining of Joule processes to streamline complex workflows.

Rest endpoints

RESTful consumer endpoints designed to facilitate seamless integration

File

These guides covers high-performance file processing using MinIO's S3-compatible storage and event-driven batch processing with an automated file watcher.

MinIO S3

MinIO file consumer using S3 object level access to cloud or local hosted buckets

File watcher

Process large event files using stream processing