Sinks

Distribute insights to consuming systems

Overview

Data sinks in Joule enable integration with various downstream systems, enhancing the application’s ability to support additional business needs.

By connecting to multiple types of systems, Joule allows users to route processed data efficiently to databases, messaging platforms, cloud storage and more.

These integrations empower users to leverage Joule's data processing across different environments and tools, creating more comprehensive, scalable solutions.

Available options

Streaming

These guides provide configurations for Joule integrations using Kafka, RabbitMQ and MQTT. Enable flexible data streaming through standard messaging platforms.

Kafka

Sends events to specified Kafka topics, enabling real-time data streams

RabbitMQ

RabbitMQ is lightweight and easy to deploy messaging platform for event-based data integration

MQTT

Allows publishing events through MQTT, ideal for IoT and lightweight streaming

Databases

These guides provide configurations for Joule integrations with SQL databases: InfluxDB, MongoDB and Geode. Enable efficient data storage and retrieval across various database types.

SQL databases

Write events in micro batches to any database which supports JDBC Type 4 drivers

InfluxDB

Standard time-series database idea for instrumentation or live KPI tracking

MongoDB

Publish JSON based events to the mature standard document storePublishes JSON-based events to MongoDB, a widely-used document store

Geode

Integrates with distributed caching for complex use cases needing high availability

HTTP

This guide provide configurations for Joule integrations using web sockets. Enable real-time data delivery to web-based clients for interactive applications.

WebSockets endpoint

Publishes processed events to web clients via WebSocket transport, ideal for real-time dashboards and applications

File

These guides provide configurations for Joule integrations with MinIO S3 and direct file storage. Enable flexible event archiving through cloud or local storage solutions.

MinIO S3

Produces files to MinIO S3-compatible storage, whether cloud or locally hosted

File transport

Write events directly to files in Joule-supported formats for archiving or later processing

Transformation

Joule connectors send data as StreamEvents, which can be transformed into various output formats.

The currently supported formats include:

1. CSV A standard format for spreadsheets and data analysis tools.

2. JSON Common for web applications and lightweight data exchanges.

3. Parquet Optimised for large-scale data processing and analytics.

4. AVRO Suitable for row-based storage and schema evolution.

For additional customisation, the Joule SDK provides interfaces to develop custom formatters. Refer to the Joule SDK documentation for further guidance.