Rest endpoints
RESTful consumer endpoints designed to facilitate seamless integration
Overview
This page outlines Joule's RESTful consumer endpoints, which enable flexible integration for rapid use case development and validation. Joule supports two endpoint types: file-based and stream-based.
File-based POST endpoint Is optimised for handling large files, which are directed to a specified download directory and then moved to a processed directory after ingestion, with support for multiple file formats (i.e., PARQUET, JSON). This bridges the gap of batch to stream processing.
Stream-based POST endpoint
Allows developers to chain Joule processes similarly to micro-services, supporting complex workflows and real-time data processing. Events are passed as StreamEvent
objects through the POST body or an events
parameter, enabling streamlined data flow between Joule processes.
Client library: io.javalin:javalin:5.6.3
Examples & DSL attributes
This example configures a REST File Consumer to consume PARQUET files from the nasdaq_quotes_file
source.
The files are downloaded from the quotes
topic and saved in the nasdaq/downloads
directory. After the files are fully received via the file-based POST endpoint, they are processed efficiently.
Once processed, the files are moved to a local processed
directory, with a timestamp marking their completion.
Attributes schema
Supported file types
PARQUET
ARROW_IPC
ORC
CSV
JSON
Stream configuration guide
The stream-based POST endpoint allows developers to link Joule processes together, similar to microservices, enabling the creation of complex workflows.
This endpoint requires a StreamEvent
object or an array of these events.
Example stream based configuration
Events can be sent either through:
The
events
parameter, which includes specific event details in the request orWithin the POST body, where the event information is contained in the HTTP POST request body.
Core attributes schema
Currently only the Joule StreamEvent
is supported
Joule properties DSL
This table outlines configurable properties in Joule for managing file uploads with the REST API, especially useful for handling large files.
These settings allow fine-tuning of file handling capabilities, helping to manage memory use and ensure smooth processing of large file uploads.
Last updated