Rest endpoints

RESTful consumer endpoints designed to facilitate seamless integration

Overview

This page outlines Joule's RESTful consumer endpoints, which enable flexible integration for rapid use case development and validation. Joule supports two endpoint types: file-based and stream-based.

File-based POST endpoint Is optimised for handling large files, which are directed to a specified download directory and then moved to a processed directory after ingestion, with support for multiple file formats (i.e., PARQUET, JSON). This bridges the gap of batch to stream processing.

Stream-based POST endpoint Allows developers to chain Joule processes similarly to micro-services, supporting complex workflows and real-time data processing. Events are passed as StreamEvent objects through the POST body or an events parameter, enabling streamlined data flow between Joule processes.

Client library: io.javalin:javalin:5.6.3

Examples & DSL attributes

This example configures a REST File Consumer to consume PARQUET files from the nasdaq_quotes_file source.

The files are downloaded from the quotes topic and saved in the nasdaq/downloads directory. After the files are fully received via the file-based POST endpoint, they are processed efficiently.

Once processed, the files are moved to a local processed directory, with a timestamp marking their completion.

restFileConsumer:
  name: nasdaq_quotes_file
  topic: quotes
  file format: PARQUET
  download dir: nasdaq/dowloads

Attributes schema

Attribute	Description	Data Type	Required
topic	User defined topic to be used as the final endpoint component	String
file format	Expected file format to process. Defined as a enumeration, see below for supported file types	Enum Default: PARQUET
download dir	User defined target directory for files received	String Default: See below

Supported file types

• PARQUET

• ARROW_IPC

• ORC

• CSV

• JSON

Stream configuration guide

The stream-based POST endpoint allows developers to link Joule processes together, similar to microservices, enabling the creation of complex workflows.

This endpoint requires a StreamEvent object or an array of these events.

Example stream based configuration

Events can be sent either through:

1. The events parameter, which includes specific event details in the request or

2. Within the POST body, where the event information is contained in the HTTP POST request body.

restEventConsumer:
  name: nasdaq_quotes_stream
  topic: quotes

Core attributes schema

Currently only the Joule StreamEvent is supported

Attribute	Description	Data Type	Required
topic	User defined topic to be used as the final endpoint component	String

Joule properties DSL

This table outlines configurable properties in Joule for managing file uploads with the REST API, especially useful for handling large files.

These settings allow fine-tuning of file handling capabilities, helping to manage memory use and ensure smooth processing of large file uploads.

Attribute	Description	Data Type
joule.rest.downloadPath	Directory to write files that exceed the in-memory limit. Default: ./downloads	String
joule.rest.maxFileSize	The maximum individual file size allowed In GB. Default: 2 GB	Integer
joule.rest.InMemoryFileSize	The maximum file size to handle in memory in MB. Default: 100 MB	Integer
joule.rest.totalRequestSize	The maximum size of the entire multipart request in GB. Default: 5 GB	Integer