Rest Endpoints

Joule offers RESTful consumer endpoints designed to facilitate seamless integration, with a focus on expediting use case development and validation processes.

Overview

Two types of endpoints are supported

• File to bridge batch to stream processing

• Stream simplifies cloud integration between Joule processors

File Configuration Guide

The file-based POST endpoint excels in efficiently processing large files once they have been fully received. Files are systematically directed to the designated local download directory. Following processing, they are then moved to the local processed directory, distinguished by a completion timestamp.

Example

restFileConsumer:
  name: nasdaq_quotes_file
  topic: quotes
  file format: PARQUET
  download dir: nasdaq/dowloads

Attributes schema

Available configuration parameters

Attribute	Description	Data Type	Required
topic	User defined topic to be used as the final endpoint component	String
file format	Expected file format to process. Defined as a enumeration, see below for supported file types.	Enum Default: PARQUET
download dir	User defined target directory for files received	String Default: See below

Supported File Types

• PARQUET

• ARROW_IPC

• ORC

• CSV

• JSON

Stream Configuration Guide

The stream-based POST endpoint empowers developers to seamlessly chain Joule processes, akin to microservices, thereby facilitating the creation of intricate use cases. For the POST endpoint, a Joule StreamEvent object or array is required.

Example stream based configuration

restEventConsumer:
  name: nasdaq_quotes_stream
  topic: quotes

Events can be posted by using either the 'events' parameter, where specific event-related information is included as part of the request, or through the POST body, where the details of the event are encapsulated within the body of the HTTP POST request.

Core Attributes

Available configuration parameters

Attribute	Description	Data Type	Required
topic	User defined topic to be used as the final endpoint component	String

Note: Currently only the Joule StreamEvent is supported.

Joule Properties

Attribute	Description	Data Type
joule.rest.downloadPath	Directory to write files that exceed the in-memory limit. Default: ./downloads	String
joule.rest.maxFileSize	The maximum individual file size allowed In GB. Default: 2 GB	Integer
joule.rest.InMemoryFileSize	The maximum file size to handle in memory in MB. Default: 100 MB	Integer
joule.rest.totalRequestSize	The maximum size of the entire multipart request in GB. Default: 5 GB	Integer

Client library

io.javalin:javalin:5.6.3