# Rest endpoints

## Overview

This page outlines Joule's RESTful consumer endpoints, which enable flexible integration for rapid use case development and validation. Joule supports two endpoint types: **file-based** and **stream-based**.

<mark style="color:green;">**File-based POST endpoint**</mark>\
Is optimised for handling large files, which are directed to a specified download directory and then moved to a processed directory after ingestion, with support for multiple file formats (i.e., PARQUET, JSON). This **bridges the gap** of batch to stream processing.

<mark style="color:green;">**Stream-based POST endpoint**</mark>\
Allows developers to **chain Joule processes similarly** to micro-services, supporting complex workflows and real-time data processing. Events are passed as `StreamEvent` objects through the POST body or an `events` parameter, enabling **streamlined data flow** between Joule processes.

{% hint style="info" %}
**Client library:** [io.javalin:javalin:5.6.3](https://mvnrepository.com/artifact/io.javalin/javalin/5.6.3)
{% endhint %}

## Examples & DSL attributes

This example configures a **REST File Consumer** to consume **PARQUET** files from the `nasdaq_quotes_file` source.

The files are downloaded from the `quotes` topic and saved in the `nasdaq/downloads` directory. After the files are fully received via the file-based POST endpoint, they are processed efficiently.

Once processed, the files are moved to a local `processed` directory, with a timestamp marking their completion.

```yaml
restFileConsumer:
  name: nasdaq_quotes_file
  topic: quotes
  file format: PARQUET
  download dir: nasdaq/dowloads
```

### Attributes schema

<table><thead><tr><th width="178">Attribute</th><th width="281">Description</th><th width="191">Data Type</th><th data-type="checkbox">Required</th></tr></thead><tbody><tr><td>topic</td><td>User defined topic to be used as the final endpoint component</td><td>String</td><td>true</td></tr><tr><td>file format</td><td>Expected file format to process. Defined as a enumeration, see below for supported file types</td><td>Enum<br>Default: PARQUET</td><td>false</td></tr><tr><td>download dir</td><td>User defined target directory for  files received</td><td>String<br>Default: See below</td><td>false</td></tr></tbody></table>

### Supported file types

* PARQUET
* ARROW\_IPC
* ORC
* CSV
* JSON

### Stream configuration guide

The stream-based POST endpoint allows developers to link Joule processes together, **similar to microservices**, enabling the creation of complex workflows.

This endpoint requires a `StreamEvent` object or an array of these events.

### Example stream based configuration

Events can be sent either through:

1. The `events` parameter, which includes specific event details in the request or
2. Within the POST body, where the event information is contained in the HTTP POST request body.

```yaml
restEventConsumer:
  name: nasdaq_quotes_stream
  topic: quotes
```

### Core attributes schema

{% hint style="info" %}
Currently only the Joule `StreamEvent` is supported
{% endhint %}

<table><thead><tr><th width="178">Attribute</th><th width="281">Description</th><th width="191">Data Type</th><th data-type="checkbox">Required</th></tr></thead><tbody><tr><td>topic</td><td>User defined topic to be used as the final endpoint component</td><td>String</td><td>true</td></tr></tbody></table>

## Joule properties DSL

This table outlines configurable properties in Joule for managing file uploads with the REST API, especially useful for handling large files.

These settings allow fine-tuning of file handling capabilities, helping to manage memory use and ensure smooth processing of large file uploads.

<table><thead><tr><th width="249">Attribute</th><th width="396">Description</th><th width="221">Data Type</th></tr></thead><tbody><tr><td>joule.rest.downloadPath</td><td>Directory to write files that exceed the in-memory limit. Default: ./downloads</td><td>String</td></tr><tr><td>joule.rest.maxFileSize</td><td><p>The maximum individual file size allowed In GB. </p><p>Default: 2 GB</p></td><td>Integer</td></tr><tr><td>joule.rest.InMemoryFileSize</td><td>The maximum file size to handle in memory in MB. Default: 100 MB</td><td>Integer</td></tr><tr><td>joule.rest.totalRequestSize</td><td>The maximum size of the entire multipart request in GB. Default: 5 GB</td><td>Integer</td></tr></tbody></table>