# Rest endpoints

## Overview

This page outlines Joule's RESTful consumer endpoints, which enable flexible integration for rapid use case development and validation. Joule supports two endpoint types: **file-based** and **stream-based**.

<mark style="color:green;">**File-based POST endpoint**</mark>\
Is optimised for handling large files, which are directed to a specified download directory and then moved to a processed directory after ingestion, with support for multiple file formats (i.e., PARQUET, JSON). This **bridges the gap** of batch to stream processing.

<mark style="color:green;">**Stream-based POST endpoint**</mark>\
Allows developers to **chain Joule processes similarly** to micro-services, supporting complex workflows and real-time data processing. Events are passed as `StreamEvent` objects through the POST body or an `events` parameter, enabling **streamlined data flow** between Joule processes.

{% hint style="info" %}
**Client library:** [io.javalin:javalin:5.6.3](https://mvnrepository.com/artifact/io.javalin/javalin/5.6.3)
{% endhint %}

## Examples & DSL attributes

This example configures a **REST File Consumer** to consume **PARQUET** files from the `nasdaq_quotes_file` source.

The files are downloaded from the `quotes` topic and saved in the `nasdaq/downloads` directory. After the files are fully received via the file-based POST endpoint, they are processed efficiently.

Once processed, the files are moved to a local `processed` directory, with a timestamp marking their completion.

```yaml
restFileConsumer:
  name: nasdaq_quotes_file
  topic: quotes
  file format: PARQUET
  download dir: nasdaq/dowloads
```

### Attributes schema

<table><thead><tr><th width="178">Attribute</th><th width="281">Description</th><th width="191">Data Type</th><th data-type="checkbox">Required</th></tr></thead><tbody><tr><td>topic</td><td>User defined topic to be used as the final endpoint component</td><td>String</td><td>true</td></tr><tr><td>file format</td><td>Expected file format to process. Defined as a enumeration, see below for supported file types</td><td>Enum<br>Default: PARQUET</td><td>false</td></tr><tr><td>download dir</td><td>User defined target directory for  files received</td><td>String<br>Default: See below</td><td>false</td></tr></tbody></table>

### Supported file types

* PARQUET
* ARROW\_IPC
* ORC
* CSV
* JSON

### Stream configuration guide

The stream-based POST endpoint allows developers to link Joule processes together, **similar to microservices**, enabling the creation of complex workflows.

This endpoint requires a `StreamEvent` object or an array of these events.

### Example stream based configuration

Events can be sent either through:

1. The `events` parameter, which includes specific event details in the request or
2. Within the POST body, where the event information is contained in the HTTP POST request body.

```yaml
restEventConsumer:
  name: nasdaq_quotes_stream
  topic: quotes
```

### Core attributes schema

{% hint style="info" %}
Currently only the Joule `StreamEvent` is supported
{% endhint %}

<table><thead><tr><th width="178">Attribute</th><th width="281">Description</th><th width="191">Data Type</th><th data-type="checkbox">Required</th></tr></thead><tbody><tr><td>topic</td><td>User defined topic to be used as the final endpoint component</td><td>String</td><td>true</td></tr></tbody></table>

## Joule properties DSL

This table outlines configurable properties in Joule for managing file uploads with the REST API, especially useful for handling large files.

These settings allow fine-tuning of file handling capabilities, helping to manage memory use and ensure smooth processing of large file uploads.

<table><thead><tr><th width="249">Attribute</th><th width="396">Description</th><th width="221">Data Type</th></tr></thead><tbody><tr><td>joule.rest.downloadPath</td><td>Directory to write files that exceed the in-memory limit. Default: ./downloads</td><td>String</td></tr><tr><td>joule.rest.maxFileSize</td><td><p>The maximum individual file size allowed In GB. </p><p>Default: 2 GB</p></td><td>Integer</td></tr><tr><td>joule.rest.InMemoryFileSize</td><td>The maximum file size to handle in memory in MB. Default: 100 MB</td><td>Integer</td></tr><tr><td>joule.rest.totalRequestSize</td><td>The maximum size of the entire multipart request in GB. Default: 5 GB</td><td>Integer</td></tr></tbody></table>


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.fractalworks.io/joule/components/connectors/sources/rest-endpoints.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.