Comment on page
Sinks
Data sinks provide independent formatted data events to down stream systems
A single publishing file provides one or more data sinks through the
sinks
tag. This forms a logic set of data sinks that enables a use case to publish results to multiple downstream consumers.The publisher file must subscribe to a Joule use case event source, in the example below this is the
tumblingWindowQuoteStream
source. Every defined event sink will receive the same collection of events to publish.publisher:
name: quoteAnalyticsPublisher
source: tumblingWindowQuoteStream
sinks:
- file:
filename: quoteWindowsStream
path: ./data/output/analytics
batchSize: 1024
timeout: 1000
formatter:
csvFormatter:
contentType: "text/csv"
delimiter: "|"
Each data publisher defines its own syntax due to their specific functional requirements.
Publisher namespace.
The logical name of the set of publisher transports.
Event source for stream publishing.
List of data publisher transports.
Publishers receive StreamEvents which need to be converted to a desired output format. Currently the following formatters are provided.
- CSV
- JSON
- Parquet
The Joule SDK provides interfaces for custom formatter code. Read the Joule SDK documentation for more information.
Last modified 11mo ago