Comment on page

Sinks

Data sinks provide independent formatted data events to down stream systems

Declaring data sinks
A single publishing file provides one or more data sinks through the sinks tag. This forms a logic set of data sinks that enables a use case to publish results to multiple downstream consumers.
The publisher file must subscribe to a Joule use case event source, in the example below this is the tumblingWindowQuoteStream source.  Every defined event sink will receive the same collection of events to publish.
File example
publisher:
  name: quoteAnalyticsPublisher
  source: tumblingWindowQuoteStream
  sinks:
    - file:
        filename: quoteWindowsStream
        path: ./data/output/analytics
        batchSize: 1024
        timeout: 1000
        formatter:
          csvFormatter:
            contentType: "text/csv"
            delimiter: "|"
Each data publisher defines its own syntax due to their specific functional requirements. 
DSL Elements
publisher
Publisher namespace.
name
The logical name of the set of publisher transports.
source
Event source for stream publishing.
sinks
List of data publisher transports.
Transformation
Publishers receive StreamEvents which need to be converted to a desired output format. Currently the following formatters are provided.
CSV
JSON
Parquet
The Joule SDK provides interfaces for custom formatter code. Read the Joule SDK documentation for more information.

Group by

Next - Connector Catalog

Sources

Last modified 11mo ago