Sinks

Data sinks provide independent formatted data events to down stream systems

Declaring data sinks

A single publishing file provides one or more data sinks through the sinks tag. This forms a logic set of data sinks that enables a use case to publish results to multiple downstream consumers.

The publisher file must subscribe to a Joule use case event source, in the example below this is the tumblingWindowQuoteStream source. Every defined event sink will receive the same collection of events to publish.

File example

publisher:
  name: quoteAnalyticsPublisher
  source: tumblingWindowQuoteStream
  sinks:
    - file:
        filename: quoteWindowsStream
        path: ./data/output/analytics
        batchSize: 1024
        timeout: 1000
        formatter:
          csvFormatter:
            contentType: "text/csv"
            delimiter: "|"

Each data publisher defines its own syntax due to their specific functional requirements.

DSL Elements

publisher

Publisher namespace.

name

The logical name of the set of publisher transports.

source

Event source for stream publishing.

sinks

List of data publisher transports.

Transformation

Publishers receive StreamEvents which need to be converted to a desired output format. Currently the following formatters are provided.

• CSV

• JSON

• Parquet

The Joule SDK provides interfaces for custom formatter code. Read the Joule SDK documentation for more information.