Sources

Data sources provide independent data events that are transformed in to an internal form to drive a use case

Declaring data sources

A single sources file provides one or more data sources through the sources tag. This forms a logic set of data sources that enables one or more use cases.

Kafka example

consumer:
  name: markets_data_streams
  sources:
    - kafkaConsumer:
        name: nasdaq_quotes_stream
        clusterAddress: localhost:9092
        consumerGroupId: nasdaq
        topics:
          - quotes

        deserializer:
          transform: com.fractalworks.streams.examples.banking.data.QuoteToStreamEventTransformer
          keyDeserializer: org.apache.kafka.common.serialization.IntegerSerializer
          valueDeserializer: com.fractalworks.streams.transport.kafka.serializers.object.ObjectDeserializer
          

Each data consumer defines its own syntax due to their specific functional requirements. A use case references a data source by using the name specified within the transport definition, as an example nasdaq_quotes_stream for the kafkaconsumer.

DSL Elements

consumer

Consumer namespace.

name

The logical name of the set of consumer transports.

sources

List of data consumer transports.

Transformation

For data to be processed it needs to be transformed in to a Joule StreamEvent object. These objects are then used throughout the Joule platform until the publishing stage of the process. For more information read the custom transformers documentation.