Comment on page
Sources
Data sources provide independent data events that are transformed in to an internal form to drive a use case
A single sources file provides one or more data sources through the
sources tag. This forms a logic set of data sources that enables one or more use cases. consumer:
name: markets_data_streams
sources:
- kafkaConsumer:
name: nasdaq_quotes_stream
clusterAddress: localhost:9092
consumerGroupId: nasdaq
topics:
- quotes
deserializer:
transform: com.fractalworks.streams.examples.banking.data.QuoteToStreamEventTransformer
keyDeserializer: org.apache.kafka.common.serialization.IntegerSerializer
valueDeserializer: com.fractalworks.streams.transport.kafka.serializers.object.ObjectDeserializer
Each data consumer defines its own syntax due to their specific functional requirements. A use case references a data source by using the name specified within the transport definition, as an example
nasdaq_quotes_stream for the kafkaconsumer.Consumer namespace.
The logical name of the set of consumer transports.
List of data consumer transports.
For data to be processed it needs to be transformed in to a Joule StreamEvent object. These objects are then used throughout the Joule platform until the publishing stage of the process. For more information read the custom transformers documentation.
Last modified 11mo ago