Low code development

Joule is a low code development platform designed to ideate, pilot and scale business use cases

Joule low-code approach

The Joule platform offers a low-code development solution that simplifies coding complexity by using a high-level language. This is accomplished through the Joule DSL (Domain-Specific Language), which enables the definition of use cases using human-readable YAML syntax.

This forms the key platform low code approach

Use case definition

A use case is defined by combining source data requirements, processing pipeline, and output destinations into a single cohesive definition.

Example

Each use case dependency is linked using a logical name which exists within an independent file.

This results in a single use case definition

use case:
  name: basic_twindow_analytics
  sources:
  - nasdaq_quotes_stream
  stream name: basic_tumbling_window_pipeline
  sinks:
  - kafka_analytics_view

Four functionally independent definitions form the Joule use case. These are outlined below with accompanying detailed documentation in further articles.

Stream

A stream defines the actual processing requirements and sequence. See the pipelines documentation for further information.

Example

The below example creates the min/max values for ask and bid values within a five-second tumbling window and only publishes symbols where they are not 'A'.

stream:
  name: basic_tumbling_window_pipeline
  eventTimeType: EVENT_TIME
  sources:
    - nasdaq_quotes_stream

  processing unit:
    pipeline:
      - time window:
          emitting type: tumblingQuoteAnalytics
          aggregate functions:
            MIN: [ask, bid]
            MAX: [ask, bid]
          policy:
            type: tumblingTime
            windowSize: 5000

  emit:
    eventType: windowQuoteEvent
    select: "symbol, ask_MIN, ask_MAX, bid_MIN, bid_MAX"
    having: "symbol !='A'"

  group by:
    - symbol

Data subscription

Users are able to subscribe to external data events through the use of source connectors. For a detailed understanding of how to subscribe to data and available connectors read the data sources documentation.

Example

The below example connects to a Kafka cluster, consumes events from the quote topic and transforms the received quote object into an internal StreamEvent object.

consumer:
  name: nasdaq_quotes_stream
  sources:
    - kafkaConsumer:
        name: nasdaq_quotes_stream
        cluster address: KAFKA_BROKER:19092
        consumerGroupId: nasdaq
        topics:
          - quotes

        deserializer:
          parser: com.fractalworks.examples.banking.data.QuoteToStreamEventParser
          key deserializer: org.apache.kafka.common.serialization.IntegerDeserializer
          value deserializer: com.fractalworks.streams.transport.kafka.serializers.object.ObjectDeserializer

Event publishing

Users are able to publish events to downstream data platforms through the use of destination connectors. For a detailed understanding of how to publish data and available connectors read the data sinks documentation.

Example

The below example generates a quoteWindowStream.csv file from the tumblingWindowQuoteStream events.

publisher:
  name: standardAnalyticsFilePublisher
  source: basic_tumbling_window_pipeline
  sinks:
    - file:
        enabled: true
        filename: nasdaqAnalytic
        path: ./data/output/analytics
        batchSize: 1024
        timeout: 1000
        formatter:
          csv formatter:
            contentType: "text/csv"
            encoding: "UTF-8"
            delimiter: "|"

Reference data

Often in stream processing additional data is required to perform analytics, generally known as reference data. Data of this form generally updates at a much slower pace and therefore is managed differently and held in data platform not architected for low latency reads. Joule has build a low latency read mechanism to overcome this limitation using in-memory caching. Further information can be found in the contextual data documentation.

Example

The below example connects to a distributed caching platform, Apache Geode, for a low latency reference data reads.

reference data:
  name: banking market data 
  data sources:
    - geode stores:
        name: us markets
        connection:
          locator address: 192.168.86.39
          locator port: 41111
        stores:
          nasdaqIndexCompanies:
            region: nasdaq-companies
            keyClass : java.lang.String
            gii: true
          holidays:
            region: us-holidays
            keyClass : java.lang.Integer