Quickstart

Gain insights from your data sources with minimised friction using the Joule prototyping platform

---

Download the getting started project here. Clone the repository and follow the instructions to run the examples

When you start working with Joule you will be editing files locally using a code editor and running use case examples using Postman. If you prefer to build your projects within an Integrated Development Environment (IDE) clone the existing banking example project from here.

Prerequisites

Getting started with Joule has minimal requirements to getting started but to take full advantage of the platform some key technical capabilities would be needed.

Step 1

Clone the getting started project to a local directory by using the following command:

git clone https://gitlab.com/joule-platform/fractalworks-stream-gettingstarted.git

Step 2

Install the required platform tools if you want to experiment with Joule SDK, see the setting up the environment document.

Note: To configure and run Joule it is important that you know some basics of the Terminal. In particular, you should understand general bash commands to navigate through the directory structure of your computer easily.

Running an example

Joule has provided the necessary scripts and configurations on getting a use case running using either a Docker image or a local unpacked installation, we shall use the local installation to get you familiar with the general directory structure, this will benefit your understanding of the provided Docker image.

1. Start up the environment

For this you need to change in to the quickstart directory and run a single command.

cd quickstart
./startupJoule.sh

This will start up the following containers ready for use case deployment:

• Joule Daemon

• Joule Database

• Redpanda (Lightweight Kafka implementation)

2. Deploy a use case

Both the banking and telco demo directories provide a set of examples in the form of a postman collection and environment. These can be found under the examples directory. Now lets get you started running the banking Getting Started demo example using Postman.

1. First import the use case demos and environment files from the banking-demo examples directory

2. Set the environment to Joule.

3. From the Getting started \ Deploy folder click Run folder from the menu.

4. Finally execute the run order by clicking the Run Joule - Banking demo button

This will deploy the source, stream, sink and the use case binding definition to the platform. Note on a restart these setting will be rehydrated and will automatically start.

3. Start the quote simulator

This will generate simulated quotes based upon the provided nasdaq csv info file.

quickstart % ./bin/startQuotePublisher.sh 
Starting Quotes Simulator v1.2.1-SNAPSHOT
ProcessId 52506
appending output to nohup.out

4. View the results

docker exec -it joule-gs-redpanda-0 rpk topic consume analytics_view --num 1
{
  "topic": "analytics_view",
  "key": "\u0000\u00015L",
  "value": "{\"symbol\":\"PHD\",\"time\":1706625828336,\"askFirst\":15.420006518870125,\"askLast\":15.420006518870125}",
  "timestamp": 1706625828093,
  "partition": 0,
  "offset": 0
}

Or from the Redpanda UI

Use this link to access the console

Redpanda UI

5. Stopping the processes

quickstart % ./bin/stopQuotePublisher.sh 
Quote simulator closed

quickstart % shutdownJoule.sh 
Joule processed closed

Note there are many other examples within the getting started project. These are described within the README.md files.

Optional Step - Setup the environment

If you choose to build the environment for development purposes this is done by simply running the below command, please ensure you have the correct build environment set out in the setting up the environment document.

gradle buildJouleDemoEnv

This will build both the banking and telco example components, and copy across the configurations, libraries, and setup the demo environment.

Whats going on?

The banking getting started use case demonstrates core features that are reusable across all use cases; connecting to data, processing and distributing events.

Hence, we shall start with a simple use case that demonstrates how to get yourself running with the platform using the following repeatable steps. The use case subscribes to a Kafka quotes topic, get the high and low price per symbol using tumbling windows and then publish resulting event onto the analytics_view topic.

1. Connect and subscribe to a streaming data source

Define either one or more event sources using provided data source connectors.

Overview of the definition

• nasdaq_quotes_event_stream is the logical name for the source definition

• Subscribe to events using the quotes Kafka topic

• Received events are deserialised using a user defined parser in to a Joule StreamEvent to enable processing

Example Kafka subscription

kafkaConsumer:
  name: nasdaq_quotes_stream
  cluster address: joule-gs-redpanda-0:9092
  consumerGroupId: nasdaq
  topics:
  - quotes
  deserializer:
    parser: com.fractalworks.examples.banking.data.QuoteToStreamEventParser
    key deserializer: org.apache.kafka.common.serialization.IntegerDeserializer
    value deserializer: com.fractalworks.streams.transport.kafka.serializers.object.ObjectDeserializer
  properties:
    partition.assignment.strategy: org.apache.kafka.clients.consumer.StickyAssignor
    max.poll.records: '7000'
    fetch.max.bytes: '10485760'

2. Process events

Use case processing is defined as a pipeline of processing stages. Joule provides a set of OOTB processors, see documentation, along with a SDK to enable developers to extend the platform capabilities.

Overview of the definition

• basic_tumbling_window_pipeline is used as the logical name to defined streaming processing pipeline, this will be used in the next step

• Processing constraints, valid date to and from, define when this stream can execute

• Event processing will use the event time provide within the received event

• The use case will subscribe to events from the nasdaq_quotes_stream data source configured in step 1.

• The use case applies 1 second tumbling window aggregate functions for two event attributes grouped by symbol

• A simple select projection emits the computed events

Example tumbling window calculations

stream:
  name: basic_tumbling_window_pipeline
  enabled: true
  eventTimeType: EVENT_TIME
  sources:
  - nasdaq_quotes_stream
  processing unit:
    pipeline:
    - time window:
        emitting type: tumblingQuoteAnalytics
        aggregate functions:
          FIRST:
          - ask
          LAST:
          - ask
        policy:
          type: tumblingTime
          window size: 1000
  emit:
    select: symbol, ask_FIRST, ask_LAST
  group by:
  - symbol

3. Distribute processed events

Distribution of processed events can be as simple as to file, dashboard, or on to another streaming channel for another process to perform further processing. For this example we are using the Kafka sink connector. For further information on available sinks can be found here.

Overview of the definition

• Provide a logical name for the distribution definition

• Bind to the use case in this case it is streamingAnalyticsPublisher

• Define one or more channels to receive events

• The published event is created by mapping the internal Joule event to the domain type defined by the transform StockAnalyticRecordTransform implementation which will then be converted to Json

Example Kafka publish connection using a custom transformation

kafkaPublisher:
  name: kafka_analytics_view
  cluster address: joule-gs-redpanda-0:9092
  topic: analytics_view
  partitionKeys:
  - symbol
  serializer:
    transform: com.fractalworks.examples.banking.data.StockAnalyticRecordTransform
    key serializer: org.apache.kafka.common.serialization.IntegerSerializer
    value serializer: com.fractalworks.streams.transport.kafka.serializers.json.ObjectJsonSerializer

Deployment artefact

Now we bring together each deployment artefact (source, use case and sinks) to form the desired use case. A use case is formed by a single app.env file which references these files. This method of deployment enables you to simply switch out the source and sinks based upon your needs i.e. development, testing and production deployments

use case file brings each descriptor file to form a single unit of deployment

Binding file used to run a use case

use case:
  name: basic_twindow_analytics
  constraints:
    valid from: "2024-01-01T08:00:00.000Z"
    valid to: "2030-01-01T23:59:00.000Z"
  sources:
  - nasdaq_quotes_stream
  stream name: basic_tumbling_window_pipeline
  sinks:
  - kafka_analytics_view