Quickstart
Gain insights from your data sources with minimised friction using the Joule prototyping platform

Download the getting started project here. Clone the repository and follow the instructions to run the examples
When you start working with Joule you will be editing files locally using a code editor and running use case examples using Postman. If you prefer to build your projects within an Integrated Development Environment (IDE) clone the existing banking example project from here.
Prerequisites
Getting started with Joule has minimal requirements to getting started but to take full advantage of the platform some key technical capabilities would be needed.
Step 1
Clone the getting started project to a local directory by using the following command:
git clone https://gitlab.com/joule-platform/fractalworks-stream-gettingstarted.git
Step 2
Install the required platform tools if you want to experiment with Joule SDK, see the setting up the environment document.
Note: To configure and run Joule it is important that you know some basics of the Terminal. In particular, you should understand general bash commands to navigate through the directory structure of your computer easily.
Running an example
Joule has provided the necessary scripts and configurations on getting a use case running using either a Docker image or a local unpacked installation, we shall use the local installation to get you familiar with the general directory structure, this will benefit your understanding of the provided Docker image.
1. Start up the environment
For this you need to change in to the quickstart
directory and run a single command.
cd quickstart
./startupJoule.sh
This will start up the following containers ready for use case deployment:
Joule Daemon
Joule Database
Redpanda (Lightweight Kafka implementation)
2. Deploy a use case
Both the banking and telco demo directories provide a set of examples in the form of a postman collection and environment. These can be found under the examples
directory. Now lets get you started running the banking Getting Started
demo example using Postman.
First import the use case demos and environment files from the banking-demo
examples
directorySet the environment to Joule.
From the
Getting started \ Deploy
folder clickRun folder
from the menu.Finally execute the
run order
by clicking theRun Joule - Banking demo
button
This will deploy the source, stream, sink and the use case binding definition to the platform. Note on a restart these setting will be rehydrated and will automatically start.
3. Start the quote simulator
This will generate simulated quotes based upon the provided nasdaq csv info file.
quickstart % ./bin/startQuotePublisher.sh
Starting Quotes Simulator v1.2.1-SNAPSHOT
ProcessId 52506
appending output to nohup.out
4. View the results
docker exec -it joule-gs-redpanda-0 rpk topic consume analytics_view --num 1
{
"topic": "analytics_view",
"key": "\u0000\u00015L",
"value": "{\"symbol\":\"PHD\",\"time\":1706625828336,\"askFirst\":15.420006518870125,\"askLast\":15.420006518870125}",
"timestamp": 1706625828093,
"partition": 0,
"offset": 0
}
Or from the Redpanda UI
Use this link to access the console

5. Stopping the processes
quickstart % ./bin/stopQuotePublisher.sh
Quote simulator closed
quickstart % shutdownJoule.sh
Joule processed closed
Note there are many other examples within the getting started
project. These are described within the README.md files.
Optional Step - Setup the environment
If you choose to build the environment for development purposes this is done by simply running the below command, please ensure you have the correct build environment set out in the setting up the environment document.
gradle buildJouleDemoEnv
This will build both the banking and telco example components, and copy across the configurations, libraries, and setup the demo environment.
Whats going on?
The banking getting started use case demonstrates core features that are reusable across all use cases; connecting to data, processing and distributing events.
Hence, we shall start with a simple use case that demonstrates how to get yourself running with the platform using the following repeatable steps. The use case subscribes to a Kafka quotes topic, get the high and low price per symbol using tumbling windows and then publish resulting event onto the analytics_view topic.
1. Connect and subscribe to a streaming data source
Define either one or more event sources using provided data source connectors.
Overview of the definition
nasdaq_quotes_event_stream
is the logical name for the source definitionSubscribe to events using the
quotes
Kafka topicReceived events are deserialised using a user defined parser in to a Joule
StreamEvent
to enable processing
Example Kafka subscription
kafkaConsumer:
name: nasdaq_quotes_stream
cluster address: joule-gs-redpanda-0:9092
consumerGroupId: nasdaq
topics:
- quotes
deserializer:
parser: com.fractalworks.examples.banking.data.QuoteToStreamEventParser
key deserializer: org.apache.kafka.common.serialization.IntegerDeserializer
value deserializer: com.fractalworks.streams.transport.kafka.serializers.object.ObjectDeserializer
properties:
partition.assignment.strategy: org.apache.kafka.clients.consumer.StickyAssignor
max.poll.records: '7000'
fetch.max.bytes: '10485760'
2. Process events
Use case processing is defined as a pipeline of processing stages. Joule provides a set of OOTB processors, see documentation, along with a SDK to enable developers to extend the platform capabilities.
Overview of the definition
basic_tumbling_window_pipeline
is used as the logical name to defined streaming processing pipeline, this will be used in the next stepProcessing constraints, valid date to and from, define when this stream can execute
Event processing will use the event time provide within the received event
The use case will subscribe to events from the
nasdaq_quotes_stream
data source configured in step 1.The use case applies 1 second tumbling window aggregate functions for two event attributes grouped by
symbol
A simple select projection emits the computed events
Example tumbling window calculations
stream:
name: basic_tumbling_window_pipeline
enabled: true
eventTimeType: EVENT_TIME
sources:
- nasdaq_quotes_stream
processing unit:
pipeline:
- time window:
emitting type: tumblingQuoteAnalytics
aggregate functions:
FIRST:
- ask
LAST:
- ask
policy:
type: tumblingTime
window size: 1000
emit:
select: symbol, ask_FIRST, ask_LAST
group by:
- symbol
3. Distribute processed events
Distribution of processed events can be as simple as to file, dashboard, or on to another streaming channel for another process to perform further processing. For this example we are using the Kafka sink connector. For further information on available sinks can be found here.
Overview of the definition
Provide a logical name for the distribution definition
Bind to the use case in this case it is
streamingAnalyticsPublisher
Define one or more channels to receive events
The published event is created by mapping the internal Joule event to the domain type defined by the transform StockAnalyticRecordTransform implementation which will then be converted to Json
Example Kafka publish connection using a custom transformation
kafkaPublisher:
name: kafka_analytics_view
cluster address: joule-gs-redpanda-0:9092
topic: analytics_view
partitionKeys:
- symbol
serializer:
transform: com.fractalworks.examples.banking.data.StockAnalyticRecordTransform
key serializer: org.apache.kafka.common.serialization.IntegerSerializer
value serializer: com.fractalworks.streams.transport.kafka.serializers.json.ObjectJsonSerializer
Deployment artefact
Now we bring together each deployment artefact (source, use case and sinks) to form the desired use case. A use case is formed by a single app.env file which references these files. This method of deployment enables you to simply switch out the source and sinks based upon your needs i.e. development, testing and production deployments

Binding file used to run a use case
use case:
name: basic_twindow_analytics
constraints:
valid from: "2024-01-01T08:00:00.000Z"
valid to: "2030-01-01T23:59:00.000Z"
sources:
- nasdaq_quotes_stream
stream name: basic_tumbling_window_pipeline
sinks:
- kafka_analytics_view
Last updated
Was this helpful?