Environment installation

Installation

The following describes how to set up a local environment mainly using docker images. Joule has provided docker-compose files to start the environment processes.

Start up infrastructure

Startup Postgres, Kafka, InfluxDB and Grafana using the provided scripts located under their respective infra directories.

Kafka Solution

Simulated events are streamed using Kafka. Either Confluent or Redpanda can be used for the Kafka solution. We prefer to use Redpanda under development environment as its a light weight standard implementation. 

# Confluent
docker-compose -f infra/confluent/docker-compose.yml up -d

# Redpanda
docker-compose -f infra/redpanda/docker-compose.yml up -d

Postgres database

If use database publisher configuration to store the resulting data you will want to fire up a postgres instance.

docker-compose -f infra/postgres/docker-compose.yml up -d

Addition infrastructure

Time series database and dashboard is useful to demonstrate KPIs and live event data. InfluxDB and Grafana solutions have been package. Use the influx based publisher configurations to connect Joule to InfluxDB.

docker-compose -f infra/influx-grafana/docker-compose.yml up -d

Postrges configurations

To use Postgres you will need to perform three simple tasks, define a target schema, copy jdbc postgres jar and link the databaseStandardAnalytics.yaml file in the bin/app.env file.

cp infra/postgres/postgresql-42.5.0.jar lib/

Example database schema file

CREATE TABLE IF NOT EXISTS quote_analytics (
    id              SERIAL PRIMARY KEY,
    symbol          VARCHAR(10),
    ask_EMA         float8,
    bid_EMA         float8
    );

Example database publisher file

publisher:
  name: standardAnalyticsDatabasePublisher
  source: standardQuoteAnalyticsStream
  sinks:
    - sqlPublisher:
        jdbcDriver: org.postgresql.Driver
        host: jdbc:postgresql://localhost:5432
        database: jouleTestDB
        table: quote_analytics
        includeTimestamps: false
        properties:
          ssl: false
          user: joule
          password: joulepwd

Influx and Grafana configurations

The infra/influx-grafana/.env file contains all the required configuration setting for InfluxDB and Grafana. Processed events will be stored using the below org and bucket.

  • user: influx

  • password: influxdb

  • organisation: banking

  • bucket: nasdaqBaseAnalytics

In the event of changing the setting these need to be replicated in the influxDB publishing conf/publishers/influxdbStandardAnalytics.yaml file.

Last updated