Data priming

Prime Joule with necessary startup data

This is an optional feature that provides the ability to prime Joule with data necessary for an active use case

Overview

Advanced use cases often require contextual data to support calculations or complex business logic. Joule enables this by offering data priming at initialisation and enrichment processing stages.

The initialisation process imports data at startup from local files into an in-memory SQL database, making it immediately available for use in processing.

To see what formats can be imported, see the Types of imports page.

Initialisation process

Joule’s initialisation process leverages an embedded SQL engine, enabling powerful features like metrics, event capturing, data exporting and access to contextual data.

This imported data, typically static contextual information, plays a vital role in supporting key functions within the event stream pipeline.

Data made available through the initialisation process can be accessed through several main components:

1. Enricher processor For adding contextual information to events.

2. Metrics engine For real-time calculations and metrics updates.

3. Select projection For choosing specific fields for further processing.

4. In-memory SQL API For direct data access and manipulation within Joule.

Attribute	Description	Data Type	Required
schema	Global database schema when set can be used for any import definition where schema is not defined. Default schema reference_data	String
parquet	List of parquet data import configurations	See parquet attributes
csv	List of CSV data import configurations	See CSV attributes

Examples & DSL attributes

This following example demonstrates how to initialise two separate data files into independent in-memory SQL database tables using CSV and Parquet formats.

1. The CSV file contains Nasdaq company information, it is treated as static reference data and is therefore stored in the reference_data schema.

2. Meanwhile, the Parquet file loads pre-calculated metrics, priming the metrics engine within the metrics schema.

This setup enables efficient access to contextual data and metrics calculations during event processing.

This feature can load and read files from existing databases!

stream:
  ...
  initialisation:
    data import:
      csv:
        - schema: reference_data
          table: nasdaq_companies
          file: 'data/csv/nasdaq.csv'
          drop table: true
          index:
            fields: [ 'Symbol' ]
            unique: true
  
      parquet:
        - schema: metrics
          table: bid_moving_averages      
          files: ['data/parquet/mvavgs-prime.parquet']
          drop table: true
          index:
            fields: [ 'symbol' ]
            unique: false

Attributes schema

These are common DSL keywords used in both parquet and CSV importing methods.

Attribute	Description	Data Type	Required
schema	Database schema to create and apply table import function	String
table	Target table to import data into	String
drop table	Drop existing table before import. This will cause a table recreation	Boolean Default true
index	Create an index on the created table	See next

Index

If this optional field is supplied the index is recreated once the data has been imported.

Attribute	Description	Data Type	Required
fields	A list of table fields to base index on	String
unique	True for a unique index	Boolean Default true