Change history

Change log

VersionDateDescription

1.2.0-SNAPSHOT

04-June

Features

Stream Joins

  • Ability to join two streams using a simple join expression

  • Inner and outer joins support

  • State managed either through an time to live function or simply delete on join

Enrichment

  • Computed metrics can be added to events using the enrichment processor and emit projection

  • Internal in-memory database can be queried for event enrichment

  • Stateful computation support

Metrics

  • Import pre-computed metrics on process startup

  • User defined metrics compaction query

Enhancements

Kafka

  • Validated consumer message partition examples using multi Joule processing

Analytics

  • New normalization, scaling, and statistics functions added for ML support

  • Stateful computation applied to computation context

  • Simplified DSL to enable correct processor to be used

  • Improvements and bug fixes to address execution of expressions and scripts

  • Documentation improvements

Miscellaneous

  • Multi Joule processing environment scripts, used within banking examples

  • groupby definition support across all processors

  • Javascript module mjs support

Upgrades

  • DuckDB 1.0.0

1.1.0

25-Jan-2024

Features

Predictive Processor

  • JPMML model initialization using local file and remote S3 stores

  • Dynamic model refresh using model update notifications

  • Offline prediction auditing that enables explainability, drift monitoring and model retraining

Avro support

  • Ability to process avro records for inbound and outbound events

  • Complex data types supported using custom mapping

  • Schema registry support

Minio S3 Transport

  • OOTB multi cloud S3 support

  • Publish and consume events and insights to/from hybrid hosted S3 buckets

  • Drive pipeline processing using S3 bucket notifications

  • Consumer supports following file formats: PARQUET, CSV, ARROW, ORC

  • Keep reference data up to date using external systems

Reference Data

  • Apply external data within stream processing tasks

  • In-memory reference data elements kept up-to-date using source change notifications

  • Support for key value and S3 stores

  • Reference data file loader utility

Rest Consumer APIs

  • File consuming endpoint that enable ease of integration to upstream systems

  • Joule event consumer endpoint to provide the ability to chain Joule processors within a cloud environment

File Watcher Consumer

  • File watcher that consumes and processes target files

  • Supported formats; Parquet, Json, CSV, ORC and Arrow IPC

Enhancements

Kafka

  • Confluent schema registry support for outbound events

  • Message partition support

  • Confluent and RedPanda support

Enricher processor

  • Query optimisation

  • SQL, OQL, and Key value enrichment support

Transports

  • Improved exception handling to fail on startup

  • Strict ordering

Apache Arrow

  • Integrated and leveraged to process file efficiently and of various file formats

  • Large file processing support

Optimisations

  • Processing optimisations that reduce both memory and CPU utilisation while increasing event throughput.

  • StreamEvent smart shallow cloning logic to reduce overall memory footprint while providing key data isolation

  • StreamEvent change tracking switch to reduce memory overhead

Upgrades

  • Javalin 5.6.3

  • Kafka 3.6.0

  • Avro 1.11.3

  • DuckDB 0.9.2

Bug Fixes

StreamEventCSVDeserializer

  • Fixed fields from holding only string values to correctly defined data types

  • Allowed for custom date format to be provided

StreamEventJSONDeserializer

  • Can now read an array of Json StreamEvent objects

JVM Configuration Additions

  • Require ‘--add-opens=java.base/java.nio=ALL-UNNAMED’ to be added to the java CLI due to Apache Arrow requirements

  • Applying the G1 GC regionalized and generational garbage collector to improved memory usage

1.1.0-SNAPSHOT

14-Sept

Features

  • Predictive Processor Model initialisation and dynamic model replacement using local file and remote S3 stores

  • Avro support for inbound and outbound events.

  • Complex data types supported using custom field mapping

  • Minio multi-cloud S3 consumer

  • Reference data support

Enhancements

  • MinIO implementation improvements to support large files and notifications

  • Kafka schema registry support

  • Apache Arrow integration to support PARQUET, CSV, ARROW, ORC file formats

  • Better transport exception handling

1.0.4

18-May-2023

Features

  • Feature engineering processor, Including plugins for binning, scaling and declarative analytics expressions

  • Prediction and features auditing enabling explainability, drift monitoring and retraining

  • In-Memory SQL Query API

  • Websocket publisher

Enhancements

  • Improved deep cloning of StreamEvent

  • Reorganisation of SDK

  • Documentation

  • Project template

Bug fixes

  • Window events failing to publish

1.0.4-SNAPSHOT

23-Apr-2023

Docker image generation, scripting fixes, StreamEvent deep copy, documentation updates.

1.0.3

01-Mar-2023

Documentation, metrics engine, Rest API, JDBC publisher, transport and processor updates and bug fixes

1.0.2

30-Jan-2023

Alpha release of metrics engine and rest API

1.0.1

13-May-2022

Bug fixes

1.0.0

05-Apr-2022

Initial release

Last updated