Joule
Search
K

Change history

Change log

Version
Date
Description
1.2.0-SNAPSHOT
25-Feb-2024
Features

Enrichment Processor

  • Computed metrics can be added to events
  • Internal in-memory database can be queried for event enrichment
1.1.0
25-Jan-2024
Features

Predictive Processor

  • JPMML model initialization using local file and remote S3 stores
  • Dynamic model refresh using model update notifications
  • Offline prediction auditing that enables explainability, drift monitoring and model retraining

Avro support

  • Ability to process avro records for inbound and outbound events
  • Complex data types supported using custom mapping
  • Schema registry support

Minio S3 Transport

  • OOTB multi cloud S3 support
  • Publish and consume events and insights to/from hybrid hosted S3 buckets
  • Drive pipeline processing using S3 bucket notifications
  • Consumer supports following file formats: PARQUET, CSV, ARROW, ORC
  • Keep reference data up to date using external systems

Reference Data

  • Apply external data within stream processing tasks
  • In-memory reference data elements kept up-to-date using source change notifications
  • Support for key value and S3 stores
  • Reference data file loader utility

Rest Consumer APIs

  • File consuming endpoint that enable ease of integration to upstream systems
  • Joule event consumer endpoint to provide the ability to chain Joule processors within a cloud environment

File Watcher Consumer

  • File watcher that consumes and processes target files
  • Supported formats; Parquet, Json, CSV, ORC and Arrow IPC

Enhancements

Kafka

  • Confluent schema registry support for outbound events
  • Message partition support
  • Confluent and RedPanda support

Enricher processor

  • Query optimisation
  • SQL, OQL, and Key value enrichment support

Transports

  • Improved exception handling to fail on startup
  • Strict ordering

Apache Arrow

  • Integrated and leveraged to process file efficiently and of various file formats
  • Large file processing support

Optimisations

  • Processing optimisations that reduce both memory and CPU utilisation while increasing event throughput.
  • StreamEvent smart shallow cloning logic to reduce overall memory footprint while providing key data isolation
  • StreamEvent change tracking switch to reduce memory overhead

Upgrades

  • Javalin 5.6.3
  • Kafka 3.6.0
  • Avro 1.11.3
  • DuckDB 0.9.2

Bug Fixes

StreamEventCSVDeserializer
  • Fixed fields from holding only string values to correctly defined data types
  • Allowed for custom date format to be provided
StreamEventJSONDeserializer
  • Can now read an array of Json StreamEvent objects

JVM Configuration Additions

  • Require ‘--add-opens=java.base/java.nio=ALL-UNNAMED’ to be added to the java CLI due to Apache Arrow requirements
  • Applying the G1 GC regionalized and generational garbage collector to improved memory usage
1.1.0-SNAPSHOT
14-Sept
Features
  • Predictive Processor Model initialisation and dynamic model replacement using local file and remote S3 stores
  • Avro support for inbound and outbound events.
  • Complex data types supported using custom field mapping
  • Minio multi-cloud S3 consumer
  • Reference data support
Enhancements
  • MinIO implementation improvements to support large files and notifications
  • Kafka schema registry support
  • Apache Arrow integration to support PARQUET, CSV, ARROW, ORC file formats
  • Better transport exception handling
1.0.4
18-May-2023
Features
  • Feature engineering processor, Including plugins for binning, scaling and declarative analytics expressions
  • Prediction and features auditing enabling explainability, drift monitoring and retraining
  • In-Memory SQL Query API
  • Websocket publisher
Enhancements
  • Improved deep cloning of StreamEvent
  • Reorganisation of SDK
  • Documentation
  • Project template
Bug fixes
  • Window events failing to publish
1.0.4-SNAPSHOT
23-Apr-2023
Docker image generation, scripting fixes, StreamEvent deep copy, documentation updates.
1.0.3
01-Mar-2023
Documentation, metrics engine, Rest API, JDBC publisher, transport and processor updates and bug fixes
1.0.2
30-Jan-2023
Alpha release of metrics engine and rest API
1.0.1
13-May-2022
Bug fixes
1.0.0
05-Apr-2022
Initial release