MinIO S3

Multi-cloud S3-compatible interface for static contextual data and ML models

Overview

The MinIO S3 driver in Joule provides a flexible storage solution for integrating static contextual data within a streaming context.

This storage type is ideal for very slow-changing data, which can be cached and used for enriching events without frequent updates.

With S3-compatible storage, developers can leverage existing data assets, making it easy to add value to real-time streaming applications.

Driver details io.minio:minio:8.5.4

Use cases

The MinIO S3 storage is well-suited for various forms of static data, including industry standards, organisational contextual data and pre-computed models.

Typical use cases include:

1. Geographic and regional data Postal codes, country codes (e.g., ISO-366).

2. Device and market data Mobile manufacture models, market exchange codes.

3. Currency data Currency codes for international transactions.

4. Machine learning models Host and deploy pre-trained models in to Joule for real-time predictions.

5. Static variables and codes Charge codes, car VINs, pre-computed data for analysis.

6. Pattern matching Regex patterns for validating inputs, such as phone numbers.

Example & DSL attributes

The following example is a sample configuration that uses the MinIO S3 connector to load a machine learning model on startup.

This setup leverages specific credentials and connection details to access and download a model from a defined bucket, ensuring that it’s available in the processing pipeline.

contextual data:
 ...
    - minio stores:
        name: Telco ML Models
        connection:
          endpoint: "https://localhost"
          port: 9000
          tls: false
          credentials:
            access key: "XXXXXXX"
            secret key: "YYYYYYYYYYYYYYY"
        stores:
          predictors:
            bucketId: models
            initial version Id: 12345
            download dir: /home/joule/telco-models/tmp

Explanation

This configuration specifies an S3 bucket (models) as the data source for machine learning models.

On startup, the model with the defined version (12345) is downloaded to the local directory (/home/joule/telco-models/tmp).

The pmml predictor processor reference implementation is then used to apply the model within Joule’s processing framework.

For further details on configuring this processor, refer to the PMML Predictor Documentation.

Attributes schema

Attribute	Description	Data Type	Required
name	Logical name of contextual data set for given target connection	String
connection	Connection details	See Connection configuration
stores	Map of logical names to bucket level configurations. See bucket attributes section	Map

Bucket Attributes

Attribute	Description	Data Type	Required
bucketId	S3 bucket where the reference data resides	String
initial version id	The version id to use when priming the system. This is a global version for the given bucket otherwise the latest version of the object is used	String
download dir	Local path for bucket objects to be downloaded before processing starts	String Default: ./tmp