Joule
  • Welcome to Joule's Docs
  • Why Joule?
    • Joule capabilities
  • What is Joule?
    • Key features
    • The tech stack
  • Use case enablement
    • Use case building framework
  • Concepts
    • Core concepts
    • Low code development
    • Unified execution engine
    • Batch and stream processing
    • Continuous metrics
    • Key Joule data types
      • StreamEvent object
      • Contextual data
      • GeoNode
  • Tutorials
    • Getting started
    • Build your first use case
    • Stream sliding window quote analytics
    • Advanced tutorials
      • Custom missing value processor
      • Stateless Bollinger band analytics
      • IoT device control
  • FAQ
  • Glossary
  • Components
    • Pipelines
      • Use case anatomy
      • Data priming
        • Types of import
      • Processing unit
      • Group by
      • Emit computed events
      • Telemetry auditing
    • Processors
      • Common attributes
      • Filters
        • By type
        • By expression
        • Send on delta
        • Remove attributes
        • Drop all events
      • Enrichment
        • Key concepts
          • Anatomy of enrichment DSL
          • Banking example
        • Metrics
        • Dynamic contextual data
          • Caching architecture
        • Static contextual data
      • Transformation
        • Field Tokeniser
        • Obfuscation
          • Encryption
          • Masking
          • Bucketing
          • Redaction
      • Triggers
        • Change Data Capture
        • Business rules
      • Stream join
        • Inner stream joins
        • Outer stream joins
        • Join attributes & policy
      • Event tap
        • Anatomy of a Tap
        • SQL Queries
    • Analytics
      • Analytic tools
        • User defined analytics
          • Streaming analytics example
          • User defined analytics
          • User defined scripts
          • User defined functions
            • Average function library
        • Window analytics
          • Tumbling window
          • Sliding window
          • Aggregate functions
        • Analytic functions
          • Stateful
            • Exponential moving average
            • Rolling Sum
          • Stateless
            • Normalisation
              • Absolute max
              • Min max
              • Standardisation
              • Mean
              • Log
              • Z-Score
            • Scaling
              • Unit scale
              • Robust Scale
            • Statistics
              • Statistic summaries
              • Weighted moving average
              • Simple moving average
              • Count
            • General
              • Euclidean
        • Advanced analytics
          • Geospatial
            • Entity geo tracker
            • Geofence occupancy trigger
            • Geo search
            • IP address resolver
            • Reverse geocoding
            • Spatial Index
          • HyperLogLog
          • Distinct counter
      • ML inferencing
        • Feature engineering
          • Scripting
          • Scaling
          • Transform
        • Online predictive analytics
        • Model audit
        • Model management
      • Metrics engine
        • Create metrics
        • Apply metrics
        • Manage metrics
        • Priming metrics
    • Contextual data
      • Architecture
      • Configuration
      • MinIO S3
      • Apache Geode
    • Connectors
      • Sources
        • Kafka
          • Ingestion
        • RabbitMQ
          • Further RabbitMQ configurations
        • MQTT
          • Topic wildcards
          • Session management
          • Last Will and Testament
        • Rest endpoints
        • MinIO S3
        • File watcher
      • Sinks
        • Kafka
        • RabbitMQ
          • Further configurations
        • MQTT
          • Persistent messaging
          • Last Will and Testament
        • SQL databases
        • InfluxDB
        • MongoDB
        • Geode
        • WebSocket endpoint
        • MinIO S3
        • File transport
        • Slack
        • Email
      • Serialisers
        • Serialisation
          • Custom transform example
          • Formatters
        • Deserialisers
          • Custom parsing example
    • Observability
      • Enabling JMX for Joule
      • Meters
      • Metrics API
  • DEVELOPER GUIDES
    • Setting up developer environment
      • Environment setup
      • Build and deploy
      • Install Joule
        • Install Docker demo environment
        • Install with Docker
        • Install from source
        • Install Joule examples
    • Joulectl CLI
    • API Endpoints
      • Mangement API
        • Use case
        • Pipelines
        • Data connectors
        • Contextual data
      • Data access API
        • Query
        • Upload
        • WebSocket
      • SQL support
    • Builder SDK
      • Connector API
        • Sources
          • StreamEventParser API
        • Sinks
          • CustomTransformer API
      • Processor API
      • Analytics API
        • Create custom metrics
        • Define analytics
        • Windows API
        • SQL queries
      • Transformation API
        • Obfuscation API
        • FieldTokenizer API
      • File processing
      • Data types
        • StreamEvent
        • ReferenceDataObject
        • GeoNode
    • System configuration
      • System properties
  • Deployment strategies
    • Deployment Overview
    • Single Node
    • Cluster
    • GuardianDB
    • Packaging
      • Containers
      • Bare metal
  • Product updates
    • Public Roadmap
    • Release Notes
      • v1.2.0 Join Streams with stateful analytics
      • v1.1.0 Streaming analytics enhancements
      • v1.0.4 Predictive stream processing
      • v1.0.3 Contextual SQL based metrics
    • Change history
Powered by GitBook
On this page
  • Overview
  • Use cases
  • Example & DSL attributes
  • Explanation
  • Attributes schema
  • Bucket Attributes

Was this helpful?

  1. Components
  2. Contextual data

MinIO S3

Multi-cloud S3-compatible interface for static contextual data and ML models

PreviousConfigurationNextApache Geode

Last updated 5 months ago

Was this helpful?

Overview

The MinIO S3 driver in Joule provides a flexible storage solution for integrating static contextual data within a streaming context.

This storage type is ideal for very slow-changing data, which can be cached and used for enriching events without frequent updates.

With S3-compatible storage, developers can leverage existing data assets, making it easy to add value to real-time streaming applications.

Use cases

The MinIO S3 storage is well-suited for various forms of static data, including industry standards, organisational contextual data and pre-computed models.

Typical use cases include:

  1. Geographic and regional data Postal codes, country codes (e.g., ISO-366).

  2. Device and market data Mobile manufacture models, market exchange codes.

  3. Currency data Currency codes for international transactions.

  4. Machine learning models Host and deploy pre-trained models in to Joule for real-time predictions.

  5. Static variables and codes Charge codes, car VINs, pre-computed data for analysis.

  6. Pattern matching Regex patterns for validating inputs, such as phone numbers.

Example & DSL attributes

The following example is a sample configuration that uses the MinIO S3 connector to load a machine learning model on startup.

This setup leverages specific credentials and connection details to access and download a model from a defined bucket, ensuring that it’s available in the processing pipeline.

contextual data:
 ...
    - minio stores:
        name: Telco ML Models
        connection:
          endpoint: "https://localhost"
          port: 9000
          tls: false
          credentials:
            access key: "XXXXXXX"
            secret key: "YYYYYYYYYYYYYYY"
        stores:
          predictors:
            bucketId: models
            initial version Id: 12345
            download dir: /home/joule/telco-models/tmp

Explanation

This configuration specifies an S3 bucket (models) as the data source for machine learning models.

On startup, the model with the defined version (12345) is downloaded to the local directory (/home/joule/telco-models/tmp).

The pmml predictor processor reference implementation is then used to apply the model within Joule’s processing framework.

Attributes schema

Attribute
Description
Data Type
Required

name

Logical name of contextual data set for given target connection

String

connection

Connection details

stores

Map

Bucket Attributes

Attribute
Description
Data Type
Required

bucketId

S3 bucket where the reference data resides

String

initial version id

The version id to use when priming the system. This is a global version for the given bucket otherwise the latest version of the object is used

String

download dir

Local path for bucket objects to be downloaded before processing starts

String Default: ./tmp

For further details on configuring this processor, refer to the .

Map of logical names to bucket level configurations.

Driver details io.minio:minio:8.5.4
PMML Predictor Documentation
See bucket attributes section
See Connection configuration