Joule
Search
K
Comment on page

MinIO S3 Backup

The MinIO publisher transport publishes batches of events as formatted files on to configured S3 local or cloud based buckets.

Driver details

io.minio:minio:8.5.4

Example configuration

minioPublisher:
name: "marketdata-S3Publisher"
connection:
endpoint: "https://localhost"
port: 9000
tls: false
credentials:
access key: "xxxx"
secret key: "yyyy"
bucket persistence:
bucketId: "marketdata"
object name: "marketdataPublisher"
date format: "yyyyMMdd/HH"
versioning: ENABLED
retries: 3
formatter:
parquet formatter:
contentType: "binary/octet-stream"
encoding: "UTF-8"
schema: "./marketdata.avro"
compression codec: SNAPPY
temp directory: "./tmp"
batchSize: 100000
The above example uses a local MinIO platform where by a configured S3 bucket receives compressed batched parquet formatted files.

Core Attributes

Configuration parameters available for the MinIO publisher transport. The parameters are organised by order of importance, ranked from high to low.
Attribute
Description
Data Type
Required
connection
Connection setting to hosted service
bucket persistence
Bucket persistence configurations
See Bucket Attributes section
batchSize
Number of events to batch send, maps to batch.size. Batch size of zero disables batching function
Integer
Default: 100000

Connection Attributes

This topic provides configuration parameters available for the connection section.
Attribute
Description
Data Type
Required
region
Target region for S3 bucket hosting
String
endpoint
Service address i.e. https://storage.googleapis.com
String
Default: https://localhost
port
Port service is hosted on. This can be provided within the endpoint setting. Valid range is between 1025 to 65536
Integer Default: 9000
tls
Use TLS security
Boolean Default: false
credentials
The credentials to use to login to the platform

Credentials Attributes

This topic provides configuration parameters available for the credentials section.
Attribute
Description
Data Type
Required
access key
Client access key
String
secret key
Client secret key
String
provider plugin
Custom plugin that provides a MinIO provider class implementation
String e.g. Full package and class name
If a provider plugin has been provided the access and secret keys are not required

Custom Credential Providers

For production based solutions credentials need to be provided by a secure environment. Joule provides a JouleProviderPlugin interface for custom implementations.

Example

minioPublisher:
name: "marketdata-S3Publisher"
connection:
endpoint: "https://localhost"
port: 9000
tls: true
credentials:
provider plugin: com.fractalworks.streams.transport.minio.credentials.JWTCredentialsProvider

Custom Provider implementation

The complete implementation of this class would provide MinIO with a custom credentials Provider instance required of an enterprise production solution. See MinIO Github for implementation examples.
public class JWTCredentialsProvider implements JouleProviderPlugin {
@Override
public Provider getProvider() {
// TODO: Implementation
return null;
}
@Override
public void initialize() throws CustomPluginException {
// TODO: Implementation
}
@Override
public void validate() throws InvalidSpecificationException {
// TODO: Implementation
}
@Override
public void setProperties(Properties properties) {
// TODO: Implementation
}
}

Bucket Attributes

Attribute
Description
Data Type
Required
bucketId
Bucket name
String
object name
File name to be used
String
formatter
Formatter to use to convert StreamEvents to target data formatter
See Formatters documentation Default: csv formatter
versioning
Keep multiple variants of an object in the same bucket Valid values: ENABLED, SUSPENDED
String Default: ENABLED
bucket policy
Bucket policy to use. Provide path and file to policy
String
partition by date
Create bucket directories based upon date
Boolean Default: True
date format
Date format to use when enabling "partition by date" feature
String Default: yyyyMMdd
custom directory
Specify a custom defined directory path for files to be placed within
String
headers
File header information
Map<String,String>
user metadata
User meta data to be applied to files
Map<String,String>
retries
Number of time to try to publish file
Integer Default: 3