Comment on page

MinIO S3 Backup

The MinIO publisher transport publishes batches of events as formatted files on to configured S3 local or cloud based buckets.
Driver details
io.minio:minio:8.5.4
Example configuration
minioPublisher:
  name: "marketdata-S3Publisher"
​
  connection:
    endpoint: "https://localhost"
    port: 9000
    tls: false
    credentials:
      access key: "xxxx"
      secret key: "yyyy"
​
  bucket persistence:
    bucketId: "marketdata"
    object name: "marketdataPublisher"
    date format: "yyyyMMdd/HH"
    versioning: ENABLED
    retries: 3
​
    formatter:
      parquet formatter:
        contentType: "binary/octet-stream"
        encoding: "UTF-8"
        schema: "./marketdata.avro"
        compression codec: SNAPPY
        temp directory: "./tmp"
​
  batchSize: 100000
The above example uses a local MinIO platform where by a configured S3 bucket receives compressed batched parquet formatted files.
Core Attributes
Configuration parameters available for the MinIO publisher transport. The parameters are organised by order of importance, ranked from high to low.
Attribute
Description
Data Type
Required
connection 
Connection setting to hosted service
See Connection Attributes section
bucket persistence
Bucket persistence configurations
See Bucket Attributes section
batchSize
Number of events to batch send, maps to batch.size. Batch size of zero disables batching function
Integer
Default: 100000
Connection Attributes
This topic provides configuration parameters available for the connection section. 
Attribute
Description
Data Type
Required
region
Target region for S3 bucket hosting
String
endpoint
Service address i.e.
https://storage.googleapis.com​
String
Default: https://localhost 
port
Port service is hosted on. This can be provided within the endpoint setting. Valid range is between 1025 to 65536
Integer
Default: 9000
tls
Use TLS security
Boolean
Default: false
credentials
The credentials to use to login to the platform
See Credentials Attributes​
Credentials Attributes
This topic provides configuration parameters available for the credentials section. 
Attribute
Description
Data Type
Required
access key
Client access key
String
secret key
Client secret key
String 
provider plugin
Custom plugin that provides a MinIO provider class implementation
String
e.g. Full package and class name
If a provider plugin has been provided the access and secret keys are not required
Custom Credential Providers
For production based solutions credentials need to be provided by a secure environment. Joule provides a JouleProviderPlugin interface for custom implementations.
Example
minioPublisher:
  name: "marketdata-S3Publisher"
​
  connection:
    endpoint: "https://localhost"
    port: 9000
    tls: true
    credentials:
      provider plugin: com.fractalworks.streams.transport.minio.credentials.JWTCredentialsProvider
Custom Provider implementation 
The complete implementation of this class would provide MinIO with a custom credentials Provider instance required of an enterprise production solution. See MinIO Github for implementation examples.
public class JWTCredentialsProvider implements JouleProviderPlugin {
    @Override
    public Provider getProvider() {
        // TODO: Implementation
        return null;
    }
​
    @Override
    public void initialize() throws CustomPluginException {
        // TODO: Implementation
    }
​
    @Override
    public void validate() throws InvalidSpecificationException {
        // TODO: Implementation
    }
​
    @Override
    public void setProperties(Properties properties) {
        // TODO: Implementation
    }
}
Bucket Attributes
Attribute
Description
Data Type
Required
bucketId
Bucket name
String
object name
File name to be used
String 
formatter
Formatter to use to convert StreamEvents to target data formatter
See Formatters documentation
Default: csv formatter
versioning
Keep multiple variants of an object in the same bucket

Valid values: ENABLED, SUSPENDED
String
Default: ENABLED
bucket policy
Bucket policy to use. Provide path and file to policy
String
partition by date
Create bucket directories based upon date 
Boolean
Default: True
date format
Date format to use when enabling "partition by date" feature
String
Default: yyyyMMdd
custom directory
Specify a custom defined directory path for files to be placed within
String
headers
File header information
Map<String,String>
user metadata
User meta data to be applied to files
Map<String,String>
retries
Number of time to try to publish file
Integer
Default: 3

Attribute	Description	Data Type
connection	Connection setting to hosted service	See Connection Attributes section
bucket persistence	Bucket persistence configurations	See Bucket Attributes section
batchSize	Number of events to batch send, maps to batch.size. Batch size of zero disables batching function	Integer Default: 100000

Attribute	Description	Data Type
region	Target region for S3 bucket hosting	String
endpoint	Service address i.e. https://storage.googleapis.com	String Default: https://localhost
port	Port service is hosted on. This can be provided within the endpoint setting. Valid range is between 1025 to 65536	Integer Default: 9000
tls	Use TLS security	Boolean Default: false
credentials	The credentials to use to login to the platform	See Credentials Attributes

Attribute	Description	Data Type
access key	Client access key	String
secret key	Client secret key	String
provider plugin	Custom plugin that provides a MinIO provider class implementation	String e.g. Full package and class name

Attribute	Description	Data Type
bucketId	Bucket name	String
object name	File name to be used	String
formatter	Formatter to use to convert StreamEvents to target data formatter	See Formatters documentation Default: csv formatter
versioning	Keep multiple variants of an object in the same bucket Valid values: ENABLED, SUSPENDED	String Default: ENABLED
bucket policy	Bucket policy to use. Provide path and file to policy	String
partition by date	Create bucket directories based upon date	Boolean Default: True
date format	Date format to use when enabling "partition by date" feature	String Default: yyyyMMdd
custom directory	Specify a custom defined directory path for files to be placed within	String
headers	File header information	Map<String,String>
user metadata	User meta data to be applied to files	Map<String,String>
retries	Number of time to try to publish file	Integer Default: 3

Reference - Previous

Change history

Last modified 4mo ago