MinIO S3 Backup

The MinIO publisher transport publishes batches of events as formatted files on to configured S3 local or cloud based buckets.

Driver details

io.minio:minio:8.5.4

Example configuration

minioPublisher:
  name: "marketdata-S3Publisher"

  connection:
    endpoint: "https://localhost"
    port: 9000
    tls: false
    credentials:
      access key: "xxxx"
      secret key: "yyyy"

  bucket persistence:
    bucketId: "marketdata"
    object name: "marketdataPublisher"
    date format: "yyyyMMdd/HH"
    versioning: ENABLED
    retries: 3

    formatter:
      parquet formatter:
        contentType: "binary/octet-stream"
        encoding: "UTF-8"
        schema: "./marketdata.avro"
        compression codec: SNAPPY
        temp directory: "./tmp"

  batchSize: 100000

The above example uses a local MinIO platform where by a configured S3 bucket receives compressed batched parquet formatted files.

Core Attributes

Configuration parameters available for the MinIO publisher transport. The parameters are organised by order of importance, ranked from high to low.

Attribute	Description	Data Type	Required
connection	Connection setting to hosted service	See Connection Attributes section
bucket persistence	Bucket persistence configurations	See Bucket Attributes section
batchSize	Number of events to batch send, maps to batch.size. Batch size of zero disables batching function	Integer Default: 100000

Connection Attributes

This topic provides configuration parameters available for the connection section.

Attribute	Description	Data Type	Required
region	Target region for S3 bucket hosting	String
endpoint	Service address i.e. https://storage.googleapis.com	String Default: https://localhost
port	Port service is hosted on. This can be provided within the endpoint setting. Valid range is between 1025 to 65536	Integer Default: 9000
tls	Use TLS security	Boolean Default: false
credentials	The credentials to use to login to the platform	See Credentials Attributes

Credentials Attributes

This topic provides configuration parameters available for the credentials section.

Attribute	Description	Data Type	Required
access key	Client access key	String
secret key	Client secret key	String
provider plugin	Custom plugin that provides a MinIO provider class implementation	String e.g. Full package and class name

If a provider plugin has been provided the access and secret keys are not required

Custom Credential Providers

For production based solutions credentials need to be provided by a secure environment. Joule provides a JouleProviderPlugin interface for custom implementations.

Example

minioPublisher:
  name: "marketdata-S3Publisher"

  connection:
    endpoint: "https://localhost"
    port: 9000
    tls: true
    credentials:
      provider plugin: com.fractalworks.streams.transport.minio.credentials.JWTCredentialsProvider

Custom Provider implementation

The complete implementation of this class would provide MinIO with a custom credentials Provider instance required of an enterprise production solution. See MinIO Github for implementation examples.

public class JWTCredentialsProvider implements JouleProviderPlugin {
    @Override
    public Provider getProvider() {
        // TODO: Implementation
        return null;
    }

    @Override
    public void initialize() throws CustomPluginException {
        // TODO: Implementation
    }

    @Override
    public void validate() throws InvalidSpecificationException {
        // TODO: Implementation
    }

    @Override
    public void setProperties(Properties properties) {
        // TODO: Implementation
    }
}

Bucket Attributes

Attribute	Description	Data Type	Required
bucketId	Bucket name	String
object name	File name to be used	String
formatter	Formatter to use to convert StreamEvents to target data formatter	See Formatters documentation Default: csv formatter
versioning	Keep multiple variants of an object in the same bucket Valid values: ENABLED, SUSPENDED	String Default: ENABLED
bucket policy	Bucket policy to use. Provide path and file to policy	String
partition by date	Create bucket directories based upon date	Boolean Default: True
date format	Date format to use when enabling "partition by date" feature	String Default: yyyyMMdd
custom directory	Specify a custom defined directory path for files to be placed within	String
headers	File header information	Map<String,String>
user metadata	User meta data to be applied to files	Map<String,String>
retries	Number of time to try to publish file	Integer Default: 3