MinIO S3

MinIO is a High-Performance Object Storage released under GNU Affero General Public License v3.0. It is API compatible with the Amazon S3 cloud storage service. The publisher transport writes bucket objects using a predefined file format.

Driver details

io.minio:minio:8.5.4

Example configuration

minioPublisher:
  name: "marketdata-S3Publisher"

  connection:
    endpoint: "https://localhost"
    port: 9000
    credentials:
      access key: "XXXXXX"
      secret key: "YYYYYYYYYYYYYYY"

  bucket:
    bucketId: "marketdata"
    object name: "stocks"
    date format: "yyyyMMdd/HH"
    versioning: ENABLED
    retries: 3

  serializer:
    formatter:
      parquet formatter:
        schema: "./avro/marketDataSchema.avsc"
        temp directory: "./tmp"

  batchSize: 500000  

The above example publishes files to a locally hosted S3 bucket using a custom defined data format provided as a avro schema file.

Core Attributes

Available configuration parameters

AttributeDescriptionData TypeRequired

name

Name of source stream

String

connection

Connection details

bucket

S3 bucket to ingest object data from

See Bucket Attributes section

serializer

Serialization configuration

batchSize

Number of eventsents to be processed and written to a single file.

Long Default: 1024

Connection Attributes

AttributeDescriptionData TypeRequired

endpoint

S3 service endpoint

String Default: https://localhost

port

Port the S3 service is hosted on

Integer Default:9000

url

Provide a fully qualified url endpoint, i.e. AWS, GCP, Azure urls. This is used over the endpoint setting if provided.

URL String

region

Region where the bucket is to be accessed

String

tls

Use a TLS connection

Boolean Default: false

credentials

IAM access credentials

Credentials Attributes

For non-production use cases the access/secret keys can be used to prove data ingestion functionality. When migrating to a production environment implement a provider plugin using the provided JouleProviderPlugin interface, see basic example below.

AttributeDescriptionData TypeRequired

access key

IAM user access key

String

secret key

IAM user secret key

String

provider plugin

Custom implementation of credentials ideal for production level deployments

JouleProviderPlugin implementation

JouleProviderPlugin Interface

public class JWTCredentialsProvider implements JouleProviderPlugin {
    @Override
    public Provider getProvider() {
        return null;
    }

    @Override
    public void initialize() throws CustomPluginException {
    }

    @Override
    public void validate() throws InvalidSpecificationException {
    }

    @Override
    public void setProperties(Properties properties) {
    }
}

Bucket Attributes

AttributeDescriptionData TypeRequired

bucketId

Bucket name

String

object name

Object name to listen for events too

String

versioning

Ability to use object versioning. Valid values are either ENABLED or SUSPENDED

ENUM

bucket policy

Policy file location to be applied

String

partition by date

Write files using date partitioning

Boolean Default: true

date format

Directory date format to apply when partitioning by date

String Default: yyyyMMdd

custom directory

Custom directory to be applied after the date path. Useful when writing multiple objects to to same bucket and date partition but want an independent directory

String

headers

Object header information

Map<String,String>

user metadata

Applied user metadata to object

Map<String,String>

Additional Resources

Last updated