MinIO S3
MinIO is a High-Performance Object Storage released under GNU Affero General Public License v3.0. It is API compatible with the Amazon S3 cloud storage service. The publisher transport writes bucket objects using a predefined file format.
Driver details
Example configuration
The above example publishes files to a locally hosted S3 bucket using a custom defined data format provided as a avro schema file.
Core Attributes
Available configuration parameters
Attribute | Description | Data Type | Required |
---|---|---|---|
name | Name of source stream | String | |
connection | Connection details | See Connection Attributes section | |
bucket | S3 bucket to ingest object data from | See Bucket Attributes section | |
serializer | Serialization configuration | See Serialization Attributes section | |
batchSize | Number of eventsents to be processed and written to a single file. | Long Default: 1024 |
Connection Attributes
Attribute | Description | Data Type | Required |
---|---|---|---|
endpoint | S3 service endpoint | String Default: https://localhost | |
port | Port the S3 service is hosted on | Integer Default:9000 | |
url | Provide a fully qualified url endpoint, i.e. AWS, GCP, Azure urls. This is used over the endpoint setting if provided. | URL String | |
region | Region where the bucket is to be accessed | String | |
tls | Use a TLS connection | Boolean Default: false | |
credentials | IAM access credentials |
Credentials Attributes
For non-production use cases the access/secret keys can be used to prove data ingestion functionality. When migrating to a production environment implement a provider plugin using the provided JouleProviderPlugin
interface, see basic example below.
Attribute | Description | Data Type | Required |
---|---|---|---|
access key | IAM user access key | String | |
secret key | IAM user secret key | String | |
provider plugin | Custom implementation of credentials ideal for production level deployments | JouleProviderPlugin implementation |
JouleProviderPlugin Interface
Bucket Attributes
Attribute | Description | Data Type | Required |
---|---|---|---|
bucketId | Bucket name | String | |
object name | Object name to listen for events too | String | |
versioning | Ability to use object versioning. Valid values are either ENABLED or SUSPENDED | ENUM | |
bucket policy | Policy file location to be applied | String | |
partition by date | Write files using date partitioning | Boolean Default: true | |
date format | Directory date format to apply when partitioning by date | String Default: yyyyMMdd | |
custom directory | Custom directory to be applied after the date path. Useful when writing multiple objects to to same bucket and date partition but want an independent directory | String | |
headers | Object header information | Map<String,String> | |
user metadata | Applied user metadata to object | Map<String,String> |
Additional Resources
Official MinIO documentation
MinIO Docker image
Last updated