Comment on page

MinIO S3

MinIO is a High-Performance Object Storage released under GNU Affero General Public License v3.0. It is API compatible with the Amazon S3 cloud storage service. The publisher transport writes  bucket objects using a predefined file format.
Driver details
io.minio:minio:8.5.4
Example configuration
minioPublisher:
  name: "marketdata-S3Publisher"
​
  connection:
    endpoint: "https://localhost"
    port: 9000
    credentials:
      access key: "XXXXXX"
      secret key: "YYYYYYYYYYYYYYY"
​
  bucket:
    bucketId: "marketdata"
    object name: "stocks"
    date format: "yyyyMMdd/HH"
    versioning: ENABLED
    retries: 3
​
  serializer:
    formatter:
      parquet formatter:
        schema: "./avro/marketDataSchema.avsc"
        temp directory: "./tmp"
​
  batchSize: 500000  
The above example publishes files to a locally hosted S3 bucket using a custom defined data format provided as a avro schema file.
Core Attributes
Available configuration parameters
Attribute
Description
Data Type
Required
name
Name of source stream 
String
connection 
Connection details
See Connection Attributes section
bucket
S3 bucket to ingest object data from
See Bucket Attributes section
serializer
Serialization configuration
See Serialization Attributes section
batchSize
Number of eventsents to be processed and written to a single file.
Long
Default:  1024
Connection Attributes
Attribute
Description
Data Type
Required
endpoint
S3 service endpoint
String
Default: https://localhost
port
Port the S3 service is hosted on
Integer
Default:9000
url
Provide a fully qualified url endpoint, i.e. AWS, GCP, Azure urls. This is used over the endpoint setting if provided.
URL String
region
Region where the bucket is to be accessed
String
tls
Use a TLS connection
Boolean
Default: false
credentials
IAM access credentials
See Credentials section​
Credentials Attributes
For non-production use cases the access/secret keys can be used to prove data ingestion functionality. When migrating to a production environment implement a provider plugin using the provided JouleProviderPlugin interface, see basic example below.
Attribute
Description
Data Type
Required
access key
IAM user access key
String
secret key
IAM user secret key
String
provider plugin
Custom implementation of credentials ideal for production level deployments
JouleProviderPlugin implementation
JouleProviderPlugin Interface
public class JWTCredentialsProvider implements JouleProviderPlugin {
    @Override
    public Provider getProvider() {
        return null;
    }
​
    @Override
    public void initialize() throws CustomPluginException {
    }
​
    @Override
    public void validate() throws InvalidSpecificationException {
    }
​
    @Override
    public void setProperties(Properties properties) {
    }
}
Bucket Attributes
Attribute
Description
Data Type
Required
bucketId
Bucket name
String
object name
Object name to listen for events too
String
versioning
Ability to use object versioning. Valid values are either ENABLED or SUSPENDED
ENUM
bucket policy
Policy file location to be applied
String
partition by date
Write files using date partitioning
Boolean
Default: true
date format
Directory date format to apply when partitioning by date
String
Default: yyyyMMdd
custom directory
Custom directory to be applied after the date path. Useful when writing multiple objects to to same bucket and date partition but want an independent directory
String
headers
Object header information
Map<String,String>
user metadata
Applied user metadata to object
Map<String,String>
Additional Resources
Official MinIO documentation​
MinIO Docker image​

Attribute	Description	Data Type
name	Name of source stream	String
connection	Connection details	See Connection Attributes section
bucket	S3 bucket to ingest object data from	See Bucket Attributes section
serializer	Serialization configuration	See Serialization Attributes section
batchSize	Number of eventsents to be processed and written to a single file.	Long Default: 1024

Attribute	Description	Data Type
endpoint	S3 service endpoint	String Default: https://localhost
port	Port the S3 service is hosted on	Integer Default:9000
url	Provide a fully qualified url endpoint, i.e. AWS, GCP, Azure urls. This is used over the endpoint setting if provided.	URL String
region	Region where the bucket is to be accessed	String
tls	Use a TLS connection	Boolean Default: false
credentials	IAM access credentials	See Credentials section

Attribute	Description	Data Type
access key	IAM user access key	String
secret key	IAM user secret key	String
provider plugin	Custom implementation of credentials ideal for production level deployments	JouleProviderPlugin implementation

Attribute	Description	Data Type
bucketId	Bucket name	String
object name	Object name to listen for events too	String
versioning	Ability to use object versioning. Valid values are either ENABLED or SUSPENDED	ENUM
bucket policy	Policy file location to be applied	String
partition by date	Write files using date partitioning	Boolean Default: true
date format	Directory date format to apply when partitioning by date	String Default: yyyyMMdd
custom directory	Custom directory to be applied after the date path. Useful when writing multiple objects to to same bucket and date partition but want an independent directory	String
headers	Object header information	Map<String,String>
user metadata	Applied user metadata to object	Map<String,String>

SQL Databases

InfluxDB

Last modified 2mo ago