Comment on page
MinIO S3 Backup
The MinIO publisher transport publishes batches of events as formatted files on to configured S3 local or cloud based buckets.
io.minio:minio:8.5.4
minioPublisher:
name: "marketdata-S3Publisher"
connection:
endpoint: "https://localhost"
port: 9000
tls: false
credentials:
access key: "xxxx"
secret key: "yyyy"
bucket persistence:
bucketId: "marketdata"
object name: "marketdataPublisher"
date format: "yyyyMMdd/HH"
versioning: ENABLED
retries: 3
formatter:
parquet formatter:
contentType: "binary/octet-stream"
encoding: "UTF-8"
schema: "./marketdata.avro"
compression codec: SNAPPY
temp directory: "./tmp"
batchSize: 100000
The above example uses a local MinIO platform where by a configured S3 bucket receives compressed batched parquet formatted files.
Configuration parameters available for the MinIO publisher transport. The parameters are organised by order of importance, ranked from high to low.
Attribute | Description | Data Type | Required |
---|---|---|---|
connection | Connection setting to hosted service | ||
bucket persistence | Bucket persistence configurations | ||
batchSize | Number of events to batch send, maps to batch.size. Batch size of zero disables batching function | Integer Default: 100000 |
This topic provides configuration parameters available for the connection section.
Attribute | Description | Data Type | Required |
---|---|---|---|
region | Target region for S3 bucket hosting | String | |
endpoint | String Default: https://localhost | ||
port | Port service is hosted on. This can be provided within the endpoint setting. Valid range is between 1025 to 65536 | Integer
Default: 9000 | |
tls | Use TLS security | Boolean
Default: false | |
credentials | The credentials to use to login to the platform |
This topic provides configuration parameters available for the credentials section.
Attribute | Description | Data Type | Required |
---|---|---|---|
access key | Client access key | String | |
secret key | Client secret key | String | |
provider plugin | Custom plugin that provides a MinIO provider class implementation | String
e.g. Full package and class name |
If a provider plugin has been provided the access and secret keys are not required
For production based solutions credentials need to be provided by a secure environment. Joule provides a
JouleProviderPlugin
interface for custom implementations.minioPublisher:
name: "marketdata-S3Publisher"
connection:
endpoint: "https://localhost"
port: 9000
tls: true
credentials:
provider plugin: com.fractalworks.streams.transport.minio.credentials.JWTCredentialsProvider
The complete implementation of this class would provide MinIO with a custom credentials Provider instance required of an enterprise production solution. See MinIO Github for implementation examples.
public class JWTCredentialsProvider implements JouleProviderPlugin {
@Override
public Provider getProvider() {
// TODO: Implementation
return null;
}
@Override
public void initialize() throws CustomPluginException {
// TODO: Implementation
}
@Override
public void validate() throws InvalidSpecificationException {
// TODO: Implementation
}
@Override
public void setProperties(Properties properties) {
// TODO: Implementation
}
}
Attribute | Description | Data Type | Required |
---|---|---|---|
bucketId | Bucket name | String | |
object name | File name to be used | String | |
formatter | Formatter to use to convert StreamEvents to target data formatter | ||
versioning | Keep multiple variants of an object in the same bucket
Valid values: ENABLED, SUSPENDED | String
Default: ENABLED | |
bucket policy | Bucket policy to use. Provide path and file to policy | String | |
partition by date | Create bucket directories based upon date | Boolean
Default: True | |
date format | Date format to use when enabling "partition by date" feature | String
Default: yyyyMMdd | |
custom directory | Specify a custom defined directory path for files to be placed within | String | |
headers | File header information | Map<String,String> | |
user metadata | User meta data to be applied to files | Map<String,String> | |
retries | Number of time to try to publish file | Integer
Default: 3 |
Last modified 4mo ago