Comment on page
MinIO S3
MinIO is a High-Performance Object Storage released under GNU Affero General Public License v3.0. It is API compatible with the Amazon S3 cloud storage service. The publisher transport writes bucket objects using a predefined file format.
io.minio:minio:8.5.4
minioPublisher:
name: "marketdata-S3Publisher"
connection:
endpoint: "https://localhost"
port: 9000
credentials:
access key: "XXXXXX"
secret key: "YYYYYYYYYYYYYYY"
bucket:
bucketId: "marketdata"
object name: "stocks"
date format: "yyyyMMdd/HH"
versioning: ENABLED
retries: 3
serializer:
formatter:
parquet formatter:
schema: "./avro/marketDataSchema.avsc"
temp directory: "./tmp"
batchSize: 500000
The above example publishes files to a locally hosted S3 bucket using a custom defined data format provided as a avro schema file.
Available configuration parameters
Attribute | Description | Data Type | Required |
---|---|---|---|
name | Name of source stream | String | |
connection | Connection details | ||
bucket | S3 bucket to ingest object data from | ||
serializer | Serialization configuration | ||
batchSize | Number of eventsents to be processed and written to a single file. | Long
Default: 1024 |
Attribute | Description | Data Type | Required |
---|---|---|---|
endpoint | S3 service endpoint | String
Default: https://localhost | |
port | Port the S3 service is hosted on | Integer
Default:9000 | |
url | Provide a fully qualified url endpoint, i.e. AWS, GCP, Azure urls. This is used over the endpoint setting if provided. | URL String | |
region | Region where the bucket is to be accessed | String | |
tls | Use a TLS connection | Boolean
Default: false | |
credentials | IAM access credentials |
For non-production use cases the access/secret keys can be used to prove data ingestion functionality. When migrating to a production environment implement a provider plugin using the provided
JouleProviderPlugin
interface, see basic example below.Attribute | Description | Data Type | Required |
---|---|---|---|
access key | IAM user access key | String | |
secret key | IAM user secret key | String | |
provider plugin | Custom implementation of credentials ideal for production level deployments | JouleProviderPlugin implementation |
public class JWTCredentialsProvider implements JouleProviderPlugin {
@Override
public Provider getProvider() {
return null;
}
@Override
public void initialize() throws CustomPluginException {
}
@Override
public void validate() throws InvalidSpecificationException {
}
@Override
public void setProperties(Properties properties) {
}
}
Attribute | Description | Data Type | Required |
---|---|---|---|
bucketId | Bucket name | String | |
object name | Object name to listen for events too | String | |
versioning | Ability to use object versioning. Valid values are either ENABLED or SUSPENDED | ENUM | |
bucket policy | Policy file location to be applied | String | |
partition by date | Write files using date partitioning | Boolean
Default: true | |
date format | Directory date format to apply when partitioning by date | String
Default: yyyyMMdd | |
custom directory | Custom directory to be applied after the date path. Useful when writing multiple objects to to same bucket and date partition but want an independent directory | String | |
headers | Object header information | Map<String,String> | |
user metadata | Applied user metadata to object | Map<String,String> |
Last modified 2mo ago