MongoDB

Publish JSON based events to the mature standard document storePublishes JSON-based events to MongoDB, a widely-used document store

Overview

Document database support is provided using the MongoDB Publisher Transport. Processed events are published to a specified MongoDB database collection.

The events are serialised using a document formatter serialiser or a custom transformer implementation.

This transport is designed to facilitate efficient data publishing into MongoDB with customisable serialisation options.

Some key features are:

1. MongoDB integration Sends processed events to a defined MongoDB collection.

2. Serialiser options Uses a default Document Formatter serialiser or allows custom serialisation.

Driver details: org.mongodb:mongodb-driver-sync:4.8.1

Examples & DSL attributes

This example sets up the MongoDB Publisher Transport to send events to a local MongoDB instance (localhost:27017).

It stores data in the customerdb database and emea_customers collection, using ordered inserts.

Authentication is done with the joule user and SCRAM-SHA-1 mechanism. Events are serialised individually (not in batches) with Base64 encoding and application/octet-stream content type. SSL is disabled.

mongodbPublisher:
  servers:
    localhost: 27017
  enable ssl: false

  database: customerdb
  collection: emea_customers
  ordered inserts: true

  credentials:
    user: joule
    password: password
    mechanism: SCRAM_SHA_1

  serializer:
    batch: false
    formatter:
      documentFormatter:
        contentType: application/octet-stream
        encoding: base64

Attributes schema

Attribute	Description	Data Type	Required
database	Name of database	String
collection	Name of collection inserts will be applied to	String
ordered inserts	Hint to MongDB to insert documents using presented ordering	Boolean Default: true
servers	Map of servers with associated ports	Map<String,Integer>
enable ssl	Enable SSL	boolean Default: false
credentials	Credential configuration	See Credential attributes section
serializer	Serialisation configuration	See Serialisation attributes section

Credential attributes schema

Attribute	Description	Data Type	Required
username	Username	String
password	password	String
authentication database	Database to use to authenticate connected user	String Default: admin
mechanism	Authentication mechanism Options PLAIN SCRAM_SHA_1 SCRAM_SHA_256	String Default: SCRAM_SHA_1

Serialisation attributes schema

This topic provides configuration parameters available object serialisation process.

Attribute	Description	Data Type	Required
transform	User provided implementation	CustomTransformer
formatter	This is used when a custom transformer is not provided. Useful when chaining Joule processes	Formatter Default: documentFormatter (See Document Formatter)
contentType	Type of content to inform receiving application	String Default: application/octet-stream
encoding	Payload encoding method	String Default: base64
batch	Flag to batch multiple messages in to a single payload	Boolean Default: false

Serialiser example

The configuration below will serialise StreamEvents as MongoDB documents.

mongodbPublisher:
  ...

  serializer:
    batch: false
    formatter:
      documentFormatter:
        contentType: application/octet-stream
        encoding: base64

Document formatter

The document formatter in Joule is a built-in tool that converts StreamEvents into MongoDB's Document type.

This allows the processed events to be formatted correctly before being inserted into a MongoDB collection. The formatter ensures that the data is serialised in a way that MongoDB can understand and store efficiently.

Additional resources

• Official MongoDB documentation

• Official MongoDB docker image

• Mongo Compass UI