Joule
  • Welcome to Joule's Docs
  • Why Joule?
    • Joule capabilities
  • What is Joule?
    • Key features
    • The tech stack
  • Use case enablement
    • Use case building framework
  • Concepts
    • Core concepts
    • Low code development
    • Unified execution engine
    • Batch and stream processing
    • Continuous metrics
    • Key Joule data types
      • StreamEvent object
      • Contextual data
      • GeoNode
  • Tutorials
    • Getting started
    • Build your first use case
    • Stream sliding window quote analytics
    • Advanced tutorials
      • Custom missing value processor
      • Stateless Bollinger band analytics
      • IoT device control
  • FAQ
  • Glossary
  • Components
    • Pipelines
      • Use case anatomy
      • Data priming
        • Types of import
      • Processing unit
      • Group by
      • Emit computed events
      • Telemetry auditing
    • Processors
      • Common attributes
      • Filters
        • By type
        • By expression
        • Send on delta
        • Remove attributes
        • Drop all events
      • Enrichment
        • Key concepts
          • Anatomy of enrichment DSL
          • Banking example
        • Metrics
        • Dynamic contextual data
          • Caching architecture
        • Static contextual data
      • Transformation
        • Field Tokeniser
        • Obfuscation
          • Encryption
          • Masking
          • Bucketing
          • Redaction
      • Triggers
        • Change Data Capture
        • Business rules
      • Stream join
        • Inner stream joins
        • Outer stream joins
        • Join attributes & policy
      • Event tap
        • Anatomy of a Tap
        • SQL Queries
    • Analytics
      • Analytic tools
        • User defined analytics
          • Streaming analytics example
          • User defined analytics
          • User defined scripts
          • User defined functions
            • Average function library
        • Window analytics
          • Tumbling window
          • Sliding window
          • Aggregate functions
        • Analytic functions
          • Stateful
            • Exponential moving average
            • Rolling Sum
          • Stateless
            • Normalisation
              • Absolute max
              • Min max
              • Standardisation
              • Mean
              • Log
              • Z-Score
            • Scaling
              • Unit scale
              • Robust Scale
            • Statistics
              • Statistic summaries
              • Weighted moving average
              • Simple moving average
              • Count
            • General
              • Euclidean
        • Advanced analytics
          • Geospatial
            • Entity geo tracker
            • Geofence occupancy trigger
            • Geo search
            • IP address resolver
            • Reverse geocoding
            • Spatial Index
          • HyperLogLog
          • Distinct counter
      • ML inferencing
        • Feature engineering
          • Scripting
          • Scaling
          • Transform
        • Online predictive analytics
        • Model audit
        • Model management
      • Metrics engine
        • Create metrics
        • Apply metrics
        • Manage metrics
        • Priming metrics
    • Contextual data
      • Architecture
      • Configuration
      • MinIO S3
      • Apache Geode
    • Connectors
      • Sources
        • Kafka
          • Ingestion
        • RabbitMQ
          • Further RabbitMQ configurations
        • MQTT
          • Topic wildcards
          • Session management
          • Last Will and Testament
        • Rest endpoints
        • MinIO S3
        • File watcher
      • Sinks
        • Kafka
        • RabbitMQ
          • Further configurations
        • MQTT
          • Persistent messaging
          • Last Will and Testament
        • SQL databases
        • InfluxDB
        • MongoDB
        • Geode
        • WebSocket endpoint
        • MinIO S3
        • File transport
        • Slack
        • Email
      • Serialisers
        • Serialisation
          • Custom transform example
          • Formatters
        • Deserialisers
          • Custom parsing example
    • Observability
      • Enabling JMX for Joule
      • Meters
      • Metrics API
  • DEVELOPER GUIDES
    • Setting up developer environment
      • Environment setup
      • Build and deploy
      • Install Joule
        • Install Docker demo environment
        • Install with Docker
        • Install from source
        • Install Joule examples
    • Joulectl CLI
    • API Endpoints
      • Mangement API
        • Use case
        • Pipelines
        • Data connectors
        • Contextual data
      • Data access API
        • Query
        • Upload
        • WebSocket
      • SQL support
    • Builder SDK
      • Connector API
        • Sources
          • StreamEventParser API
        • Sinks
          • CustomTransformer API
      • Processor API
      • Analytics API
        • Create custom metrics
        • Define analytics
        • Windows API
        • SQL queries
      • Transformation API
        • Obfuscation API
        • FieldTokenizer API
      • File processing
      • Data types
        • StreamEvent
        • ReferenceDataObject
        • GeoNode
    • System configuration
      • System properties
  • Deployment strategies
    • Deployment Overview
    • Single Node
    • Cluster
    • GuardianDB
    • Packaging
      • Containers
      • Bare metal
  • Product updates
    • Public Roadmap
    • Release Notes
      • v1.2.0 Join Streams with stateful analytics
      • v1.1.0 Streaming analytics enhancements
      • v1.0.4 Predictive stream processing
      • v1.0.3 Contextual SQL based metrics
    • Change history
Powered by GitBook
On this page
  • Overview
  • Examples & DSL attributes
  • Attributes schema
  • Credential attributes schema
  • Serialisation attributes schema
  • Serialiser example
  • Document formatter
  • Additional resources

Was this helpful?

  1. Components
  2. Connectors
  3. Sinks

MongoDB

Publish JSON based events to the mature standard document storePublishes JSON-based events to MongoDB, a widely-used document store

PreviousInfluxDBNextGeode

Last updated 6 months ago

Was this helpful?

Overview

Document database support is provided using the MongoDB Publisher Transport. Processed events are published to a specified MongoDB database collection.

The events are serialised using a serialiser or a custom transformer implementation.

This transport is designed to facilitate efficient data publishing into MongoDB with customisable serialisation options.

Some key features are:

  1. MongoDB integration Sends processed events to a defined MongoDB collection.

  2. Serialiser options Uses a default Document Formatter serialiser or allows custom serialisation.

Driver details:

Examples & DSL attributes

This example sets up the MongoDB Publisher Transport to send events to a local MongoDB instance (localhost:27017).

It stores data in the customerdb database and emea_customers collection, using ordered inserts.

Authentication is done with the joule user and SCRAM-SHA-1 mechanism. Events are serialised individually (not in batches) with Base64 encoding and application/octet-stream content type. SSL is disabled.

mongodbPublisher:
  servers:
    localhost: 27017
  enable ssl: false

  database: customerdb
  collection: emea_customers
  ordered inserts: true

  credentials:
    user: joule
    password: password
    mechanism: SCRAM_SHA_1

  serializer:
    batch: false
    formatter:
      documentFormatter:
        contentType: application/octet-stream
        encoding: base64

Attributes schema

Attribute
Description
Data Type
Required

database

Name of database

String

collection

Name of collection inserts will be applied to

String

ordered inserts

Hint to MongDB to insert documents using presented ordering

Boolean

Default: true

servers

Map of servers with associated ports

Map<String,Integer>

enable ssl

Enable SSL

boolean

Default: false

credentials

Credential configuration

serializer

Serialisation configuration

Credential attributes schema

Attribute
Description
Data Type
Required

username

Username

String

password

password

String

authentication database

Database to use to authenticate connected user

String

Default: admin

mechanism

Authentication mechanism

Options

  • PLAIN

  • SCRAM_SHA_1

  • SCRAM_SHA_256

String

Default: SCRAM_SHA_1

Serialisation attributes schema

This topic provides configuration parameters available object serialisation process.

Attribute
Description
Data Type
Required

transform

User provided implementation

CustomTransformer

formatter

This is used when a custom transformer is not provided. Useful when chaining Joule processes

Formatter

Default: documentFormatter (See Document Formatter)

contentType

Type of content to inform receiving application

String

Default: application/octet-stream

encoding

Payload encoding method

String

Default: base64

batch

Flag to batch multiple messages in to a single payload

Boolean

Default: false

Serialiser example

The configuration below will serialise StreamEvents as MongoDB documents.

mongodbPublisher:
  ...

  serializer:
    batch: false
    formatter:
      documentFormatter:
        contentType: application/octet-stream
        encoding: base64

Document formatter

The document formatter in Joule is a built-in tool that converts StreamEvents into MongoDB's Document type.

This allows the processed events to be formatted correctly before being inserted into a MongoDB collection. The formatter ensures that the data is serialised in a way that MongoDB can understand and store efficiently.

Additional resources

See section

See section

Official MongoDB

Official MongoDB

Mongo Compass

documentation
docker image
UI
org.mongodb:mongodb-driver-sync:4.8.1
document formatter
Credential attributes
Serialisation attributes