Data connectors

Register, list, detail and unregister data sources and sinks

Overview

Transports provide the integration links required to consume and publish computed stream results. Joule Management API provides the ability to register, list, detail and undeploy transports.

For further information on the structure of the content payload refer to this documentation.

Example data connector content

{
    "kafkaConsumer": {
        "name": "nasdaq_quotes_stream",
        "cluster address": "joule-gs-redpanda-0:9092",
        "consumerGroupId": "nasdaq",
        "topics": [
        "quotes"
        ],
        "deserializer": {
        "parser": "com.fractalworks.examples.banking.data.QuoteToStreamEventParser",
        "key deserializer": "org.apache.kafka.common.serialization.IntegerDeserializer",
        "value deserializer": "com.fractalworks.streams.transport.kafka.serializers.object.ObjectDeserializer"
        },
        "properties": {
        "partition.assignment.strategy": "org.apache.kafka.clients.consumer.StickyAssignor",
        "max.poll.records" : "7000",
        "fetch.max.bytes" : "10485760"
        }
    }
}

Register transport

Register transport that either consumes or publishes data. A json definition is provided within the body as a raw data typeIf the registration process has succeed a response code of 201 is returned otherwise 500.

POST/joule/management/transports/register

Response

201500501502503

Successful registered transport

Request

JavaScriptCurlPython

const response = await fetch('/joule/management/transports/register', {
    method: 'POST',
    headers: {},
});
const data = await response.json();

List transports by type

GET/joule/management/transports/list

Query parameters

Response

404500501

Failed to find transport using provided name.

Request

JavaScriptCurlPython

const response = await fetch('/joule/management/transports/list?type=text', {
    method: 'GET',
    headers: {},
});
const data = await response.json();

Get transport specification

GET/joule/management/transports/detail

Query parameters

Response

200404501

Provide the transport specification

Request

JavaScriptCurlPython

const response = await fetch('/joule/management/transports/detail?name=text&type=text', {
    method: 'GET',
    headers: {},
});
const data = await response.json();

Unregister transport

Unregister transport that either consumes or publishes data.If the unregistion process has succeed a response code of 201 is returned.

DELETE/joule/management/transports/unregister

Query parameters

Response

201500501502503

Successful transport unregister

Request

JavaScriptCurlPython

const response = await fetch('/joule/management/transports/unregister?name=text&type=text', {
    method: 'DELETE',
    headers: {},
});
const data = await response.json();