Data connectors

Register, list, detail and unregister data sources and sinks

Overview

Transports provide the integration links required to consume and publish computed stream results. Joule Management API provides the ability to register, list, detail and undeploy transports.

For further information on the structure of the content payload refer to this documentation.

Example data connector content

{
    "kafkaConsumer": {
        "name": "nasdaq_quotes_stream",
        "cluster address": "joule-gs-redpanda-0:9092",
        "consumerGroupId": "nasdaq",
        "topics": [
        "quotes"
        ],
        "deserializer": {
        "parser": "com.fractalworks.examples.banking.data.QuoteToStreamEventParser",
        "key deserializer": "org.apache.kafka.common.serialization.IntegerDeserializer",
        "value deserializer": "com.fractalworks.streams.transport.kafka.serializers.object.ObjectDeserializer"
        },
        "properties": {
        "partition.assignment.strategy": "org.apache.kafka.clients.consumer.StickyAssignor",
        "max.poll.records" : "7000",
        "fetch.max.bytes" : "10485760"
        }
    }
}

Register transport

post

Register transport that either consumes or publishes data. A json definition is provided within the body as a raw data typeIf the registration process has succeed a response code of 201 is returned otherwise 500.

Responses

201

Successful registered transport

500

Internal Joule error. Check log files.

501

Failed to register transport due to malformed DSL. Check Joule service log files.

502

Transport has already been registered

503

Missing transport specification

post

HTTPcURLJavaScriptPython

POST /joule/management/transports/register HTTP/1.1
Host: 
Accept: */*

No content

List transports by type

get

Query parameters

typestringRequired

Transport type (i.e. SINK, SOURCE, REFERENCE_DATA)

Responses

404

Failed to find transport using provided name.

500

Internal Joule error. Check log files.

501

Invalid transport type. Valid types are SINK, SOURCE and REFERENCE_DATA.

get

HTTPcURLJavaScriptPython

GET /joule/management/transports/list?type=text HTTP/1.1
Host: 
Accept: */*

No content

Get transport specification

get

Query parameters

namestringRequired

Use case DSL configuration.

typestringRequired

Transport type (i.e. SINK, SOURCE, REFERENCE_DATA)

Responses

200

Provide the transport specification

404

Failed to find transport using provided name.

501

Invalid transport type. Valid types are SINK, SOURCE and REFERENCE_DATA.

get

HTTPcURLJavaScriptPython

GET /joule/management/transports/detail?name=text&type=text HTTP/1.1
Host: 
Accept: */*

No content

Unregister transport

delete

Unregister transport that either consumes or publishes data.If the unregistion process has succeed a response code of 201 is returned.

Query parameters

namestringRequired

Transport logical name.

typestringRequired

Transport type (i.e. SINK, SOURCE, REFERENCE_DATA)

Responses

201

Successful transport unregister

500

Internal Joule error. Check log files.

501

Failed to unregister transport using provided name.

502

Invalid transport type. Valid types are SINK, SOURCE and REFERENCE_DATA.

503

Unknown transport for provided name.

delete

HTTPcURLJavaScriptPython

DELETE /joule/management/transports/unregister?name=text&type=text HTTP/1.1
Host: 
Accept: */*

No content