Event Tap

Tap events directly in to an in-memory database to enable on/off line processing

Joule provides the ability to tap an event stream in to an in-process / memory database. This feature is useful to gain data in SQL format for addtional purposes such as off-line analytic. Combining this feature with the provided Rest API functionality it becomes a powerful feature.

Example

tap:
  target schema: nasdaq_quotes
  queue capacity: 15000
  flush frequency: 5
  index:
    unique: false
    fields:
      - symbol

Attributes

AttributeDescriptionData TypeRequired

target schema

Target table name

String

queue capacity

Number of events to queue before flushing to database.

Must be greater than 99.

Integer

Default :10000

flush frequency

Frequency the queue is flushed to database table. Either the queue capacity or this attribute triggers queue flush whatever comes first.

Must me greater than zereo.

Long

Default: 5 Seconds

alias

Alias to use instead of the schema name

String

is insert

Update or insert to apply to table changes

Boolean

Default: true

update criteria fields

Array of fields used for the update criteria predicate. Only used if is insert set to false

String[]

index

Create a table index

See Index attributes

Index Attributes

AttributeDescriptionData TypeRequired

uniquie index

Create a unique index

Boolean

Default: false

index fields

Array of fields to create index

String[]

SQL Queries

On processor initialisation a set of queries are auto generated which are used to bind Rest API endpoints. The following queries are generated:

Select

Select data using pagnation either with or without an ingestTime date range.

Export

Ability to export the table either by CSV or Parquet formats.

Delete

Query to delete rows from the table between an ingestTime date range.

Last updated