Anatomy of a Tap

Construction of a Tap

Overview

This article should give context about how a Tap is constructed.

Example & DSL attributes

This code defines a tap configuration for streaming data to the nasdaq_quotes schema.

1. target schema Specifies that the data will be sent to the nasdaq_quotes schema.

2. queue capacity Sets the maximum capacity of the queue to 15,000 events.

3. flush frequency Data will be flushed (written) every 5 events.

4. index An index is created on the symbol field, but it is not marked as unique (unique: false).

This configuration controls how data is buffered and indexed before being sent to the target schema.

Example

tap:
  target schema: nasdaq_quotes
  queue capacity: 15000
  flush frequency: 5
  index:
    unique: false
    fields:
      - symbol

Attributes schema

Attribute	Description	Data Type	Required
target schema	Target table name	String
queue capacity	Number of events to queue before flushing to database. Must be greater than 99	Integer Default :10000
flush frequency	Frequency the queue is flushed to database table. Either the queue capacity or this attribute triggers queue flush whatever comes first Must me greater than zero	Long Default: 5 Seconds
alias	Alias to use instead of the schema name	String
is insert	Update or insert to apply to table changes	Boolean Default: true
update criteria fields	Array of fields used for the update criteria predicate. Only used if is insert set to false	String[]
index	Create a table index	See Index attributes

Index Attributes

Attribute	Description	Data Type	Required
unique index	Create a unique index	Boolean Default: false
index fields	Array of fields to create index	String[]