Change Data Capture

Distribute delta records for key attribute changes

Objective

Change Data Capture (CDC) processor identifies and processes only data that has changed and then makes this changed data available for further use as a delta event.

Uses

The CDC process determines change based upon the previous values held for the tracking fields. This presents various uses for this processor such as:

1. In financial services Provide a configurable inform on update consumer pattern. Implement a system where traders receive notifications only for changes in stock prices or account balances, rather than the entire transaction details, allowing for quicker decision-making.

2. In healthcare Optimised delta processing for event consumers and receive updates only on specific tracking fields vs the full event. Enable healthcare providers to receive alerts about changes in patient data, such as medication dosages or vital signs, without needing the entire medical record, ensuring focused and timely responses.

3. In retail Allow inventory management systems to inform staff only when stock levels change for specific items, rather than sending complete inventory reports, streamlining operations and improving response times.

Example & DSL attributes

This will emit changed StreamEvents keyed on symbol when either, OR function, the broker, bid or ask values change.

change data capture:
  key field: symbol
  tracking fields:
    - broker
    - bid
    - ask
  emit only changes: true
  drop duplicates: true

Attributes schema

Attribute	Description	Data Type	Required
key field	Key field to	String
fields	List of fields to track and determine changes	List<String>
emit only changes	A flag to emit only a delta event for the changes seen on the record. If set to false changes found will be will be added, as a map, to the CDC_FIELD event along with a CDC field being set to true to signify record availability	Boolean Default: true
drop duplicates	Control flag which can be set to validate processor functionality at runtime. Use a JMX console to set	Boolean Default: true