Send on delta

Stateful filter processor that filters previous seen events

Objective

Filter duplicate events from the processing stream to ensure that only unique events are processed once when specified attribute values change.

This filter uses an efficient send-on-delta algorithm, bloom filter, that looks at user-defined fields to determine the new received event has changed.

Only unique events that show these changes are sent on to the next processing stage.

Proactive filter replacement

Overtime the likelihood of false positives occurring increases with the number of events processed. This is mitigated by triggering an automatically filter update when the fpp value is breached. Any StreamEvent with the deltaBreachedFPP field indicates a filter change after the event is published.

Furthermore the following attributes can be used to fine tune when to reset the filter:

• reset by time delay

• reset by event count

Both of the reset functions can be applied with one being executed before the other.

Examples

Automatic replacement

This configuration will replace the internal bloom filter once the False Positive Probability (fpp) value is breached.

send on delta:
  fields:
    - customerId
    - cash_balance
    - available_credit
  expected unique events: 100000
  fpp: 0.98

Proactive replacement

This configuration will replace the internal bloom filter on one of three conditions:

• False Positive Probability (fpp) value has breached 0.98.

• Time since filter creation is passed 15 minutes.

• Number of events seen has breached 50,000.

send on delta:
  fields:
    - customerId
    - cash_balance
    - available_credit
  expected unique events: 100000
  fpp: 0.98
  reset by time delay: 900
  reset by event count: 500000

Attributes schema

Attribute	Description	Data Type	Required
fields	Javascript expression that returns a boolean value	List<String>
expected unique events	It's important to calculate this accurately; otherwise, false positives will occur as the number nears or exceeds the set limit. A minimum of 1000 events is required to create the filter.	Long Default: 1000
fpp	Desired false positive probability. must be positive and less than 1.0	Double Default: 0.97
reset by time delay	Reset the bloom filter after a set number of seconds.	Long Default: 0
reset by event count	Reset the bloom filter after a set number of events processed.	Integer Default: 0