Send on delta

Stateful filter processor that filters previous seen events

Objective

Filter duplicate events from the processing stream to ensure that only unique events are processed once when specified attribute values change.

This filter uses an efficient send-on-delta algorithm, bloom filter, that looks at user-defined fields to determine the new received event has changed.

Only unique events that show these changes are sent on to the next processing stage.

Uses

There are various uses for this filter such as:

  1. In retail Configurable inform on update consumer pattern. Track changes to inventory stock levels and trigger alerts only when there is an update in the quantity of a particular product.

  2. In finance Only perform stream processing on key field changes. Monitor key financial metrics, such as stock prices or exchange rates, and generate alerts when a significant field update occurs.

  3. Healthcare Trigger alerting based upon a field update. Observe critical patient metrics and notify relevant personnel only when there’s a change in specific key health indicators.

Proactive filter replacement

Overtime the likelihood of false positives occurring increases with the number of events processed. This is mitigated by triggering an automatically filter update when the False Positive Probability (fpp) value is breached.

Any StreamEvent with the deltaBreachedFPP field indicates a filter change after the event is published.

Furthermore the following attributes can be used to fine tune when to reset the filter:

  • reset by time delay

  • reset by event count

Both of the reset functions can be applied with one being executed before the other.

Examples & DSL attributes

This configuration will replace the internal bloom filter once fpp value is breached.

send on delta:
  fields:
    - customerId
    - cash_balance
    - available_credit
  expected unique events: 100000
  fpp: 0.98

Attributes schema

AttributeDescriptionData TypeRequired

fields

Javascript expression that returns a boolean value

List<String>

expected unique events

It's important to calculate this accurately; otherwise, false positives will occur as the number nears or exceeds the set limit. A minimum of 1000 events is required to create the filter.

Long Default: 1000

fpp

Desired false positive probability. must be positive and less than 1.0

Double

Default: 0.97

reset by time delay

Reset the bloom filter after a set number of seconds.

Long

Default: 0

reset by event count

Reset the bloom filter after a set number of events processed.

Integer

Default: 0

Last updated