Send on delta
Stateful filter processor that filters previous seen events
Objective
Filter duplicate events from the processing stream to ensure that only unique events are processed once when specified attribute values change.
This filter uses an efficient send-on-delta algorithm, bloom filter, that looks at user-defined fields to determine the new received event has changed.
Only unique events that show these changes are sent on to the next processing stage.
Uses
There are various uses for this filter such as:
In retail Configurable inform on update consumer pattern. Track changes to inventory stock levels and trigger alerts only when there is an update in the quantity of a particular product.
In finance Only perform stream processing on key field changes. Monitor key financial metrics, such as stock prices or exchange rates, and generate alerts when a significant field update occurs.
Healthcare Trigger alerting based upon a field update. Observe critical patient metrics and notify relevant personnel only when there’s a change in specific key health indicators.
Proactive filter replacement
Overtime the likelihood of false positives occurring increases with the number of events processed. This is mitigated by triggering an automatically filter update when the False Positive Probability (fpp
) value is breached.
Any StreamEvent
with the deltaBreachedFPP
field indicates a filter change after the event is published.
Furthermore the following attributes can be used to fine tune when to reset the filter:
reset by time delay
reset by event count
Both of the reset functions can be applied with one being executed before the other.
Examples
This configuration will replace the internal bloom filter once fpp
value is breached.
Attributes schema
Attribute | Description | Data Type | Required |
---|---|---|---|
fields | Javascript expression that returns a boolean value | List<String> | |
expected unique events | It's important to calculate this accurately; otherwise, false positives will occur as the number nears or exceeds the set limit. A minimum of 1000 events is required to create the filter. | Long Default: 1000 | |
fpp | Desired false positive probability. must be positive and less than 1.0 | Double Default: 0.97 | |
reset by time delay | Reset the bloom filter after a set number of seconds. | Long Default: 0 | |
reset by event count | Reset the bloom filter after a set number of events processed. | Integer Default: 0 |
Last updated