Filters
Reduce stream processing overhead by filtering irrelevant events
Overview
Filters are the foundational component to manage incoming events.
This is a technique that removes irrelevant or sensitive data from a stream by evaluating each element with pre-set conditions.
Filters aim to improve processing efficiency, especially when only a subset of the data is needed for your use case.
These filters allow you a variety of options such as filtering event based using a configurable criteria or a custom expression.
Use cases
Here are some use cases of how it can be applied in stream processing:
Apply customer opt in / out policies Manage the inclusion of customer events within the processing stream using static lists.
Addressing missing data elements Validate all required data is present before further stream processing by dropping events.
Handling PII data Remove or mask PII field data within the event.
Handling out-of-range values Validate and process events that meet a within range criteria.
Available filters options
By type
Filter events based on specific types from the processing pipeline
By expression
Use Javascript expressions to define custom filters.
On delta
Stateful filter processor that filters previous seen events
Remove attributes
Remove sensitive attributes from an event
Drop all
Drop all incoming events in the processor
Last updated