Filters

Reduce stream processing overhead by filtering irrelevant events

Overview

Filters are the foundational component to manage incoming events.

This is a technique that removes irrelevant or sensitive data from a stream by evaluating each element with pre-set conditions.

Filters aim to improve processing efficiency, especially when only a subset of the data is needed for your use case.

These filters allow you a variety of options such as filtering event based using a configurable criteria or a custom expression.

Use cases

Here are some use cases of how it can be applied in stream processing:

1. Apply customer opt in / out policies Manage the inclusion of customer events within the processing stream using static lists.

2. Addressing missing data elements Validate all required data is present before further stream processing by dropping events.

3. Handling PII data Remove or mask PII field data within the event.

4. Handling out-of-range values Validate and process events that meet a within range criteria.

Available filters options

By type

Filter events based on specific types from the processing pipeline

By expression

Use Javascript expressions to define custom filters.

On delta

Stateful filter processor that filters previous seen events

Remove attributes

Remove sensitive attributes from an event

Drop all

Drop all incoming events in the processor