Anatomy of enrichment DSL
The enrichment DSL provides a flexible syntax that enables the user to define field level enrichments
What will we learn on this article?
In this section, we will learn about the enrichment process in Joule and focusing on how to add valuable data at the field level using different methods to an event in real-time.
We will cover:
Enrichment structure Overview of enrichment types, including the query, response, and store binding approaches.
DSL for enrichment How to define enrichment fields, set query and response parameters, and bind data stores using Joule’s Domain-Specific Language (DSL).
Query vs. key-based lookups Using key or query-based methods to retrieve reference data, with examples for each approach.
Response handling Options for handling responses, such as adding specific values, all attributes, or linking as objects.
Enrichment stores Connecting fields to data stores, including custom and out-of-the-box stores supported by Joule.
Overview
Enrichment is applied at the field level whereby each returned data element is added to the defined field either as map of values or as a domain object.
The fields
attribute is logical organised as three definition type:
Query approach
Response approach
Binding store
Enrichment DSL structure
Query and response DSL explainer
Two key attributes are required for the enricher processor:
with values
: is to define which fields to enrichusing
: provides the data store binding
Attributes explainer
Fields List of fields to populated with reference data using a look up criteria.
Query type Data is retrieved using one of two methods, by key or by query. The key is considered the indexed primary key within the query store. This will ensure a fast query look up. Whereas the query method allows you to use a ANSI SQL to perform the required data lookup. However, this query must return one row only.
Response On a successful query a
ReferenceDataObject
is returned which is unpacked using the defined attributes or linked to the event. See ReferenceDataObject for further informationData store This informs the query process which data store to execute against. The store can be either a custom or OOTB supported store.
Query syntax
Contextual data is retrieved using one of two methods, by key or by query.
by key
Using the key based look up approach enables you to perform a look up against a store using either the primary key or the key within a caching solution.
Example returns specific attributes from ReferenceDataObject
assigned to: deviceManufacturer
ReferenceDataObject
assigned to: deviceManufacturer
Example returns a ReferenceDataObject
as a linked object assigned to: modelDetails
ReferenceDataObject
as a linked object assigned to: modelDetails
by query
To fine tune your enrichment process you can define a query rather than a strict key based look up. This would provide you with a greater flexibility to drive further pipeline processing.
Below represents a OQL based query using an in-memory cache solution.
Attributes schema
Attribute | Description | Data Type |
---|---|---|
by query | Dependent upon linked data store | String |
query fields | Event field values to be applied to the query | Ordered list of Strings |
Response syntax
On a successful data retrieval the response object (ReferenceDataObject
) key values are added directly in to the event or added as an object.
One of the attributes must be provided
Attribute | Description | Data Type |
---|---|---|
with values | Values to add to the field as a map of key value pairs | List of Strings |
all attributes | Map all returned values attributes to event | Boolean |
as object | Returned value is linked as a nobject to the event | Boolean |
with values
Add specific attributes to the event.
Here we have selected the values deviceManufacturer
and year_released
to be included in the enrichment process.
all attributes
Add all attributes to the event.
Here we have done a select *
to ad all attributes to the enrichment process.
as object
Add the returned object to the event using the field name.
Here we have included the returned ReferenceDataObejct
in the enrichment process.
Binding stores syntax
Bind the field configuration to a data store using a logical store name. This would either be custom or using the pre-defined stores.
Attribute schema
Attribute | Description | Data Type | Required |
---|---|---|---|
using | Store name to apply query processing. Either a custom or supported store | String |
Supported stores
If either one of these are provided there is no need to specify the stores attribute.
JouleDB Use custom data sets imported on process initialisation such as static data.
MetricsDB Use computed live metrics.
Last updated