Anatomy of enrichment DSL

The enrichment DSL provides a flexible syntax that enables the user to define field level enrichments

What will we learn on this article?

In this section, we will learn about the enrichment process in Joule and focusing on how to add valuable data at the field level using different methods to an event in real-time.

We will cover:

1. Enrichment structure Overview of enrichment types, including the query, response, and store binding approaches.

2. DSL for enrichment How to define enrichment fields, set query and response parameters, and bind data stores using Joule’s Domain-Specific Language (DSL).

3. Query vs. key-based lookups Using key or query-based methods to retrieve reference data, with examples for each approach.

4. Response handling Options for handling responses, such as adding specific values, all attributes, or linking as objects.

5. Enrichment stores Connecting fields to data stores, including custom and out-of-the-box stores supported by Joule.

Overview

Enrichment is applied at the field level whereby each returned data element is added to the defined field either as map of values or as a domain object.

The fields attribute is logical organised as three definition type:

1. Query approach

2. Response approach

3. Binding store

Enrichment DSL structure

enricher:
  fields:      
    # field query definition
    deviceManufacturer:   
      ## Query and response per field
      ## :start          
         
      # 1. Query type and event variable value to use 
      by key: tac
      
      # 2. Response attribute/s              
      with values: [deviceManufacturer, year_released]

      # 3. Data store to query
      using: deviceStore
      
      ## :end

Query and response DSL explainer

Two key attributes are required for the enricher processor:

1. with values: is to define which fields to enrich

2. using: provides the data store binding

Attributes explainer

Attribute	Description
Fields	List of fields to populate with reference data using a lookup criteria.
Query type	Data is retrieved using either: - by key: Uses the indexed primary key within the query store for fast lookup. - by query: Uses ANSI SQL for data lookup, but the query must return only one row.
Response	A ReferenceDataObject is returned on a successful query, which is unpacked using the defined attributes or linked to the event.
Data store	Specifies which data store to query against, which can be either a custom or an OOTB supported store.

Query syntax

Contextual data is retrieved using one of two methods, by key or by query.

by key

Using the key based look up approach enables you to perform a look up against a store using either the primary key or the key within a caching solution.

Example returns specific attributes from ReferenceDataObject assigned to: deviceManufacturer

deviceManufacturer:
    by key: tac
    with values: [deviceManufacturer, year_released]
    using: deviceStore

Example returns a ReferenceDataObject as a linked object assigned to: modelDetails

modelDetails:
  by key: tac
  as object: true
  using: deviceStore

by query

To fine tune your enrichment process you can define a query rather than a strict key based look up. This would provide you with a greater flexibility to drive further pipeline processing.

Below represents a OQL based query using an in-memory cache solution.

contractedDataBundle:
    by query:  "select * from /userBundle where imsi = ?"
    query fields: [imsi]
    all attributes: true
    using: dataBundleStore

Attributes schema

Attribute	Description	Data Type
by query	Dependent upon linked data store	String
query fields	Event field values to be applied to the query	Ordered list of Strings

Response syntax

On a successful data retrieval the response object (ReferenceDataObject) key values are added directly in to the event or added as an object.

One of the attributes must be provided

Attribute	Description	Data Type
with values	Values to add to the field as a map of key value pairs	List of Strings
all attributes	Map all returned values attributes to event	Boolean
as object	Returned value is linked as a nobject to the event	Boolean

with values

Add specific attributes to the event.

Here we have selected the values deviceManufacturer and year_released to be included in the enrichment process.

deviceManufacturer:
    by key: tac
    with values: [deviceManufacturer, year_released]
    using: deviceStore

all attributes

Add all attributes to the event.

Here we have done a select * to ad all attributes to the enrichment process.

contractedDataBundle:
    by query:  "select * from /userBundle where imsi = ?"
    query fields: [imsi]
    all attributes: true
    using: dataBundleStore

as object

Add the returned object to the event using the field name.

Here we have included the returned ReferenceDataObejct in the enrichment process.

deviceInformation:
    by key: tac
    as object: true    
    using: deviceStore

Binding stores syntax

Bind the field configuration to a data store using a logical store name. This would either be custom or using the pre-defined stores.

deviceInformation:
    by key: tac
    as object: true    
    using: deviceStore

Attribute schema

Attribute	Description	Data Type	Required
using	Store name to apply query processing. Either a custom or supported store	String

Supported stores

If either one of these are provided there is no need to specify the stores attribute.

• JouleDB Use custom data sets imported on process initialisation such as static data.

• MetricsDB Use computed live metrics.