# Old version of Key concepts

***

## Low latency enrichment architecture

OOTB Joule has provided the heavy lifting required to deploy a low-latency solution so that developers can focus on building advanced use case that required addition contextual data. The Joule contextual data architecture supports various low-latency in-memory data stores that reduce the inherent I/O overhead of out-of-process databases and thus enable stream based enrichment.

<figure><img src="https://3062398388-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FUU6FZlV07ZD90OzbzGww%2Fuploads%2F7JzhhG8jO2sWHJo8DL5a%2Fcontextual_enricher_arch.png?alt=media&#x26;token=d1a42663-2f2c-452e-bbab-20169f5d930c" alt=""><figcaption></figcaption></figure>

***

## Example

The below example enriches events using the linked reference data and metrics. Reference data is imported at process startup using a provided parquet file. Metrics are calculated after 1 minute using a  three minute tumbling window approach.&#x20;

```yaml
stream:
  name: quoteAnalyticsStream
  enabled: true
  validFrom: 2020-01-01
  validTo: 2025-12-31
  eventTimeType: EVENT_TIME
  sources: [ nasdaq_quotes_stream ]

  initialisation:
    sql import:
      schema: reference_data
      parquet:
        - table: nasdaq_companies
          asView: false
          files: [ 'data/parquet/nasdaq.parquet' ]
          index:
            fields: [ 'symbol' ]
            unique: true

  processing unit:
    metrics engine:
      runtime policy:
        frequency: 1
        startup delay: 1
        time unit: MINUTES

      foreach metric compute:
        metrics:
          -
            name: BidMovingAverage
            metric key: symbol
            table definition: bid_moving_averages (symbol VARCHAR, avg_bid_min FLOAT, avg_bid_avg FLOAT,avg_bid_max FLOAT,createdTimestamp TIMESTAMP)
            query:
              SELECT symbol,
              MIN(bid) AS 'avg_bid_min',
              AVG(bid) AS 'avg_bid_avg',
              MAX(bid) AS 'avg_bid_max'
              FROM quotes.nasdaq
              WHERE
              ingestTime >= epoch_ms(date_trunc('minutes',now() - INTERVAL 3 MINUTES)) AND ingestTime <= epoch_ms(now())
              GROUP BY symbol
              ORDER BY 1;
            truncate on start: true
            compaction policy:
              frequency: 8
              time unit: HOURS

    pipeline:
      - tap:
          target schema: quotes
          flush frequency: 5
          index:
            unique: false
            fields:
              - symbol
      - enricher:
          fields:
            company_info:
              by query: "select * from reference_data.nasdaq_companies where Symbol = ?"
              query fields: [ symbol ]
              with values: [ Name,Country ]
              using: JouleDB
            quote_metrics:
              by metric family: BidMovingAverage
              by key: symbol
              with values: [avg_bid_min, avg_bid_avg, avg_bid_max]
              using: MetricsDB
  emit:
    select: "symbol, Name, Country, avg_bid_min, avg_bid_avg, avg_bid_max"

  group by:
    - symbol
```

#### Output

The above example generates the following output, in this case CSV.

```csv
nasdaq_View|null|1709894745240|1709894745240|MTZ|113.97318|103.25833|110.11051|MasTec Inc. Common Stock|Basic Industries
nasdaq_View|null|1709894745244|1709894745244|SBRA|18.71446|13.90475|16.472204|Sabra Health Care REIT Inc. Common Stock|Consumer Services
nasdaq_View|null|1709894745240|1709894745240|MUA|17.244429|14.651974|15.772599|Blackrock MuniAssets Fund Inc Common Stock|Finance
```

## General Enricher DSL&#x20;

The enricher processor provide users to the ability to enrich an event with multiple data elements from various data sources through the use of enhanced mapping.

### Example

```yaml
enricher:
  fields:      
    deviceManufacturer:
      by key: tac
      with values: [deviceManufacturer, year_released]
      using: deviceStore

    modelDetails:
      by key: tac
      as object: true   
      using: deviceStore

    contractedDataBundle:
      by query:  "select * from /userBundle where imsi = ?"
      query fields: [imsi]
      all attributes: true
      using: dataBundleStore

  stores:
    deviceStore:
      store name: mobiledevices

    dataBundleStore:
      store name: mobilecontracts
```

## Top level attribute schema

Two key attributes are required for the enricher processor; one is to define which fields to enrich whereas the other provides the data store binding.

<table><thead><tr><th width="193">Attribute</th><th width="217">Description</th><th width="219">Data Type</th><th data-type="checkbox">Required</th></tr></thead><tbody><tr><td>fields</td><td>List of fields to populated with reference data using a look up criteria.</td><td>List of field to reference data criteria configurations </td><td>true</td></tr><tr><td>stores</td><td>List of stores to perform the reference data lookup</td><td>List of store configurations</td><td>true</td></tr></tbody></table>

## Fields Attribute

Enrichment is applied at the field level whereby each returned data element is added to the defined field either as map of values or as a domain object.

The field attribute is logical organised as three definition type:

* Query approach
* Response approach
* Binding store

### Query approach&#x20;

Contextual data is retrieved using one of two methods, by key or by query.&#x20;

### By Key

Using the key based look up approach enables you to perform a look up against a store using either the primary key or the key within a caching solution.

#### Example returns specific attributes from ReferenceDataObject

```yaml
deviceManufacturer:
    by key: tac
    with values: [deviceManufacturer, year_released]
    using: deviceStore
```

#### Example returns a ReferenceDataObject as a linked object

```yaml
modelDetails:
  by key: tac
  as object: true
  using: deviceStore
```

See [ReferenceDataObject](https://docs.fractalworks.io/joule/developer-guides/builder-sdk/data-types/referencedataobject) for further information

### By Query

To fine tune your enrichment process you can define a query rather than a strict key based look up. This would provide you with a greater flexibility to drive further pipeline processing. Below represents a OQL based query using an in-memory cache solution.

```yaml
contractedDataBundle:
    by query:  "select * from /userBundle where imsi = ?"
    query fields: [imsi]
    all attributes: true
    using: dataBundleStore
```

#### Attributes

<table><thead><tr><th width="193">Attribute</th><th width="373">Description</th><th width="219">Data Type</th></tr></thead><tbody><tr><td>by query</td><td>Dependent upon linked data store.</td><td>String</td></tr><tr><td>query fields</td><td>Event field values to be applied to the query</td><td>Ordered list of Strings</td></tr></tbody></table>

***

### Response Approach

On a successful data retrieval the response object, *ReferenceDataObejct*,  key values are added directly in to the event or added as an object.

<table><thead><tr><th width="152">Attribute</th><th width="330">Description</th><th width="261">Data Type</th></tr></thead><tbody><tr><td>with values</td><td>Values to add to the field as a map of key value pairs</td><td>List of Strings</td></tr><tr><td>all attributes</td><td>Map all returned values attributes to event</td><td>Boolean</td></tr><tr><td>as object</td><td>Returned value is linked as a object to the event</td><td>Boolean</td></tr></tbody></table>

Either one of the attributes must be provided.

### with values

Add selected attributes to the event.

```yaml
deviceManufacturer:
    by key: tac
    with values: [deviceManufacturer, year_released]
    using: deviceStore
```

### all attributes

Add all attributes to the event.

```yaml
contractedDataBundle:
    by query:  "select * from /userBundle where imsi = ?"
    query fields: [imsi]
    all attributes: true
    using: dataBundleStore
```

### as object

Add the returned object to the event using the field name.

```yaml
deviceInformation:
    by key: tac
    as object: true    
    using: deviceStore
```

### Binding store

Bind the field configuration to a data store using a logical store name. This would either be custom or using the pre-defined stores. A custom store should be defined under the Stores Attribute, see [section](#stores-attribute) for more details.

<table><thead><tr><th width="152">Attribute</th><th width="330">Description</th><th width="164">Data Type</th><th data-type="checkbox">Required</th></tr></thead><tbody><tr><td>using</td><td>Store name to apply query processing. Either a custom or supported store </td><td>String</td><td>true</td></tr></tbody></table>

#### Supported stores

* JouleDB
* MetricsDB

If either one of these are provided there is no need to specify the stores attribute.

## Stores Attribute

Bind the processor to one or more linked data stores using a logical store name mapped to a set of configuration attributes. The defined store name configuration needs to have been provided, see [reference data documentation](https://docs.fractalworks.io/joule/developer-guides/builder-sdk/data-types/referencedataobject) for further details.

#### Example

The example below uses the `nasdaqIndexCompanies` as a logical name to bind the reference data lookup criteria to be performed.&#x20;

```yaml
stores:
  deviceStore:
    store name: mobiledevices

  dataBundleStore:
    store name: mobilecontracts
```

### Attributes

<table><thead><tr><th width="193">Attribute</th><th width="425">Description</th><th width="160">Data Type</th></tr></thead><tbody><tr><td>stores</td><td>List of stores required for the enricher. Usage is local logical name mapped to <code>store name</code></td><td>String</td></tr><tr><td>store name</td><td>Actual store name specified in the reference data store configuration</td><td>String</td></tr></tbody></table>

## ReferenceDataObject

For further information read [Reference Data](https://docs.fractalworks.io/joule/developer-guides/builder-sdk/data-types/referencedataobject) documentation