Create custom metrics

We will create our own custom metrics from scratch

Overview

In this tutorial we will learn how to create metrics by defining a metric family.

This includes the result table store metrics and the runtime frequency using a policy.

One or more metric families can be configured, each has its own management policy.

What this tutorial is about

This tutorial covers the creation, definition and the management of metrics within Joule, providing step-by-step guidance on setting up a complete metrics framework using metric families, policies and table schemas.

Who this tutorial is intended for

This tutorial is designed for developers and data engineers looking to implement and optimise custom metrics in Joule, particularly those working on data-heavy or KPI-driven applications.

What we will learn

We will learn how to:

1. Define metric families

2. Build and validate efficient queries

3. Assign stable metric keys

4. Set up target table schemas

5. Manage initial metric images and data storage policies

Prerequisites

Basic knowledge of SQL, familiarity with metrics generation concepts and access to:

1. DuckDB local installation and understanding how to interact with it, see the documentation.
2. Import dataset, build query and validate results.

3. Joule platform, follow this tutorial to see how to set Joule up Getting started.

Implementation

Step 1 - Define a metric family name for the metrics

We need to come up with a naming convention that makes sense and is descriptive for our metric.

We should expect this output

Step 2 - Write and validate our metrics generation query

We will build, validate and optimise metrics query.

• Import dataset, build query and validate results.

• Ensure query performance is optimal and sits below 5 seconds.

We should expect this output

Step 3 - Assign a unique stable metric key

We will assign a stable metrics key which is used to query the target metrics table.

A successful metric query must only return either 0 or 1 row.

We should expect this output

Step 4 - Define target metrics table schema

Define the target metrics table using the metrics family name as the table name.

We should expect this output

Step 5 - Do you need a metric image on start up?

Some processing requirements need to have initial metric values to work correctly, such as KPIs for example.

For this scenario an initial image of metric family is required and may come from pre-validated data or from a previous run without additional processing steps. This data may need to be imported.

In the event of an import we will need to use the import feature provided by Joule.

We should expect this output

Step 6 - Finally consider storage requirements

With respect to memory usage we should consider how much historical data we will need to store in-memory.

We can use the compaction policy to delete data that is not needed.

We should expect this output

What Did We Learn?

In this tutorial, we learned:

1. How to create and manage metrics within Joule by defining metric families.

2. Writing and validating optimised queries

3. Setting up target table schemas

We also covered best practices for assigning stable metric keys, initialising metric images for specific use cases and managing memory usage with compaction policies.

What’s Next?

With these foundational skills in place, the next steps involve exploring advanced metrics configurations, such as:

1. Creating complex metrics aggregations.

2. Applying more sophisticated data transformation pipelines

3. Integrating real-time observability tools.

Additionally, users can dive into Joule’s processor SDK to develop custom processors, further enhancing the metrics functionality to meet specific business requirements.