Create custom metrics
We will create our own custom metrics from scratch
Overview
In this tutorial we will learn how to create metrics by defining a metric family.
This includes the result table store metrics and the runtime frequency using a policy.
One or more metric families can be configured, each has its own management policy.
What this tutorial is about
This tutorial covers the creation, definition and the management of metrics within Joule, providing step-by-step guidance on setting up a complete metrics framework using metric families, policies and table schemas.
Who this tutorial is intended for
This tutorial is designed for developers and data engineers looking to implement and optimise custom metrics in Joule, particularly those working on data-heavy or KPI-driven applications.
What we will learn
We will learn how to:
Define metric families
Build and validate efficient queries
Assign stable metric keys
Set up target table schemas
Manage initial metric images and data storage policies
Prerequisites
Basic knowledge of SQL, familiarity with metrics generation concepts and access to:
DuckDB local installation and understanding how to interact with it, see the documentation.
Import dataset, build query and validate results.
Joule platform, follow this tutorial to see how to set Joule up Getting started.
Implementation
Step 1 - Define a metric family name for the metrics
We need to come up with a naming convention that makes sense and is descriptive for our metric.
We should expect this output
Step 2 - Write and validate our metrics generation query
We will build, validate and optimise metrics query.
Import dataset, build query and validate results.
Ensure query performance is optimal and sits below 5 seconds.
We should expect this output
Step 3 - Assign a unique stable metric key
We will assign a stable metrics key which is used to query the target metrics table.
A successful metric query must only return either 0 or 1 row.
We should expect this output
Step 4 - Define target metrics table schema
Define the target metrics table using the metrics family name as the table name.
We should expect this output
Step 5 - Do you need a metric image on start up?
Some processing requirements need to have initial metric values to work correctly, such as KPIs for example.
For this scenario an initial image of metric family is required and may come from pre-validated data or from a previous run without additional processing steps. This data may need to be imported.
In the event of an import we will need to use the import feature provided by Joule.
We should expect this output
Step 6 - Finally consider storage requirements
With respect to memory usage we should consider how much historical data we will need to store in-memory.
We can use the compaction policy to delete data that is not needed.
We should expect this output
What Did We Learn?
In this tutorial, we learned:
How to create and manage metrics within Joule by defining metric families.
Writing and validating optimised queries
Setting up target table schemas
We also covered best practices for assigning stable metric keys, initialising metric images for specific use cases and managing memory usage with compaction policies.
What’s Next?
With these foundational skills in place, the next steps involve exploring advanced metrics configurations, such as:
Creating complex metrics aggregations.
Applying more sophisticated data transformation pipelines
Integrating real-time observability tools.
Additionally, users can dive into Joule’s processor SDK to develop custom processors, further enhancing the metrics functionality to meet specific business requirements.
Last updated