Understanding and Implementing Recording Rules in Prometheus

Prometheus is an open-source monitoring system that is widely used in the industry to monitor the health and performance of applications and infrastructure. One of the key features of Prometheus is its ability to define recording rules. Recording rules allow users to create new time-series data based on existing data, apply functions and filters, and store the results in the Prometheus database. We will explore the composition of recording rules, their advantages, how to create and add them to the Prometheus configuration, and provide an example YAML file of a recording rule.

Composition of Recording Rules

Recording rules in Prometheus are defined using PromQL (Prometheus Query Language). PromQL is a powerful language that allows users to express complex queries and apply mathematical functions to time-series data. Recording rules consist of two main components: a name and a PromQL expression.

The name of the recording rule is used to identify the new time-series data that will be created. It must be unique within the Prometheus instance and follow the same naming convention as metric names. For example, if the recording rule is used to create a new metric that represents the total number of requests per minute, the name could be requests_per_minute_total.

The PromQL expression is used to define how the new time-series data is created. The expression can include one or more existing metrics, as well as mathematical functions and filters. For example, the following PromQL expression creates a new metric that represents the total number of requests per minute:

requests_per_minute_total = sum(rate(http_requests_total[1m]))

In this expression, http_requests_total is an existing metric that records the total number of HTTP requests. The rate function calculates the rate of change of the metric over the last minute, and the sum function calculates the total of the rate for all time-series that match the metric name. The result of the expression is a new metric called requests_per_minute_total.

Advantages of Recording Rules

Recording rules provide several advantages over raw time-series data. First, they allow users to simplify complex queries and reduce the amount of data that needs to be stored. Instead of storing raw time-series data, users can create new metrics that represent the information they need. This can help reduce the storage requirements of Prometheus.

Second, recording rules can be used to preprocess data and apply filters to remove noise or outliers. For example, if a metric contains occasional spikes, a recording rule can be used to smooth out the data by applying a moving average filter. This can make it easier to identify trends and anomalies in the data.

Finally, recording rules can be used to calculate derived metrics that are not available in the original data. For example, a recording rule can be used to calculate the percentage of successful requests by dividing the number of successful requests by the total number of requests.

Creating and Adding Recording Rules

Creating a recording rule in Prometheus is a simple process. First, users need to define the name and PromQL expression for the new metric. This can be done using a text editor or a dedicated rule editor such as Grafana.

Once the recording rule is defined, it needs to be added to the Prometheus configuration file. The configuration file is a YAML file that defines the Prometheus instance, including its targets, rules, and other settings. To add a recording rule, users need to specify the name and PromQL expression in the rule_files section of the configuration file. For example:

rule_files:
  - /path/to/recording_rules.yml

In this example, recording_rules.yml is a file that contains one or more recording rules. When Prometheus starts, it reads the configuration file and loads the recording rules into memory. The new metrics created by the recording rule are then available for use in PromQL queries and visualization tools like Grafana.

Example Recording Rule YAML File

Here’s an example of what a my_recording_rules.yml file might look like:

groups:
- name: my_recording_rules
  rules:
  - record: requests_per_minute_total
    expr: sum(rate(http_requests_total[1m]))
  - record: successful_requests_percentage
    expr: sum(rate(http_requests_success_total[1m])) / sum(rate(http_requests_total[1m])) * 100

In this example, the file contains a group of recording rules named my_recording_rules. The group contains two rules:

1. The first rule creates a new metric called requests_per_minute_total. The metric represents the total number of HTTP requests per minute and is calculated by taking the sum of the rate of change of the http_requests_total metric over the last minute.
2. The second rule creates a new metric called successful_requests_percentage. The metric represents the percentage of successful HTTP requests and is calculated by dividing the rate of change of the http_requests_success_total metric over the last minute by the rate of change of the http_requests_total metric over the last minute, and then multiplying the result by 100.

To use this file in Prometheus, we would add the following lines to the rule_files section of the Prometheus configuration file:

rule_files:
  - my_recording_rules.yml

Once Prometheus is restarted, the new metrics created by the recording rules will be available for use in PromQL queries and visualizations.

You can see the recording rules in the Prometheus user interface, can be found in the “Rules” section in the top navigation bar.

Conclusion

Recording rules in Prometheus are a powerful feature that allows users to create new time-series data based on existing data, apply functions and filters, and store the results in the Prometheus database. Recording rules provide several advantages, including simplifying complex queries, reducing storage requirements, and preprocessing data. Creating and adding recording rules to the Prometheus configuration file is a straightforward process that can be done using a text editor or a dedicated rule editor. With recording rules, users can get more out of their Prometheus monitoring system and make better-informed decisions about the health and performance of their applications and infrastructure.