Operators in Prometheus

Prometheus is a popular system and application monitoring tool used widely in the world of computing. It is an open-source tool that enables users to collect metrics, store them in a time-series database, and visualize and alert on them. Prometheus comes with a powerful query language called PromQL, which allows users to query the collected data and derive useful insights.

One of the most useful features of Prometheus is the ability to define and use operators. Operators are essentially functions that take in one or more PromQL expressions and return a boolean value. Operators can be used in alerting rules to trigger alerts when certain conditions are met.

Here are some examples of the most commonly used operators in Prometheus:

1.- Comparison Operators: Comparison operators are used to compare two values. For example, to find all the instances where the CPU usage is greater than 90%, you can use the following query:

100 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90

This query calculates the average CPU usage and subtracts it from 100 to get the percentage of idle CPU. If this percentage is less than 10%, then the CPU usage is greater than 90%. The output of this query might look something like this:

{instance="example-instance-1"} 1
{instance="example-instance-2"} 0
{instance="example-instance-3"} 1

This output shows that the CPU usage is greater than 90% for instances 1 and 3, but not for instance 2.

2.- Logical Operators: Logical operators are used to combine multiple expressions or conditions. For example, to find all the instances where the CPU usage is greater than 90% and the memory usage is less than 80%, you can use the following query:

(100 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)) > 90 and (avg(node_memory_MemFree_bytes) / avg(node_memory_MemTotal_bytes)) * 100 > 80

This query calculates the average CPU usage and subtracts it from 100 to get the percentage of idle CPU. It also calculates the percentage of free memory and checks if it is greater than 80%. The and operator combines these two conditions to find instances where both conditions are true. The output of this query might look something like this:

{instance="example-instance-1"} 1
{instance="example-instance-2"} 0
{instance="example-instance-3"} 0

This output shows that only instance 1 meets both conditions.

3.- Arithmetic Operators: Arithmetic operators are used to perform mathematical operations. For example, to find the total network traffic (in bytes) over the last 5 minutes, you can use the following query:

sum(rate(node_network_receive_bytes_total[5m])) + sum(rate(node_network_transmit_bytes_total[5m]))

This query calculates the rate of network traffic (in bytes per second) for both receiving and transmitting data and sums them up to get the total network traffic over the last 5 minutes. The output of this query might look something like this:

{instance="example-instance-1"} 123456789
{instance="example-instance-2"} 987654321
{instance="example-instance-3"} 246801357

This output shows the total network traffic (in bytes) for each instance over the last 5 minutes. For example, instance 1 had 123456789 bytes of network traffic, instance 2 had 987654321 bytes, and instance 3 had 246801357 bytes.

In conclusion, operators are a powerful feature in Prometheus that can be used to define complex queries and trigger alerts based on certain conditions. Understanding and using operators can help you gain deeper insights into the performance of your systems and applications. The output of these queries can be used to take actions such as scaling up or down resources, investigating performance issues, and ensuring system stability.