On this page

PromQL Cheat Sheet

Quick reference for PromQL selectors, label matchers, range vectors, aggregation operators, common queries, recording rules, and alerting rules.


Data Types

TypeDescriptionExample
Instant vectorSet of time series, single timestamphttp_requests_total
Range vectorSet of time series, range of timestampshttp_requests_total[5m]
ScalarSingle numeric value42, 3.14
StringSingle string value (limited use)"hello"

Selectors & Label Matchers

Instant Vector Selectors

# Metric name only
http_requests_total

# With label matchers
http_requests_total{method="GET"}
http_requests_total{method="GET", status="200"}
http_requests_total{job="api-server", instance=~"10.0.0.*"}

Label Matching Operators

OperatorDescriptionExample
=Exact match{method="GET"}
!=Not equal{status!="200"}
=~Regex match{path=~"/api/v[12]/.*"}
!~Regex not match{method!~"OPTIONS|HEAD"}

Regex matchers are fully anchored (RE2 syntax): {path=~"/api"} matches only exactly /api; write "/api.*" for a prefix. Since Prometheus 3.0, . also matches newlines. A selector must contain at least one matcher that does not match the empty string, so {job=~".*"} alone is rejected.

# Match on the metric name like any other label
{__name__=~"http_requests_.*", job="api"}

# UTF-8 metric / label names (Prometheus 3.0+) need quoting
{"my.dotted.metric", "service.name"="checkout"}

Range Vector Selectors

# Last 5 minutes of data
http_requests_total[5m]

# Time durations
http_requests_total[30s]      # 30 seconds
http_requests_total[5m]       # 5 minutes
http_requests_total[1h]       # 1 hour
http_requests_total[1d]       # 1 day
http_requests_total[1w]       # 1 week
http_requests_total[1h30m]    # combined units (ms, s, m, h, d, w, y)

# Subquery: range vector from an instant expression, [range:resolution]
max_over_time(rate(http_requests_total[5m])[1h:1m])

Since Prometheus 3.0, range selectors are left-open: a sample exactly at the start of the window is excluded. A [1m] window over a 1 m scrape interval can therefore contain only one sample, and rate() returns nothing.

Offset Modifier

# Value 1 hour ago
http_requests_total offset 1h

# Rate 1 day ago (compare to yesterday)
rate(http_requests_total[5m] offset 1d)

# @ modifier (at specific timestamp)
http_requests_total @ 1704067200

# Pin to the end / start of the graph range (stable value across the range)
topk(5, rate(http_requests_total[5m] @ end()))

Rate & Counter Functions

FunctionDescriptionUse With
rate(v[t])Per-second average rateCounters (always use for counters)
irate(v[t])Instantaneous rate (last 2 points)Counters (volatile, dashboards)
increase(v[t])Total increase over rangeCounters
delta(v[t])Change over rangeGauges
idelta(v[t])Change between last 2 pointsGauges
deriv(v[t])Per-second derivative (linear regression)Gauges

TIP

Always apply rate() or increase() to counters before any aggregation. Never use raw counter values directly -- they only go up and reset on restart.

# Correct: rate first, then sum
sum(rate(http_requests_total[5m])) by (method)

# Wrong: sum first, then rate (hides counter resets)
# rate(sum(http_requests_total)[5m])

rate vs irate vs increase: Pitfalls

PitfallWhat happensDo this
Window too shortrate() needs at least 2 samples in the window, otherwise no resultUse a range of at least 4x the scrape interval ([1m] for 15 s scrapes); in Grafana use $__rate_interval
increase() is not an integerrate and increase extrapolate to the window edges, so 3 requests may show as 3.4Expected; round for display, don't alert on == 1
irate() on graphs zoomed outirate looks only at the last 2 samples in the window; at 1 point per 10 min you see a random 15 s sliceUse rate for anything beyond a few-hours dashboard
irate() in alertsFlaps on every spikerate with for:
rate() of a gaugeGarbage: counter-reset logic treats every drop as a resetderiv() or delta() for gauges
rate(sum(...))Counter resets in one series look like a huge dropAlways sum(rate(...))
Counter appears mid-windowFirst increment from 0 is lost: there is no earlier sample to compare toInitialize counters to 0 at startup in your code
Short-lived series (batch jobs)Too few samples to rateUse increase() over a longer window, or a Pushgateway gauge
# Grafana: let the dashboard pick a safe window
sum(rate(http_requests_total[$__rate_interval])) by (job)

Aggregation Operators

Syntax

<aggr_op>([parameter,] <vector>) [by|without (<labels>)]

Operators

OperatorDescriptionExample
sumSum valuessum(rate(http_requests_total[5m]))
avgAverageavg(rate(node_cpu_seconds_total{mode="idle"}[5m]))
minMinimummin(node_memory_available_bytes)
maxMaximummax(container_memory_usage_bytes) by (pod)
countCount seriescount(up == 1)
stddevStandard deviationstddev(request_duration_seconds)
stdvarPopulation variancestdvar(request_duration_seconds)
topkTop K seriestopk(5, rate(http_requests_total[5m]))
bottomkBottom K seriesbottomk(3, up)
quantileQuantile across seriesquantile(0.95, request_duration_seconds)
count_valuesCount by valuecount_values("version", build_info)
groupGroup (all values become 1)group(up) by (job)
limitkAny K series per group (experimental)limitk(3, up) by (job)
limit_ratioDeterministic sample of series (experimental)limit_ratio(0.1, http_requests_total)

limitk and limit_ratio need --enable-feature=promql-experimental-functions.

by vs without

# Keep only these labels in result
sum(rate(http_requests_total[5m])) by (method, status)

# Remove these labels from result (keep everything else)
sum(rate(http_requests_total[5m])) without (instance)

Binary Operators

Arithmetic

OperatorDescription
+Addition
-Subtraction
*Multiplication
/Division
%Modulo
^Power

Comparison (Filter)

OperatorDescription
==Equal
!=Not equal
>Greater than
<Less than
>=Greater or equal
<=Less or equal
# Filter: keep only series where value > 100
http_requests_total > 100

# Return 0/1 instead of filtering (bool modifier)
http_requests_total > bool 100

Vector Matching

# One-to-one matching (same labels)
http_requests_total / http_requests_duration_seconds

# Ignore labels for matching
method_a{job="a"} / ignoring(job) method_b{job="b"}

# Match on specific labels only
method_a / on(instance, method) method_b

# Many-to-one / one-to-many
http_errors / ignoring(code) group_left http_requests

# Copy a label from an "info" metric onto your series (classic join)
rate(http_requests_total[5m])
  * on(instance) group_left(version) build_info

Logical / Set Operators

OperatorResult
a and bSeries in a that have a matching series in b
a or bAll of a, plus series in b with no match in a
a unless bSeries in a with no matching series in b
# Alert only for targets that are down AND not in maintenance
up == 0 unless on(instance) maintenance_mode == 1

Built-in Functions

Math

FunctionDescription
abs(v)Absolute value
ceil(v)Round up
floor(v)Round down
round(v, to)Round to nearest
clamp(v, min, max)Clamp values
clamp_min(v, min)Minimum clamp
clamp_max(v, max)Maximum clamp
ln(v)Natural log
log2(v)Log base 2
log10(v)Log base 10
sqrt(v)Square root
exp(v)Exponential
sgn(v)Sign (-1, 0, 1)

Range Vector Functions

FunctionDescription
avg_over_time(v[t])Average over range
min_over_time(v[t])Minimum over range
max_over_time(v[t])Maximum over range
sum_over_time(v[t])Sum over range
count_over_time(v[t])Count of samples in range
quantile_over_time(q, v[t])Quantile over range
stddev_over_time(v[t])Std deviation over range
last_over_time(v[t])Most recent value
present_over_time(v[t])1 if any sample exists
first_over_time(v[t])Oldest value in range
stdvar_over_time(v[t])Variance over range
mad_over_time(v[t])Median absolute deviation (experimental)
ts_of_max_over_time(v[t])Timestamp of the max (experimental; also ts_of_min_, ts_of_last_, ts_of_first_)
changes(v[t])Number of value changes
resets(v[t])Number of counter resets

Label Functions

FunctionDescription
label_replace(v, dst, repl, src, regex)Regex replace label
label_join(v, dst, sep, src1, src2, ...)Join labels
# Extract "api" from path="/api/v1/users"
label_replace(metric, "service", "$1", "path", "/([^/]+)/.*")

# Combine labels
label_join(metric, "full_name", "-", "first", "last")

Other Functions

FunctionDescription
histogram_quantile(q, v)Quantile from histogram buckets
predict_linear(v[t], secs)Linear prediction
double_exponential_smoothing(v[t], sf, tf)Smoothed value (renamed from holt_winters in 3.0; experimental)
day_of_week(), hour(), day_of_month()Calendar helpers (UTC), e.g. business-hours alerts
sort_by_label(v, "label")Sort by label values (experimental)
info(v)Enrich with labels from target_info (experimental)
sort(v)Sort ascending
sort_desc(v)Sort descending
time()Current Unix timestamp
timestamp(v)Timestamp of each sample
vector(scalar)Scalar to vector
scalar(v)Single-element vector to scalar
absent(v)1 if vector is empty
absent_over_time(v[t])1 if no samples in range

Common Queries

Error Rate

# Error rate (fraction)
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

# Error rate per service
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
/
sum(rate(http_requests_total[5m])) by (service)

# Error rate percentage
100 * sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m]))

Latency Percentiles (Histograms)

# p50 latency
histogram_quantile(0.50,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
)

# p95 latency
histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
)

# p99 latency per endpoint
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, handler)
)

# Average request duration
sum(rate(http_request_duration_seconds_sum[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))

# Fraction of requests faster than 300 ms (an SLI; needs a bucket at le="0.3")
sum(rate(http_request_duration_seconds_bucket{le="0.3"}[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))

histogram_quantile interpolates linearly inside a bucket, so its accuracy depends on where your bucket edges are. A p99 of "0.25" with buckets at 0.1 and 0.5 only tells you it is somewhere between them.

Native Histograms

Native (sparse, exponential-bucket) histograms are stable since Prometheus 3.8. Enable ingestion with scrape_native_histograms: true in the global or scrape config (the old --enable-feature=native-histograms flag stops doing anything in 3.9), and scrape over protobuf. A native histogram is one series: there are no _bucket, _sum or _count series and no le label.

# p95: no _bucket suffix, no "by (le)"
histogram_quantile(0.95, sum(rate(http_request_duration_seconds[5m])))

# p99 per handler
histogram_quantile(0.99, sum by (handler) (rate(http_request_duration_seconds[5m])))

# Request rate and average latency from the same series
histogram_count(sum(rate(http_request_duration_seconds[5m])))
histogram_avg(sum(rate(http_request_duration_seconds[5m])))
histogram_sum(sum(rate(http_request_duration_seconds[5m])))

# Fraction of observations between 0 and 300 ms (any threshold, no bucket needed)
histogram_fraction(0, 0.3, sum(rate(http_request_duration_seconds[5m])))

# Spread of observations
histogram_stddev(sum(rate(http_request_duration_seconds[5m])))
FunctionClassic histogram equivalent
histogram_quantile(q, h)histogram_quantile(q, sum by (le) (rate(x_bucket[5m])))
histogram_count(h)rate(x_count[5m])
histogram_sum(h)rate(x_sum[5m])
histogram_avg(h)rate(x_sum[5m]) / rate(x_count[5m])
histogram_fraction(lo, hi, h)Ratio of two _bucket series (only at existing bucket edges)
histogram_stddev(h) / histogram_stdvar(h)Not available

Migrating

always_scrape_classic_histograms: true ingests both forms during a migration, so old dashboards keep working. convert_classic_histograms_to_nhcb: true stores classic histograms as native histograms with custom buckets. Native histograms are cheaper (one series instead of one per bucket) and need no bucket tuning.

Saturation (Resource Usage)

# CPU utilization per instance
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory utilization
1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

# Disk utilization
1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})

# Container CPU usage vs limit
sum(rate(container_cpu_usage_seconds_total[5m])) by (pod)
/
sum(container_spec_cpu_quota / container_spec_cpu_period) by (pod)

# Container memory usage vs limit
sum(container_memory_working_set_bytes) by (pod)
/
sum(container_spec_memory_limit_bytes) by (pod)

Traffic

# Requests per second
sum(rate(http_requests_total[5m]))

# Requests per second by method
sum(rate(http_requests_total[5m])) by (method)

# Top 5 endpoints by request rate
topk(5, sum(rate(http_requests_total[5m])) by (handler))

Availability & Uptime

# Targets that are down
up == 0

# Count of healthy vs total
count(up == 1) / count(up)

# Availability over 30 days
avg_over_time(up[30d])
# Predict disk full (in seconds)
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 24*3600)

# Disk will be full within 24 hours
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 24*3600) < 0

# Rate of change
deriv(process_resident_memory_bytes[1h])

Recording Rules

Recording rules precompute expensive queries and store them as new time series.

# prometheus.yml or rules file
groups:
  - name: http_rules
    interval: 30s
    rules:
      # Request rate by service
      - record: job:http_requests:rate5m
        expr: sum(rate(http_requests_total[5m])) by (job)

      # Error ratio by service
      - record: job:http_errors:ratio5m
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
          /
          sum(rate(http_requests_total[5m])) by (job)

      # p95 latency by handler
      - record: handler:http_duration:p95_5m
        expr: |
          histogram_quantile(0.95,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (le, handler)
          )

TIP

Naming convention for recording rules: level:metric:operations. Example: job:http_requests:rate5m means aggregated by job, metric is http_requests, operation is rate over 5 minutes.


Alerting Rules

groups:
  - name: app_alerts
    rules:
      # High error rate
      - alert: HighErrorRate
        expr: job:http_errors:ratio5m > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate on {{ $labels.job }}"
          description: "Error rate is {{ $value | humanizePercentage }} for {{ $labels.job }}"

      # High latency
      - alert: HighLatency
        expr: handler:http_duration:p95_5m > 0.5
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High p95 latency on {{ $labels.handler }}"
          description: "p95 latency is {{ $value | humanizeDuration }}"

      # Target down
      - alert: TargetDown
        expr: up == 0
        for: 3m
        labels:
          severity: critical
        annotations:
          summary: "Target {{ $labels.instance }} is down"

      # Disk filling up
      - alert: DiskFillingUp
        expr: predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 24*3600) < 0
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Disk predicted full within 24h on {{ $labels.instance }}"

      # Pod crash looping
      - alert: PodCrashLooping
        expr: increase(kube_pod_container_status_restarts_total[1h]) > 5
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Pod {{ $labels.pod }} is crash looping"

Template Functions (Annotations)

FunctionDescriptionExample
humanizeHuman-readable number1234567 to 1.235M
humanize1024Binary unitsBytes to KiB, MiB
humanizeDurationDuration string3661 to 1h 1m 1s
humanizePercentagePercentage0.05 to 5%
humanizeTimestampTimestamp to dateUnix to readable date

Gotchas & Best Practices

PitfallSolution
Aggregating raw countersAlways apply rate() or increase() first
rate() returns no result for new or sparse series (fewer than 2 samples in the window)Use a range of at least 4x the scrape interval, e.g. [10m]
Missing series gives no resultUse or vector(0) for default
histogram_quantile wrong labelsMust keep le label in by() clause
High cardinality explosionsAvoid labels with unbounded values (user IDs, URLs)
Stale series after restartUse up to detect target health
irate vs raterate for alerts/recording rules, irate only for dashboards
or vector(0) drops all labelsOnly use it on fully aggregated (label-less) expressions
absent() on a selector with regexReturned series has no labels from the regex matcher; use absent_over_time with exact matchers in alerts
Old holt_winters queries failRenamed double_exponential_smoothing in 3.0 and behind the experimental flag
Comparisons filter, not return 0x > 5 drops series below 5; add bool for 0/1
# Default to 0 when no error series exist
sum(rate(http_requests_total{status=~"5.."}[5m])) or vector(0)

# Handle division by zero
sum(rate(errors[5m])) / (sum(rate(total[5m])) > 0)

When to Use X vs Y

DecisionChoice AChoice BUse A WhenUse B When
Rate functionrate()irate()Alerts, recording rules, smoothDashboards, spiky detail
Counter totalincrease()rate()Want total count over windowWant per-second rate
Gauge changedelta()deriv()Actual changePer-second rate of change
Aggregationby (labels)without (labels)Few labels to keepFew labels to remove
Percentilehistogram_quantilequantileHistogram metric (buckets)Across existing series
Histogram typeNativeClassicPrometheus 3.8+, new instrumentationOld clients / remote stores without native support
Latency metricHistogramSummaryNeed to aggregate across instancesSingle-instance precise quantiles (summaries cannot be aggregated)
Missing dataabsent()up == 0Metric disappeared entirelyTarget is down
Time windowShort [1m]Long [15m]High-resolution, volatileSmooth, stable for alerts

Test Yourself
  1. What function should you always apply to a counter before aggregation? rate() or increase()

  2. How do you calculate the p99 latency from a histogram? histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

  3. What is the difference between by and without in aggregation? by keeps only the listed labels; without removes the listed labels and keeps everything else.

  4. How do you provide a default value of 0 when a series does not exist? Append or vector(0) to the query.

  5. What function predicts a future value based on linear regression? predict_linear(v[t], seconds_ahead)

  6. What is the naming convention for recording rules? level:metric:operations (e.g., job:http_requests:rate5m)

  7. How do you get the value of a metric from 1 hour ago? metric_name offset 1h

  8. What modifier returns 0 or 1 instead of filtering in comparison operators? bool (e.g., http_requests_total > bool 100)

  9. How do you calculate CPU utilization from node_cpu_seconds_total? 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

  10. What function detects that a metric has completely disappeared? absent(metric_name)

Common Gotchas

  • Aggregating raw counters without rate(). Counters only go up and reset on restart. Summing raw counters hides resets and gives meaningless numbers.
  • Forgetting le in by() for histogram_quantile. Without the le label, the function cannot compute quantiles and returns garbage.
  • Using irate() in alerting rules. irate uses only the last two data points and is too volatile for stable alerts. Use rate() for alerts and recording rules.
  • Rate windows shorter than 4x the scrape interval. Since 3.0 ranges are left-open, so rate(x[1m]) with a 1 m scrape interval often has a single sample and returns nothing.
  • Querying a native histogram like a classic one. There is no _bucket series and no le label: histogram_quantile(0.9, sum(rate(x[5m]))), not ... x_bucket ... by (le).
  • Averaging percentiles. avg(histogram_quantile(...)) across instances is meaningless. Aggregate buckets first (sum by (le)), then take the quantile.
  • High-cardinality labels (user IDs, request paths). Each unique label combination creates a new time series. Unbounded labels cause memory explosions in Prometheus.

One-Liner Summary

PromQL is Prometheus's query language for slicing time-series metrics -- master rate(), histogram_quantile(), and aggregation operators to build dashboards and alerts that actually catch production issues.