On this page

Monitoring & Alerting: From Symptom to Playbook

The Monitoring section explains how to collect metrics, logs, and traces. The Alerting section explains how to turn signals into pages that humans respond to. The Debugging Playbooks explain what to do once you are paged. This page connects the three: for each symptom the playbooks cover, it lists the signal that detects it, the alert rule that fires on it, the dashboard panel you open first, and the exact playbook branch to follow.

If you have ever been paged by an alert that said HighLatency with no link, no dashboard, and no hint where to start, this page is the fix.

The Detection Chain

Every symptom should travel the same chain, and every link should be explicit in configuration rather than in someone's memory.

flowchart LR
    S[Symptom<br/>users feel it] --> SIG[Signal<br/>metric / log / trace]
    SIG --> AR[Alert rule<br/>PromQL + for + severity]
    AR --> AN[Annotations<br/>summary, dashboard_url,<br/>runbook_url]
    AN --> DB[Dashboard<br/>first panel to open]
    AN --> PB[Playbook<br/>decision tree]
    DB --> PB
    PB --> FIX[Fix + postmortem]
    FIX -.->|new signal or rule| SIG

    style S fill:#dc2626,color:#fff
    style PB fill:#2563eb,color:#fff

Three rules hold the chain together:

  1. Every paging alert carries a runbook_url that points at a playbook, not at a wiki search page.
  2. Every alert carries a dashboard_url whose time range and variables are pre-filled from the alert's labels.
  3. Every playbook names the alert(s) that lead to it, so a postmortem can check whether detection worked.

Symptom → Signal → Alert → Dashboard → Playbook

This is the master table. Use it to audit coverage: every row should exist in your Prometheus rules and Grafana folders.

SymptomPrimary signalSupporting logs / tracesPaging alertTicket-level alerts (cause hints)First dashboard panelPlaybook
API is slowp99 of http_request_duration_seconds vs latency SLOSlow traces (duration > SLO), slow-query logsLatencySLOBurnDBPoolSaturated, CPUThrottled, GCPauseHigh, UpstreamSlowLatency heatmap by routeAPI Is Slow
Error rate spiked5xx ratio of http_requests_total vs availability SLOError logs grouped by exception, error spansAvailabilitySLOBurnErrorsAfterDeploy, DependencyErrorsError ratio by route + deploy annotationsError Rate Spiked
Intermittent 502s502 ratio at the ingress / load balancerIngress access logs with upstream_status, LB logsGateway502BurnGateway502DuringRollout, UpstreamConnReset502 rate overlaid on pod restarts and rolloutsIntermittent 502s
Database CPU at 100%DB CPU utilization, active sessionspg_stat_statements top queries, slow-query logDBCPUSaturatedDBLongTransaction, DBConnectionStorm, DBDeadTuplesActive sessions by state + top queriesDatabase CPU
Memory keeps growingWorking-set trend vs memory limitGC logs, heap snapshot artifactsMemoryExhaustionPredictedHeapBaselineRising, GCReclaimFallingWorking set vs limit with predict_linearMemory Leak
Pods keep restartingRestart count, termination reasonPrevious-container logs, Kubernetes eventsPodsRestartingUserImpactPodOOMKilled, LivenessProbeFailing, CrashLoopRestarts by reason + probe failuresPods Restarting

Page on Symptoms, Ticket on Causes

The "paging alert" column is symptom-based: it fires only when users are affected, usually through an SLO burn rate. The "ticket-level" column is cause-based: it fires on things like a saturated connection pool that might be the reason. Cause alerts should not wake anyone up, but they should appear in the same Slack thread as the page, because they tell the responder which playbook branch to jump to. See Symptom-Based Alert Design for the reasoning.

Minimum Instrumentation

None of the rules below work without the underlying telemetry. Before writing alerts, confirm each of these exists for every production service. The Observability Readiness checklist has the full list.

SourceProvidesNeeded for
App HTTP histogram (http_request_duration_seconds) with service, route, method, status labelsRED metricsSlow, errors
Runtime metrics (Node nodejs_*, JVM jvm_*, Go go_*, process_*)Heap, GC, event-loop lagSlow, memory
Connection pool metrics (HikariCP, pg-pool, pgx)Pool active / idle / pendingSlow, DB CPU
Ingress / LB metrics (nginx_ingress_controller_*, ALB via CloudWatch exporter)Edge status codes, upstream latency502s, errors
kube-state-metricsRestarts, termination reasons, replicas, rolloutsPods, 502s, memory
cAdvisor / kubeletContainer CPU, throttling, working set, probe resultsSlow, memory, pods
postgres_exporter with pg_stat_statementsSessions, locks, query stats, dead tuplesDB CPU, slow
Structured logs with trace_idPer-request detailAll
OpenTelemetry traces with exemplars on the latency histogramLatency attributionSlow, errors, 502s

Logs need structured fields and correlation IDs; traces come from OpenTelemetry. Metric naming and label cardinality rules are in Custom Metrics.

Shared Recording Rules

Every symptom section below reuses a few pre-computed series. Recording them once keeps alert expressions short and dashboards fast. See Recording Rules for the mechanics.

# rules/recording.yaml
groups:
  - name: service-red
    interval: 30s
    rules:
      # Request rate per service
      - record: service:http_requests:rate5m
        expr: sum by (namespace, service) (rate(http_requests_total[5m]))

      # 5xx ratio per service, at several windows for burn-rate alerts
      - record: service:http_errors:ratio_rate5m
        expr: |
          sum by (namespace, service) (rate(http_requests_total{status=~"5.."}[5m]))
          / sum by (namespace, service) (rate(http_requests_total[5m]))
      - record: service:http_errors:ratio_rate1h
        expr: |
          sum by (namespace, service) (rate(http_requests_total{status=~"5.."}[1h]))
          / sum by (namespace, service) (rate(http_requests_total[1h]))
      - record: service:http_errors:ratio_rate6h
        expr: |
          sum by (namespace, service) (rate(http_requests_total{status=~"5.."}[6h]))
          / sum by (namespace, service) (rate(http_requests_total[6h]))

      # Share of requests slower than the 500ms latency SLO threshold
      # (requires a bucket boundary at exactly 0.5)
      - record: service:http_slow:ratio_rate5m
        expr: |
          1 - (
            sum by (namespace, service) (rate(http_request_duration_seconds_bucket{le="0.5"}[5m]))
            / sum by (namespace, service) (rate(http_request_duration_seconds_count[5m]))
          )
      - record: service:http_slow:ratio_rate1h
        expr: |
          1 - (
            sum by (namespace, service) (rate(http_request_duration_seconds_bucket{le="0.5"}[1h]))
            / sum by (namespace, service) (rate(http_request_duration_seconds_count[1h]))
          )

      # p99 latency per service and route, for dashboards
      - record: service_route:http_latency_seconds:p99_5m
        expr: |
          histogram_quantile(0.99,
            sum by (namespace, service, route, le) (rate(http_request_duration_seconds_bucket[5m])))

Latency SLOs Need a Matching Bucket

service:http_slow:ratio_* reads the le="0.5" bucket directly. If your histogram's buckets are [0.1, 0.25, 1], there is no 0.5 bucket and the rule silently returns nothing. Choose bucket boundaries that include every SLO threshold you alert on.

Alert Annotation Conventions

Every rule on this page uses the same annotation contract, so Alertmanager templates and chat integrations can render a consistent message:

annotations:
  summary: "One line: what is wrong, for whom"
  description: "Current value and threshold, rendered from $value"
  dashboard_url: "https://grafana.example.com/d/<uid>?var-service={{ $labels.service }}&from=now-1h&to=now"
  runbook_url: "https://docs.example.com/debugging-playbooks/<playbook>"
  playbook_step: "Which branch of the decision tree to start at"

playbook_step is the field most teams skip and the one responders thank you for. "Start at Step 4: connection pool" turns a ten-minute orientation into a thirty-second one.

Symptom 1: API Is Slow

What users feel: pages hang, mobile clients time out, checkout abandonment rises.

Detection strategy: page on the latency SLO burning, not on a raw p99 threshold. A p99 spike on a low-traffic admin route should not page; a sustained rise in the share of slow requests on the main API should.

# rules/symptom-latency.yaml
groups:
  - name: symptom-latency
    rules:
      # Page: 99% of requests should complete under 500ms over 30 days.
      # 14.4x burn over 1h (confirmed by 5m) spends 2% of the monthly budget.
      - alert: LatencySLOBurn
        expr: |
          service:http_slow:ratio_rate1h > (14.4 * 0.01)
          and
          service:http_slow:ratio_rate5m > (14.4 * 0.01)
          and
          service:http_requests:rate5m > 1
        for: 2m
        labels:
          severity: page
          symptom: api-slow
        annotations:
          summary: "{{ $labels.service }} latency SLO burning 14x"
          description: "{{ $value | humanizePercentage }} of requests slower than 500ms"
          dashboard_url: "https://grafana.example.com/d/svc-latency?var-service={{ $labels.service }}"
          runbook_url: "https://docs.example.com/debugging-playbooks/api-slow"
          playbook_step: "Step 1: quantify — is it all routes or one?"

      # Ticket: cause hints that route the responder into a playbook branch
      - alert: DBPoolSaturated
        expr: |
          max by (namespace, service) (hikaricp_connections_pending) > 0
          and
          max by (namespace, service) (hikaricp_connections_active / hikaricp_connections_max) > 0.9
        for: 5m
        labels:
          severity: ticket
          symptom: api-slow
        annotations:
          summary: "{{ $labels.service }} DB pool >90% with waiters"
          runbook_url: "https://docs.example.com/debugging-playbooks/api-slow"
          playbook_step: "Step 4: check database connection pool"

      - alert: CPUThrottled
        expr: |
          sum by (namespace, pod) (rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m]))
          / sum by (namespace, pod) (rate(container_cpu_cfs_periods_total{container!=""}[5m])) > 0.25
        for: 10m
        labels:
          severity: ticket
          symptom: api-slow
        annotations:
          summary: "{{ $labels.pod }} throttled {{ $value | humanizePercentage }} of CFS periods"
          runbook_url: "https://docs.example.com/debugging-playbooks/api-slow"
          playbook_step: "Step 2: check CPU saturation"

      - alert: EventLoopLagHigh
        expr: max by (namespace, service) (nodejs_eventloop_lag_p99_seconds) > 0.1
        for: 5m
        labels:
          severity: ticket
          symptom: api-slow
        annotations:
          summary: "{{ $labels.service }} event loop p99 lag above 100ms"
          runbook_url: "https://docs.example.com/debugging-playbooks/api-slow"
          playbook_step: "Step 8: check garbage collection / Step 10: profile"

Dashboard panels to open first:

PanelQueryWhat it tells you
Latency heatmapsum by (le) (rate(http_request_duration_seconds_bucket{service="$service"}[1m]))Whole distribution shifted (resource problem) vs a new slow mode (one route or dependency)
p99 by route (top 10)topk(10, service_route:http_latency_seconds:p99_5m{service="$service"})Which route to investigate
Pool utilizationhikaricp_connections_active / hikaricp_connections_maxPlaybook Step 4
CPU throttlingRatio from CPUThrottledPlaybook Step 2

Enable exemplars on the heatmap so you can click a slow bucket and land on a trace. In Tempo, the equivalent search is:

{ resource.service.name = "checkout" && span.http.route = "/api/orders" && duration > 500ms }

The slowest child span in that trace usually names the playbook branch: a database span points to Steps 5-6 (query performance, N+1), an outbound HTTP span to Step 9 (upstream dependencies).

Symptom 2: Error Rate Spiked

What users feel: failed requests, error pages, retries.

Detection strategy: multi-window burn rate on the availability SLO, plus a cause alert that fires when errors begin within minutes of a rollout — the single most common trigger, and the first fork in the error-rate playbook.

# rules/symptom-errors.yaml
groups:
  - name: symptom-errors
    rules:
      # Page: 99.9% availability SLO, fast burn
      - alert: AvailabilitySLOBurn
        expr: |
          service:http_errors:ratio_rate1h > (14.4 * 0.001)
          and
          service:http_errors:ratio_rate5m > (14.4 * 0.001)
          and
          service:http_requests:rate5m > 1
        for: 2m
        labels:
          severity: page
          symptom: high-error-rate
        annotations:
          summary: "{{ $labels.service }} burning error budget 14x"
          description: "5xx ratio {{ $value | humanizePercentage }}"
          dashboard_url: "https://grafana.example.com/d/svc-errors?var-service={{ $labels.service }}"
          runbook_url: "https://docs.example.com/debugging-playbooks/high-error-rate"
          playbook_step: "Step 1: correlate with time — deploy, traffic, or neither?"

      # Ticket: slow burn, catches gradual degradation
      - alert: AvailabilitySLOSlowBurn
        expr: |
          service:http_errors:ratio_rate6h > (6 * 0.001)
          and
          service:http_errors:ratio_rate1h > (6 * 0.001)
        for: 15m
        labels:
          severity: ticket
          symptom: high-error-rate
        annotations:
          summary: "{{ $labels.service }} burning error budget 6x over 6h"
          runbook_url: "https://docs.example.com/debugging-playbooks/high-error-rate"

      # Cause hint: errors rose and a rollout happened in the same namespace
      - alert: ErrorsAfterDeploy
        expr: |
          (service:http_errors:ratio_rate5m > 0.01)
          and on (namespace)
          (sum by (namespace) (changes(kube_deployment_status_observed_generation[15m])) > 0)
        labels:
          severity: ticket
          symptom: high-error-rate
        annotations:
          summary: "{{ $labels.service }} errors >1% within 15m of a rollout"
          runbook_url: "https://docs.example.com/debugging-playbooks/high-error-rate"
          playbook_step: "Step 2a: deployment-related — consider rollback first"

Logs that pinpoint the cause. Group errors by type so a single new exception stands out from background noise (Loki):

sum by (error_type) (
  count_over_time({namespace="prod", service="checkout"} | json | level="error" [5m])
)

A single error_type dominating usually means a code or data issue (playbook Steps 2a and 4); errors spread across types and services usually mean infrastructure or a shared dependency (Step 3).

Dashboard: error ratio per route with Grafana annotations for every deploy (see Annotations). If the error line steps up exactly at an annotation, you have your answer before opening the playbook.

Symptom 3: Intermittent 502s

What users feel: occasional "Bad Gateway" pages that succeed on refresh.

Detection strategy: 502s are generated at the edge, so measure them there. Application metrics often show nothing, because the request never reached the app or the connection was dropped before a response was recorded. The 502 playbook has its own threshold rules; the additions here are the rollout correlation and the edge-versus-app comparison that tell you which branch to take.

# rules/symptom-502.yaml
groups:
  - name: symptom-502
    rules:
      - record: ingress:http_502:ratio_rate5m
        expr: |
          sum by (exported_namespace, exported_service) (
            rate(nginx_ingress_controller_requests{status="502"}[5m]))
          / sum by (exported_namespace, exported_service) (
            rate(nginx_ingress_controller_requests[5m]))

      # Page only when 502s are a meaningful share of traffic for 10+ minutes
      - alert: Gateway502Burn
        expr: ingress:http_502:ratio_rate5m > 0.005
        for: 10m
        labels:
          severity: page
          symptom: intermittent-502
        annotations:
          summary: "{{ $labels.exported_service }} returning {{ $value | humanizePercentage }} 502s at ingress"
          dashboard_url: "https://grafana.example.com/d/edge-502?var-service={{ $labels.exported_service }}"
          runbook_url: "https://docs.example.com/debugging-playbooks/intermittent-502"
          playbook_step: "Step 1: locate where the 502 is generated"

      # Cause hint: 502s only during rollouts → graceful shutdown / deregistration
      - alert: Gateway502DuringRollout
        expr: |
          (ingress:http_502:ratio_rate5m > 0)
          and on (exported_namespace)
          label_replace(
            sum by (namespace) (changes(kube_deployment_status_observed_generation[10m])) > 0,
            "exported_namespace", "$1", "namespace", "(.*)")
        labels:
          severity: ticket
          symptom: intermittent-502
        annotations:
          summary: "502s at ingress coincide with a rollout in {{ $labels.exported_namespace }}"
          runbook_url: "https://docs.example.com/debugging-playbooks/intermittent-502"
          playbook_step: "Step 5: check graceful shutdown (preStop, terminationGracePeriod)"

The comparison panel that decides the branch:

Edge 502 rateApp 5xx ratePod restarts / rolloutLikely branch
RisingFlatRollout in progressGraceful shutdown (Step 5)
RisingFlatNoneKeep-alive mismatch (Step 4) or DNS (Step 6)
RisingRisingRestarts risingPods dying under load — go to Pods Restarting
RisingFlatHPA scaling upResource exhaustion / slow scale-up (Step 8)

Put edge 502 rate, service:http_errors:ratio_rate5m, and sum(increase(kube_pod_container_status_restarts_total[5m])) on one time-series panel with three axes, plus rollout annotations. In ingress-nginx access logs, upstream_status of - or a connection-reset error with a 502 response confirms the request never got an app-level response.

Symptom 4: Database CPU at 100%

What users feel: everything that touches the database slows down at once; often surfaces first as Symptom 1.

Detection strategy: DB CPU on its own is a cause, not a symptom — a batch job can legitimately pin CPU at 3 AM. Page when CPU is saturated and active sessions are queuing (work is waiting), and use cause alerts to point at the branch.

# rules/symptom-db-cpu.yaml
# Metric names follow postgres_exporter; for RDS/Aurora, CPU comes from the
# CloudWatch exporter (e.g. aws_rds_cpuutilization_average) — swap accordingly.
groups:
  - name: symptom-db-cpu
    rules:
      # Normalize CPU to a 0-1 ratio. Self-hosted with node_exporter:
      #   1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))
      - record: db:cpu_utilization:ratio
        expr: avg by (instance) (aws_rds_cpuutilization_average) / 100

      # vCPU count per instance. Self-hosted: count by (instance) (node_cpu_seconds_total{mode="idle"})
      # Managed: a static series per instance class, e.g. via a small exporter or file_sd labels.
      - record: db:vcpus
        expr: max by (instance) (db_instance_vcpus)

      - alert: DBCPUSaturated
        expr: |
          (db:cpu_utilization:ratio > 0.9)
          and on (instance)
          (sum by (instance) (pg_stat_activity_count{state="active"}) > on (instance) 2 * db:vcpus)
        for: 5m
        labels:
          severity: page
          symptom: database-cpu
        annotations:
          summary: "{{ $labels.instance }} CPU >90% with active sessions >2x vCPUs"
          dashboard_url: "https://grafana.example.com/d/postgres?var-instance={{ $labels.instance }}"
          runbook_url: "https://docs.example.com/debugging-playbooks/database-cpu"
          playbook_step: "Step 2: find the CPU-consuming queries in pg_stat_statements"

      - alert: DBConnectionStorm
        expr: |
          deriv(sum by (instance) (pg_stat_activity_count)[10m:1m]) > 5
        for: 5m
        labels:
          severity: ticket
          symptom: database-cpu
        annotations:
          summary: "{{ $labels.instance }} connections climbing >5/s"
          runbook_url: "https://docs.example.com/debugging-playbooks/database-cpu"
          playbook_step: "Step 7: check for connection storms"

      - alert: DBSeqScansSpiking
        expr: |
          sum by (instance, relname) (rate(pg_stat_user_tables_seq_tup_read[10m]))
          > 10 * sum by (instance, relname) (rate(pg_stat_user_tables_seq_tup_read[1d] offset 1d))
        for: 15m
        labels:
          severity: ticket
          symptom: database-cpu
        annotations:
          summary: "Sequential scan volume on {{ $labels.relname }} 10x yesterday's rate"
          runbook_url: "https://docs.example.com/debugging-playbooks/database-cpu"
          playbook_step: "Step 3: check for sequential scans (missing index)"

Active sessions above twice the vCPU count means work is queuing for CPU, which is what turns high utilization into user-visible latency. Relabel the CloudWatch and postgres_exporter targets so both expose the same instance label, or the on (instance) join matches nothing.

Dashboard panels: active sessions stacked by state (active, idle in transaction, idle), lock waits, the top-10 queries by total execution time from pg_stat_statements, and replication lag. The PostgreSQL dashboard has a production JSON you can import. A large idle in transaction band points to the lock-contention branch (Step 4); a growing active band with one dominant query points to Steps 2-3.

Long-running transactions and dead-tuple alerts already live in the playbook's Prevention section — reuse them rather than defining near-duplicates.

Symptom 5: Memory Keeps Growing

What users feel: nothing, until the pod is OOM-killed — then a burst of errors or 502s every few hours.

Detection strategy: do not alert on absolute memory. Alert on time until the limit using predict_linear, which fires early enough to capture a heap snapshot before the kill destroys the evidence. The memory-leak playbook uses deriv on RSS for its own rule; predict_linear against the container limit is the complementary page-worthy form.

# rules/symptom-memory.yaml
groups:
  - name: symptom-memory
    rules:
      - record: pod:memory_working_set_bytes
        expr: |
          sum by (namespace, pod, container) (
            container_memory_working_set_bytes{container!="", container!="POD"})

      - record: pod:memory_limit_bytes
        expr: |
          sum by (namespace, pod, container) (
            kube_pod_container_resource_limits{resource="memory"})

      # Page: at the current 6h trend, the container hits its limit within 4h
      - alert: MemoryExhaustionPredicted
        expr: |
          predict_linear(pod:memory_working_set_bytes[6h], 4 * 3600)
            > pod:memory_limit_bytes
          and
          pod:memory_working_set_bytes / pod:memory_limit_bytes > 0.6
        for: 30m
        labels:
          severity: page
          symptom: memory-leak
        annotations:
          summary: "{{ $labels.pod }} will hit its memory limit within ~4h; capture a heap snapshot now"
          dashboard_url: "https://grafana.example.com/d/svc-memory?var-pod={{ $labels.pod }}"
          runbook_url: "https://docs.example.com/debugging-playbooks/memory-leak"
          playbook_step: "Step 1: confirm the leak is real; Step 3: take heap snapshots before OOM"

      # Ticket: the post-GC baseline keeps rising (sawtooth with a climbing floor)
      - alert: HeapBaselineRising
        expr: |
          deriv(min_over_time(nodejs_heap_size_used_bytes[30m])[6h:30m]) > 0
          and
          min_over_time(nodejs_heap_size_used_bytes[30m])
            > 1.5 * min_over_time(nodejs_heap_size_used_bytes[30m] offset 6h)
        for: 1h
        labels:
          severity: ticket
          symptom: memory-leak
        annotations:
          summary: "{{ $labels.pod }} heap floor up 50% in 6h"
          runbook_url: "https://docs.example.com/debugging-playbooks/memory-leak"
          playbook_step: "Step 2: heap vs non-heap"

Why min_over_time

Heap usage is a sawtooth: it climbs between garbage collections and drops after each one. The peaks tell you little. The floor — the minimum over a window longer than a GC cycle — is what a leak raises. Alerting on the floor eliminates nearly all false positives from normal allocation churn.

Dashboard panels: working set versus limit with the predict_linear projection drawn as a dashed line; heap used floor (min_over_time) versus heap total; GC duration and frequency; pod age (a leak shows as memory correlated with uptime across replicas). If RSS grows but heap does not, the playbook's non-heap branch (buffers, native modules) applies.

Symptom 6: Pods Keep Restarting

What users feel: depends on replica count. With enough replicas and good readiness probes, nothing. With few replicas, errors and 502s on every restart.

Detection strategy: restarts alone are a ticket; restarts that reduce available capacity are a page. The pods-restarting playbook ships per-pod alert rules; the version below aggregates to the workload level and routes by termination reason so the first message already names the branch.

# rules/symptom-pods.yaml
groups:
  - name: symptom-pods
    rules:
      # Page: restarts are reducing available replicas
      - alert: PodsRestartingUserImpact
        expr: |
          sum by (namespace, deployment) (kube_deployment_status_replicas_unavailable) > 0
          and on (namespace)
          sum by (namespace) (increase(kube_pod_container_status_restarts_total[15m])) > 3
        for: 5m
        labels:
          severity: page
          symptom: pods-restarting
        annotations:
          summary: "{{ $labels.deployment }} has unavailable replicas and repeated restarts"
          dashboard_url: "https://grafana.example.com/d/k8s-workload?var-namespace={{ $labels.namespace }}&var-deployment={{ $labels.deployment }}"
          runbook_url: "https://docs.example.com/debugging-playbooks/pods-restarting"
          playbook_step: "Step 2: check the exit code"

      # Cause hints, one per termination reason / waiting reason
      - alert: LivenessProbeFailing
        expr: |
          sum by (namespace, pod) (
            rate(prober_probe_total{probe_type="Liveness", result="failed"}[5m])) > 0
        for: 10m
        labels:
          severity: ticket
          symptom: pods-restarting
        annotations:
          summary: "Liveness probe failing on {{ $labels.pod }}"
          runbook_url: "https://docs.example.com/debugging-playbooks/pods-restarting"
          playbook_step: "Step 7: diagnose liveness probe failures"

      - alert: CrashLoop
        expr: |
          max by (namespace, pod, container) (
            kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}) == 1
        for: 5m
        labels:
          severity: ticket
          symptom: pods-restarting
        annotations:
          summary: "{{ $labels.pod }}/{{ $labels.container }} in CrashLoopBackOff"
          runbook_url: "https://docs.example.com/debugging-playbooks/pods-restarting"
          playbook_step: "Step 3: previous container logs; Step 5: CrashLoopBackOff"

Termination reason → branch:

last_terminated_reason / exit codeMeaningBranch
OOMKilled / 137Exceeded memory limitStep 4, then Memory Leak if usage trends upward
Error / 1Application crashed at startup or runtimeSteps 3 and 5
Error / 143SIGTERM not handled within grace periodGraceful shutdown fix
Completed / 0Process exited cleanly but should not haveStep 5 (entrypoint/command)
Waiting ImagePullBackOffRegistry or tag problemStep 6

Dashboard panels: restarts by reason as a stacked bar (sum by (reason) (kube_pod_container_status_last_terminated_reason * on (namespace, pod, container) group_left increase(kube_pod_container_status_restarts_total[1h]))), liveness and readiness probe failure rates, memory working set versus limit (shared with Symptom 5), and node pressure conditions. Kubernetes events in Loki — {job="kubernetes-events"} |= "Back-off restarting failed container" — give a timeline. For cluster-level diagnosis, see Kubernetes Troubleshooting.

How Symptoms Cascade

Symptoms rarely arrive alone. A leaking service gets OOM-killed, which produces 502s during restarts, which raises the error rate. If all four alerts page, the responder gets four pages for one problem and may start in the wrong playbook.

flowchart LR
    ML[Memory leak] --> OOM[Pods restarting<br/>OOMKilled]
    DBC[DB CPU saturated] --> SLOW[API slow]
    SLOW --> TO[Upstream timeouts]
    OOM --> E502[Intermittent 502s]
    TO --> ERR[Error rate spike]
    E502 --> ERR

    style ML fill:#fef3c7,stroke:#f59e0b
    style DBC fill:#fef3c7,stroke:#f59e0b
    style ERR fill:#fee2e2,stroke:#ef4444

Use Alertmanager inhibition so the upstream cause suppresses the downstream symptom for the same service, and group alerts by service so they land in one notification:

# alertmanager.yaml (excerpt)
route:
  receiver: slack-triage
  group_by: [namespace, service]
  group_wait: 30s
  group_interval: 5m
  routes:
    - matchers: [severity="page"]
      receiver: pagerduty
      continue: true
    - matchers: [severity="ticket"]
      receiver: slack-triage

inhibit_rules:
  # Pods crash-looping explains 502s and errors for the same namespace
  - source_matchers: [alertname="PodsRestartingUserImpact"]
    target_matchers: [alertname=~"Gateway502Burn|AvailabilitySLOBurn"]
    equal: [namespace]
  # Saturated DB explains slow APIs; responders start at the DB playbook
  - source_matchers: [alertname="DBCPUSaturated"]
    target_matchers: [alertname="LatencySLOBurn"]
    equal: [namespace]

Inhibit Carefully

Inhibition hides alerts. If the inhibition rule is wrong — the DB is fine but the API is slow for an unrelated reason — you have suppressed the page that would have told you. Only inhibit pairs where the causal link is near-certain, and always keep the inhibited alert visible in the triage channel. Routing and grouping are covered in depth in Escalation Policies.

The Triage Dashboard

Build one dashboard that mirrors the master table: one row per symptom, each row showing the primary signal, a status indicator for the paging alert, and a text panel linking the playbook. On-call engineers open it first when they do not yet know which symptom they are looking at.

{
  "title": "Symptom Triage",
  "uid": "symptom-triage",
  "templating": {
    "list": [
      { "name": "namespace", "type": "query", "query": "label_values(service:http_requests:rate5m, namespace)" },
      { "name": "service", "type": "query", "query": "label_values(service:http_requests:rate5m{namespace=\"$namespace\"}, service)" }
    ]
  },
  "panels": [
    { "type": "row", "title": "API slow → /debugging-playbooks/api-slow", "gridPos": { "x": 0, "y": 0, "w": 24, "h": 1 } },
    {
      "type": "timeseries", "title": "Share of requests > 500ms",
      "gridPos": { "x": 0, "y": 1, "w": 16, "h": 6 },
      "targets": [{ "expr": "service:http_slow:ratio_rate5m{namespace=\"$namespace\", service=\"$service\"}" }],
      "fieldConfig": { "defaults": { "unit": "percentunit",
        "thresholds": { "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 0.144 } ] } } },
      "links": [{ "title": "API Is Slow playbook", "url": "https://docs.example.com/debugging-playbooks/api-slow" }]
    },
    {
      "type": "stat", "title": "LatencySLOBurn",
      "gridPos": { "x": 16, "y": 1, "w": 8, "h": 6 },
      "targets": [{ "expr": "count(ALERTS{alertname=\"LatencySLOBurn\", alertstate=\"firing\", service=\"$service\"}) or vector(0)" }],
      "fieldConfig": { "defaults": { "mappings": [
        { "type": "value", "options": { "0": { "text": "OK", "color": "green" } } } ],
        "thresholds": { "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 1 } ] } } }
    },
    { "type": "row", "title": "Errors → /debugging-playbooks/high-error-rate", "gridPos": { "x": 0, "y": 7, "w": 24, "h": 1 } },
    {
      "type": "timeseries", "title": "5xx ratio (app) vs 502 ratio (edge)",
      "gridPos": { "x": 0, "y": 8, "w": 24, "h": 6 },
      "targets": [
        { "expr": "service:http_errors:ratio_rate5m{namespace=\"$namespace\", service=\"$service\"}", "legendFormat": "app 5xx" },
        { "expr": "ingress:http_502:ratio_rate5m{exported_namespace=\"$namespace\", exported_service=\"$service\"}", "legendFormat": "edge 502" }
      ],
      "fieldConfig": { "defaults": { "unit": "percentunit" } }
    }
  ]
}

The excerpt shows two rows; the full dashboard repeats the pattern for DB CPU, memory, and pods. Provision it as code alongside the rules (see Provisioning Dashboards as Code) so a rule change and its panel change land in the same pull request.

Testing the Chain

Alert rules are code, and the detection chain can break silently: a renamed metric, a label that changed from status to code, a playbook that moved. Two checks catch most breakage.

Unit-Test Rules with promtool

# tests/symptom-memory_test.yaml
rule_files:
  - ../rules/symptom-memory.yaml

evaluation_interval: 1m

tests:
  - interval: 5m
    input_series:
      # 1 GiB limit; working set starts at 600 MiB and grows 5 MiB every 5 minutes (60 MiB/h).
      # The 4h projection crosses the limit at ~3.1h; with for: 30m it fires by ~3.6h.
      - series: 'container_memory_working_set_bytes{namespace="prod", pod="api-1", container="api"}'
        values: '629145600+5242880x120'
      - series: 'kube_pod_container_resource_limits{namespace="prod", pod="api-1", container="api", resource="memory"}'
        values: '1073741824x120'
    alert_rule_test:
      - eval_time: 2h
        alertname: MemoryExhaustionPredicted
        exp_alerts: []
      - eval_time: 5h
        alertname: MemoryExhaustionPredicted
        exp_alerts:
          - exp_labels:
              severity: page
              symptom: memory-leak
              namespace: prod
              pod: api-1
              container: api
            exp_annotations:
              summary: "api-1 will hit its memory limit within ~4h; capture a heap snapshot now"
              dashboard_url: "https://grafana.example.com/d/svc-memory?var-pod=api-1"
              runbook_url: "https://docs.example.com/debugging-playbooks/memory-leak"
              playbook_step: "Step 1: confirm the leak is real; Step 3: take heap snapshots before OOM"

Run promtool test rules tests/*.yaml in CI. The Alert Testing Framework section covers testing burn-rate rules across windows.

A small CI check that every paging rule has a runbook_url pointing at a playbook that actually exists:

#!/usr/bin/env python3
"""Fail CI if a paging alert lacks a dashboard or a valid playbook link."""
import pathlib
import sys

import yaml

PLAYBOOK_DIR = pathlib.Path("docs/debugging-playbooks")
PREFIX = "https://docs.example.com/debugging-playbooks/"
REQUIRED = ("summary", "dashboard_url", "runbook_url")


def check(rule_file: pathlib.Path) -> list[str]:
    problems = []
    for group in yaml.safe_load(rule_file.read_text()).get("groups", []):
        for rule in group.get("rules", []):
            if "alert" not in rule:
                continue  # recording rule
            name = rule["alert"]
            ann = rule.get("annotations", {})
            severity = rule.get("labels", {}).get("severity")
            if severity == "page":
                missing = [k for k in REQUIRED if not ann.get(k)]
                if missing:
                    problems.append(f"{rule_file}:{name} missing {', '.join(missing)}")
            url = ann.get("runbook_url", "")
            if url.startswith(PREFIX):
                slug = url[len(PREFIX):].split("#")[0].strip("/")
                if not (PLAYBOOK_DIR / f"{slug}.md").exists():
                    problems.append(f"{rule_file}:{name} runbook_url -> missing playbook {slug}")
    return problems


if __name__ == "__main__":
    files = [pathlib.Path(p) for p in sys.argv[1:]] or sorted(pathlib.Path("rules").glob("*.yaml"))
    problems = [p for f in files for p in check(f)]
    print("\n".join(problems) or f"OK: {len(files)} rule files")
    sys.exit(1 if problems else 0)

Coverage Gaps to Watch For

GapHow it shows upFix
Symptom with no paging alertCustomers report the problem before the pager doesAdd a row to the master table; every postmortem should ask "did an alert fire?"
Cause alert that pagesOn-call woken for a busy-but-healthy DB at 3 AMDowngrade to ticket; page only on the user-facing symptom
Alert with no playbook_stepResponder opens the playbook at the top every timeAdd the step; review annotations in the postmortem
Edge metrics missing502s invisible because the app never saw the requestScrape the ingress controller or LB; see Symptom 3
Low-traffic servicesBurn-rate ratios swing wildly on a handful of requestsKeep the rate5m > 1 guard; use synthetic probes for low-traffic paths
Dashboards drift from rulesPanel shows a metric the alert no longer usesProvision dashboards and rules from the same repository

For broader pitfalls — dashboard rot, vanity metrics, cardinality explosions — see Monitoring Antipatterns.

War Story (illustrative)

A team had good alerts for every symptom in the master table, but their runbook_url annotations all pointed at a wiki page that had been reorganized a year earlier. Every page during a regional database failover led responders to a "page not found". The incident ran 40 minutes longer than it needed to, mostly spent finding the right playbook. The fix was the link-lint script above in CI, plus moving playbooks into the same repository as the rules so a rename broke the build instead of the incident.

Key Takeaways

  • Treat detection as a chain — symptom, signal, alert, dashboard, playbook — and make every link explicit in configuration.
  • Page on user-facing symptoms (SLO burn rates); route cause-level alerts to a ticket channel where they hint at the playbook branch.
  • Put dashboard_url, runbook_url, and playbook_step on every paging alert.
  • Measure 502s at the edge, memory as time-to-limit, and DB CPU together with session queuing.
  • Use inhibition and grouping so one root cause produces one notification, but inhibit only near-certain causal pairs.
  • Test rules with promtool and lint playbook links in CI so the chain cannot rot silently.