Monitoring & Alerting: From Symptom to Playbook
The Monitoring section explains how to collect metrics, logs, and traces. The Alerting section explains how to turn signals into pages that humans respond to. The Debugging Playbooks explain what to do once you are paged. This page connects the three: for each symptom the playbooks cover, it lists the signal that detects it, the alert rule that fires on it, the dashboard panel you open first, and the exact playbook branch to follow.
If you have ever been paged by an alert that said HighLatency with no link, no dashboard, and no hint where to start, this page is the fix.
The Detection Chain
Every symptom should travel the same chain, and every link should be explicit in configuration rather than in someone's memory.
flowchart LR
S[Symptom<br/>users feel it] --> SIG[Signal<br/>metric / log / trace]
SIG --> AR[Alert rule<br/>PromQL + for + severity]
AR --> AN[Annotations<br/>summary, dashboard_url,<br/>runbook_url]
AN --> DB[Dashboard<br/>first panel to open]
AN --> PB[Playbook<br/>decision tree]
DB --> PB
PB --> FIX[Fix + postmortem]
FIX -.->|new signal or rule| SIG
style S fill:#dc2626,color:#fff
style PB fill:#2563eb,color:#fff
Three rules hold the chain together:
- Every paging alert carries a
runbook_urlthat points at a playbook, not at a wiki search page. - Every alert carries a
dashboard_urlwhose time range and variables are pre-filled from the alert's labels. - Every playbook names the alert(s) that lead to it, so a postmortem can check whether detection worked.
Symptom → Signal → Alert → Dashboard → Playbook
This is the master table. Use it to audit coverage: every row should exist in your Prometheus rules and Grafana folders.
| Symptom | Primary signal | Supporting logs / traces | Paging alert | Ticket-level alerts (cause hints) | First dashboard panel | Playbook |
|---|---|---|---|---|---|---|
| API is slow | p99 of http_request_duration_seconds vs latency SLO | Slow traces (duration > SLO), slow-query logs | LatencySLOBurn | DBPoolSaturated, CPUThrottled, GCPauseHigh, UpstreamSlow | Latency heatmap by route | API Is Slow |
| Error rate spiked | 5xx ratio of http_requests_total vs availability SLO | Error logs grouped by exception, error spans | AvailabilitySLOBurn | ErrorsAfterDeploy, DependencyErrors | Error ratio by route + deploy annotations | Error Rate Spiked |
| Intermittent 502s | 502 ratio at the ingress / load balancer | Ingress access logs with upstream_status, LB logs | Gateway502Burn | Gateway502DuringRollout, UpstreamConnReset | 502 rate overlaid on pod restarts and rollouts | Intermittent 502s |
| Database CPU at 100% | DB CPU utilization, active sessions | pg_stat_statements top queries, slow-query log | DBCPUSaturated | DBLongTransaction, DBConnectionStorm, DBDeadTuples | Active sessions by state + top queries | Database CPU |
| Memory keeps growing | Working-set trend vs memory limit | GC logs, heap snapshot artifacts | MemoryExhaustionPredicted | HeapBaselineRising, GCReclaimFalling | Working set vs limit with predict_linear | Memory Leak |
| Pods keep restarting | Restart count, termination reason | Previous-container logs, Kubernetes events | PodsRestartingUserImpact | PodOOMKilled, LivenessProbeFailing, CrashLoop | Restarts by reason + probe failures | Pods Restarting |
Page on Symptoms, Ticket on Causes
The "paging alert" column is symptom-based: it fires only when users are affected, usually through an SLO burn rate. The "ticket-level" column is cause-based: it fires on things like a saturated connection pool that might be the reason. Cause alerts should not wake anyone up, but they should appear in the same Slack thread as the page, because they tell the responder which playbook branch to jump to. See Symptom-Based Alert Design for the reasoning.
Minimum Instrumentation
None of the rules below work without the underlying telemetry. Before writing alerts, confirm each of these exists for every production service. The Observability Readiness checklist has the full list.
| Source | Provides | Needed for |
|---|---|---|
App HTTP histogram (http_request_duration_seconds) with service, route, method, status labels | RED metrics | Slow, errors |
Runtime metrics (Node nodejs_*, JVM jvm_*, Go go_*, process_*) | Heap, GC, event-loop lag | Slow, memory |
| Connection pool metrics (HikariCP, pg-pool, pgx) | Pool active / idle / pending | Slow, DB CPU |
Ingress / LB metrics (nginx_ingress_controller_*, ALB via CloudWatch exporter) | Edge status codes, upstream latency | 502s, errors |
kube-state-metrics | Restarts, termination reasons, replicas, rollouts | Pods, 502s, memory |
| cAdvisor / kubelet | Container CPU, throttling, working set, probe results | Slow, memory, pods |
postgres_exporter with pg_stat_statements | Sessions, locks, query stats, dead tuples | DB CPU, slow |
Structured logs with trace_id | Per-request detail | All |
| OpenTelemetry traces with exemplars on the latency histogram | Latency attribution | Slow, errors, 502s |
Logs need structured fields and correlation IDs; traces come from OpenTelemetry. Metric naming and label cardinality rules are in Custom Metrics.
Shared Recording Rules
Every symptom section below reuses a few pre-computed series. Recording them once keeps alert expressions short and dashboards fast. See Recording Rules for the mechanics.
# rules/recording.yaml
groups:
- name: service-red
interval: 30s
rules:
# Request rate per service
- record: service:http_requests:rate5m
expr: sum by (namespace, service) (rate(http_requests_total[5m]))
# 5xx ratio per service, at several windows for burn-rate alerts
- record: service:http_errors:ratio_rate5m
expr: |
sum by (namespace, service) (rate(http_requests_total{status=~"5.."}[5m]))
/ sum by (namespace, service) (rate(http_requests_total[5m]))
- record: service:http_errors:ratio_rate1h
expr: |
sum by (namespace, service) (rate(http_requests_total{status=~"5.."}[1h]))
/ sum by (namespace, service) (rate(http_requests_total[1h]))
- record: service:http_errors:ratio_rate6h
expr: |
sum by (namespace, service) (rate(http_requests_total{status=~"5.."}[6h]))
/ sum by (namespace, service) (rate(http_requests_total[6h]))
# Share of requests slower than the 500ms latency SLO threshold
# (requires a bucket boundary at exactly 0.5)
- record: service:http_slow:ratio_rate5m
expr: |
1 - (
sum by (namespace, service) (rate(http_request_duration_seconds_bucket{le="0.5"}[5m]))
/ sum by (namespace, service) (rate(http_request_duration_seconds_count[5m]))
)
- record: service:http_slow:ratio_rate1h
expr: |
1 - (
sum by (namespace, service) (rate(http_request_duration_seconds_bucket{le="0.5"}[1h]))
/ sum by (namespace, service) (rate(http_request_duration_seconds_count[1h]))
)
# p99 latency per service and route, for dashboards
- record: service_route:http_latency_seconds:p99_5m
expr: |
histogram_quantile(0.99,
sum by (namespace, service, route, le) (rate(http_request_duration_seconds_bucket[5m])))
Latency SLOs Need a Matching Bucket
service:http_slow:ratio_* reads the le="0.5" bucket directly. If your histogram's buckets are [0.1, 0.25, 1], there is no 0.5 bucket and the rule silently returns nothing. Choose bucket boundaries that include every SLO threshold you alert on.
Alert Annotation Conventions
Every rule on this page uses the same annotation contract, so Alertmanager templates and chat integrations can render a consistent message:
annotations:
summary: "One line: what is wrong, for whom"
description: "Current value and threshold, rendered from $value"
dashboard_url: "https://grafana.example.com/d/<uid>?var-service={{ $labels.service }}&from=now-1h&to=now"
runbook_url: "https://docs.example.com/debugging-playbooks/<playbook>"
playbook_step: "Which branch of the decision tree to start at"
playbook_step is the field most teams skip and the one responders thank you for. "Start at Step 4: connection pool" turns a ten-minute orientation into a thirty-second one.
Symptom 1: API Is Slow
What users feel: pages hang, mobile clients time out, checkout abandonment rises.
Detection strategy: page on the latency SLO burning, not on a raw p99 threshold. A p99 spike on a low-traffic admin route should not page; a sustained rise in the share of slow requests on the main API should.
# rules/symptom-latency.yaml
groups:
- name: symptom-latency
rules:
# Page: 99% of requests should complete under 500ms over 30 days.
# 14.4x burn over 1h (confirmed by 5m) spends 2% of the monthly budget.
- alert: LatencySLOBurn
expr: |
service:http_slow:ratio_rate1h > (14.4 * 0.01)
and
service:http_slow:ratio_rate5m > (14.4 * 0.01)
and
service:http_requests:rate5m > 1
for: 2m
labels:
severity: page
symptom: api-slow
annotations:
summary: "{{ $labels.service }} latency SLO burning 14x"
description: "{{ $value | humanizePercentage }} of requests slower than 500ms"
dashboard_url: "https://grafana.example.com/d/svc-latency?var-service={{ $labels.service }}"
runbook_url: "https://docs.example.com/debugging-playbooks/api-slow"
playbook_step: "Step 1: quantify — is it all routes or one?"
# Ticket: cause hints that route the responder into a playbook branch
- alert: DBPoolSaturated
expr: |
max by (namespace, service) (hikaricp_connections_pending) > 0
and
max by (namespace, service) (hikaricp_connections_active / hikaricp_connections_max) > 0.9
for: 5m
labels:
severity: ticket
symptom: api-slow
annotations:
summary: "{{ $labels.service }} DB pool >90% with waiters"
runbook_url: "https://docs.example.com/debugging-playbooks/api-slow"
playbook_step: "Step 4: check database connection pool"
- alert: CPUThrottled
expr: |
sum by (namespace, pod) (rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m]))
/ sum by (namespace, pod) (rate(container_cpu_cfs_periods_total{container!=""}[5m])) > 0.25
for: 10m
labels:
severity: ticket
symptom: api-slow
annotations:
summary: "{{ $labels.pod }} throttled {{ $value | humanizePercentage }} of CFS periods"
runbook_url: "https://docs.example.com/debugging-playbooks/api-slow"
playbook_step: "Step 2: check CPU saturation"
- alert: EventLoopLagHigh
expr: max by (namespace, service) (nodejs_eventloop_lag_p99_seconds) > 0.1
for: 5m
labels:
severity: ticket
symptom: api-slow
annotations:
summary: "{{ $labels.service }} event loop p99 lag above 100ms"
runbook_url: "https://docs.example.com/debugging-playbooks/api-slow"
playbook_step: "Step 8: check garbage collection / Step 10: profile"
Dashboard panels to open first:
| Panel | Query | What it tells you |
|---|---|---|
| Latency heatmap | sum by (le) (rate(http_request_duration_seconds_bucket{service="$service"}[1m])) | Whole distribution shifted (resource problem) vs a new slow mode (one route or dependency) |
| p99 by route (top 10) | topk(10, service_route:http_latency_seconds:p99_5m{service="$service"}) | Which route to investigate |
| Pool utilization | hikaricp_connections_active / hikaricp_connections_max | Playbook Step 4 |
| CPU throttling | Ratio from CPUThrottled | Playbook Step 2 |
Enable exemplars on the heatmap so you can click a slow bucket and land on a trace. In Tempo, the equivalent search is:
{ resource.service.name = "checkout" && span.http.route = "/api/orders" && duration > 500ms }
The slowest child span in that trace usually names the playbook branch: a database span points to Steps 5-6 (query performance, N+1), an outbound HTTP span to Step 9 (upstream dependencies).
Symptom 2: Error Rate Spiked
What users feel: failed requests, error pages, retries.
Detection strategy: multi-window burn rate on the availability SLO, plus a cause alert that fires when errors begin within minutes of a rollout — the single most common trigger, and the first fork in the error-rate playbook.
# rules/symptom-errors.yaml
groups:
- name: symptom-errors
rules:
# Page: 99.9% availability SLO, fast burn
- alert: AvailabilitySLOBurn
expr: |
service:http_errors:ratio_rate1h > (14.4 * 0.001)
and
service:http_errors:ratio_rate5m > (14.4 * 0.001)
and
service:http_requests:rate5m > 1
for: 2m
labels:
severity: page
symptom: high-error-rate
annotations:
summary: "{{ $labels.service }} burning error budget 14x"
description: "5xx ratio {{ $value | humanizePercentage }}"
dashboard_url: "https://grafana.example.com/d/svc-errors?var-service={{ $labels.service }}"
runbook_url: "https://docs.example.com/debugging-playbooks/high-error-rate"
playbook_step: "Step 1: correlate with time — deploy, traffic, or neither?"
# Ticket: slow burn, catches gradual degradation
- alert: AvailabilitySLOSlowBurn
expr: |
service:http_errors:ratio_rate6h > (6 * 0.001)
and
service:http_errors:ratio_rate1h > (6 * 0.001)
for: 15m
labels:
severity: ticket
symptom: high-error-rate
annotations:
summary: "{{ $labels.service }} burning error budget 6x over 6h"
runbook_url: "https://docs.example.com/debugging-playbooks/high-error-rate"
# Cause hint: errors rose and a rollout happened in the same namespace
- alert: ErrorsAfterDeploy
expr: |
(service:http_errors:ratio_rate5m > 0.01)
and on (namespace)
(sum by (namespace) (changes(kube_deployment_status_observed_generation[15m])) > 0)
labels:
severity: ticket
symptom: high-error-rate
annotations:
summary: "{{ $labels.service }} errors >1% within 15m of a rollout"
runbook_url: "https://docs.example.com/debugging-playbooks/high-error-rate"
playbook_step: "Step 2a: deployment-related — consider rollback first"
Logs that pinpoint the cause. Group errors by type so a single new exception stands out from background noise (Loki):
sum by (error_type) (
count_over_time({namespace="prod", service="checkout"} | json | level="error" [5m])
)
A single error_type dominating usually means a code or data issue (playbook Steps 2a and 4); errors spread across types and services usually mean infrastructure or a shared dependency (Step 3).
Dashboard: error ratio per route with Grafana annotations for every deploy (see Annotations). If the error line steps up exactly at an annotation, you have your answer before opening the playbook.
Symptom 3: Intermittent 502s
What users feel: occasional "Bad Gateway" pages that succeed on refresh.
Detection strategy: 502s are generated at the edge, so measure them there. Application metrics often show nothing, because the request never reached the app or the connection was dropped before a response was recorded. The 502 playbook has its own threshold rules; the additions here are the rollout correlation and the edge-versus-app comparison that tell you which branch to take.
# rules/symptom-502.yaml
groups:
- name: symptom-502
rules:
- record: ingress:http_502:ratio_rate5m
expr: |
sum by (exported_namespace, exported_service) (
rate(nginx_ingress_controller_requests{status="502"}[5m]))
/ sum by (exported_namespace, exported_service) (
rate(nginx_ingress_controller_requests[5m]))
# Page only when 502s are a meaningful share of traffic for 10+ minutes
- alert: Gateway502Burn
expr: ingress:http_502:ratio_rate5m > 0.005
for: 10m
labels:
severity: page
symptom: intermittent-502
annotations:
summary: "{{ $labels.exported_service }} returning {{ $value | humanizePercentage }} 502s at ingress"
dashboard_url: "https://grafana.example.com/d/edge-502?var-service={{ $labels.exported_service }}"
runbook_url: "https://docs.example.com/debugging-playbooks/intermittent-502"
playbook_step: "Step 1: locate where the 502 is generated"
# Cause hint: 502s only during rollouts → graceful shutdown / deregistration
- alert: Gateway502DuringRollout
expr: |
(ingress:http_502:ratio_rate5m > 0)
and on (exported_namespace)
label_replace(
sum by (namespace) (changes(kube_deployment_status_observed_generation[10m])) > 0,
"exported_namespace", "$1", "namespace", "(.*)")
labels:
severity: ticket
symptom: intermittent-502
annotations:
summary: "502s at ingress coincide with a rollout in {{ $labels.exported_namespace }}"
runbook_url: "https://docs.example.com/debugging-playbooks/intermittent-502"
playbook_step: "Step 5: check graceful shutdown (preStop, terminationGracePeriod)"
The comparison panel that decides the branch:
| Edge 502 rate | App 5xx rate | Pod restarts / rollout | Likely branch |
|---|---|---|---|
| Rising | Flat | Rollout in progress | Graceful shutdown (Step 5) |
| Rising | Flat | None | Keep-alive mismatch (Step 4) or DNS (Step 6) |
| Rising | Rising | Restarts rising | Pods dying under load — go to Pods Restarting |
| Rising | Flat | HPA scaling up | Resource exhaustion / slow scale-up (Step 8) |
Put edge 502 rate, service:http_errors:ratio_rate5m, and sum(increase(kube_pod_container_status_restarts_total[5m])) on one time-series panel with three axes, plus rollout annotations. In ingress-nginx access logs, upstream_status of - or a connection-reset error with a 502 response confirms the request never got an app-level response.
Symptom 4: Database CPU at 100%
What users feel: everything that touches the database slows down at once; often surfaces first as Symptom 1.
Detection strategy: DB CPU on its own is a cause, not a symptom — a batch job can legitimately pin CPU at 3 AM. Page when CPU is saturated and active sessions are queuing (work is waiting), and use cause alerts to point at the branch.
# rules/symptom-db-cpu.yaml
# Metric names follow postgres_exporter; for RDS/Aurora, CPU comes from the
# CloudWatch exporter (e.g. aws_rds_cpuutilization_average) — swap accordingly.
groups:
- name: symptom-db-cpu
rules:
# Normalize CPU to a 0-1 ratio. Self-hosted with node_exporter:
# 1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))
- record: db:cpu_utilization:ratio
expr: avg by (instance) (aws_rds_cpuutilization_average) / 100
# vCPU count per instance. Self-hosted: count by (instance) (node_cpu_seconds_total{mode="idle"})
# Managed: a static series per instance class, e.g. via a small exporter or file_sd labels.
- record: db:vcpus
expr: max by (instance) (db_instance_vcpus)
- alert: DBCPUSaturated
expr: |
(db:cpu_utilization:ratio > 0.9)
and on (instance)
(sum by (instance) (pg_stat_activity_count{state="active"}) > on (instance) 2 * db:vcpus)
for: 5m
labels:
severity: page
symptom: database-cpu
annotations:
summary: "{{ $labels.instance }} CPU >90% with active sessions >2x vCPUs"
dashboard_url: "https://grafana.example.com/d/postgres?var-instance={{ $labels.instance }}"
runbook_url: "https://docs.example.com/debugging-playbooks/database-cpu"
playbook_step: "Step 2: find the CPU-consuming queries in pg_stat_statements"
- alert: DBConnectionStorm
expr: |
deriv(sum by (instance) (pg_stat_activity_count)[10m:1m]) > 5
for: 5m
labels:
severity: ticket
symptom: database-cpu
annotations:
summary: "{{ $labels.instance }} connections climbing >5/s"
runbook_url: "https://docs.example.com/debugging-playbooks/database-cpu"
playbook_step: "Step 7: check for connection storms"
- alert: DBSeqScansSpiking
expr: |
sum by (instance, relname) (rate(pg_stat_user_tables_seq_tup_read[10m]))
> 10 * sum by (instance, relname) (rate(pg_stat_user_tables_seq_tup_read[1d] offset 1d))
for: 15m
labels:
severity: ticket
symptom: database-cpu
annotations:
summary: "Sequential scan volume on {{ $labels.relname }} 10x yesterday's rate"
runbook_url: "https://docs.example.com/debugging-playbooks/database-cpu"
playbook_step: "Step 3: check for sequential scans (missing index)"
Active sessions above twice the vCPU count means work is queuing for CPU, which is what turns high utilization into user-visible latency. Relabel the CloudWatch and postgres_exporter targets so both expose the same instance label, or the on (instance) join matches nothing.
Dashboard panels: active sessions stacked by state (active, idle in transaction, idle), lock waits, the top-10 queries by total execution time from pg_stat_statements, and replication lag. The PostgreSQL dashboard has a production JSON you can import. A large idle in transaction band points to the lock-contention branch (Step 4); a growing active band with one dominant query points to Steps 2-3.
Long-running transactions and dead-tuple alerts already live in the playbook's Prevention section — reuse them rather than defining near-duplicates.
Symptom 5: Memory Keeps Growing
What users feel: nothing, until the pod is OOM-killed — then a burst of errors or 502s every few hours.
Detection strategy: do not alert on absolute memory. Alert on time until the limit using predict_linear, which fires early enough to capture a heap snapshot before the kill destroys the evidence. The memory-leak playbook uses deriv on RSS for its own rule; predict_linear against the container limit is the complementary page-worthy form.
# rules/symptom-memory.yaml
groups:
- name: symptom-memory
rules:
- record: pod:memory_working_set_bytes
expr: |
sum by (namespace, pod, container) (
container_memory_working_set_bytes{container!="", container!="POD"})
- record: pod:memory_limit_bytes
expr: |
sum by (namespace, pod, container) (
kube_pod_container_resource_limits{resource="memory"})
# Page: at the current 6h trend, the container hits its limit within 4h
- alert: MemoryExhaustionPredicted
expr: |
predict_linear(pod:memory_working_set_bytes[6h], 4 * 3600)
> pod:memory_limit_bytes
and
pod:memory_working_set_bytes / pod:memory_limit_bytes > 0.6
for: 30m
labels:
severity: page
symptom: memory-leak
annotations:
summary: "{{ $labels.pod }} will hit its memory limit within ~4h; capture a heap snapshot now"
dashboard_url: "https://grafana.example.com/d/svc-memory?var-pod={{ $labels.pod }}"
runbook_url: "https://docs.example.com/debugging-playbooks/memory-leak"
playbook_step: "Step 1: confirm the leak is real; Step 3: take heap snapshots before OOM"
# Ticket: the post-GC baseline keeps rising (sawtooth with a climbing floor)
- alert: HeapBaselineRising
expr: |
deriv(min_over_time(nodejs_heap_size_used_bytes[30m])[6h:30m]) > 0
and
min_over_time(nodejs_heap_size_used_bytes[30m])
> 1.5 * min_over_time(nodejs_heap_size_used_bytes[30m] offset 6h)
for: 1h
labels:
severity: ticket
symptom: memory-leak
annotations:
summary: "{{ $labels.pod }} heap floor up 50% in 6h"
runbook_url: "https://docs.example.com/debugging-playbooks/memory-leak"
playbook_step: "Step 2: heap vs non-heap"
Why min_over_time
Heap usage is a sawtooth: it climbs between garbage collections and drops after each one. The peaks tell you little. The floor — the minimum over a window longer than a GC cycle — is what a leak raises. Alerting on the floor eliminates nearly all false positives from normal allocation churn.
Dashboard panels: working set versus limit with the predict_linear projection drawn as a dashed line; heap used floor (min_over_time) versus heap total; GC duration and frequency; pod age (a leak shows as memory correlated with uptime across replicas). If RSS grows but heap does not, the playbook's non-heap branch (buffers, native modules) applies.
Symptom 6: Pods Keep Restarting
What users feel: depends on replica count. With enough replicas and good readiness probes, nothing. With few replicas, errors and 502s on every restart.
Detection strategy: restarts alone are a ticket; restarts that reduce available capacity are a page. The pods-restarting playbook ships per-pod alert rules; the version below aggregates to the workload level and routes by termination reason so the first message already names the branch.
# rules/symptom-pods.yaml
groups:
- name: symptom-pods
rules:
# Page: restarts are reducing available replicas
- alert: PodsRestartingUserImpact
expr: |
sum by (namespace, deployment) (kube_deployment_status_replicas_unavailable) > 0
and on (namespace)
sum by (namespace) (increase(kube_pod_container_status_restarts_total[15m])) > 3
for: 5m
labels:
severity: page
symptom: pods-restarting
annotations:
summary: "{{ $labels.deployment }} has unavailable replicas and repeated restarts"
dashboard_url: "https://grafana.example.com/d/k8s-workload?var-namespace={{ $labels.namespace }}&var-deployment={{ $labels.deployment }}"
runbook_url: "https://docs.example.com/debugging-playbooks/pods-restarting"
playbook_step: "Step 2: check the exit code"
# Cause hints, one per termination reason / waiting reason
- alert: LivenessProbeFailing
expr: |
sum by (namespace, pod) (
rate(prober_probe_total{probe_type="Liveness", result="failed"}[5m])) > 0
for: 10m
labels:
severity: ticket
symptom: pods-restarting
annotations:
summary: "Liveness probe failing on {{ $labels.pod }}"
runbook_url: "https://docs.example.com/debugging-playbooks/pods-restarting"
playbook_step: "Step 7: diagnose liveness probe failures"
- alert: CrashLoop
expr: |
max by (namespace, pod, container) (
kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}) == 1
for: 5m
labels:
severity: ticket
symptom: pods-restarting
annotations:
summary: "{{ $labels.pod }}/{{ $labels.container }} in CrashLoopBackOff"
runbook_url: "https://docs.example.com/debugging-playbooks/pods-restarting"
playbook_step: "Step 3: previous container logs; Step 5: CrashLoopBackOff"
Termination reason → branch:
last_terminated_reason / exit code | Meaning | Branch |
|---|---|---|
OOMKilled / 137 | Exceeded memory limit | Step 4, then Memory Leak if usage trends upward |
Error / 1 | Application crashed at startup or runtime | Steps 3 and 5 |
Error / 143 | SIGTERM not handled within grace period | Graceful shutdown fix |
Completed / 0 | Process exited cleanly but should not have | Step 5 (entrypoint/command) |
Waiting ImagePullBackOff | Registry or tag problem | Step 6 |
Dashboard panels: restarts by reason as a stacked bar (sum by (reason) (kube_pod_container_status_last_terminated_reason * on (namespace, pod, container) group_left increase(kube_pod_container_status_restarts_total[1h]))), liveness and readiness probe failure rates, memory working set versus limit (shared with Symptom 5), and node pressure conditions. Kubernetes events in Loki — {job="kubernetes-events"} |= "Back-off restarting failed container" — give a timeline. For cluster-level diagnosis, see Kubernetes Troubleshooting.
How Symptoms Cascade
Symptoms rarely arrive alone. A leaking service gets OOM-killed, which produces 502s during restarts, which raises the error rate. If all four alerts page, the responder gets four pages for one problem and may start in the wrong playbook.
flowchart LR
ML[Memory leak] --> OOM[Pods restarting<br/>OOMKilled]
DBC[DB CPU saturated] --> SLOW[API slow]
SLOW --> TO[Upstream timeouts]
OOM --> E502[Intermittent 502s]
TO --> ERR[Error rate spike]
E502 --> ERR
style ML fill:#fef3c7,stroke:#f59e0b
style DBC fill:#fef3c7,stroke:#f59e0b
style ERR fill:#fee2e2,stroke:#ef4444
Use Alertmanager inhibition so the upstream cause suppresses the downstream symptom for the same service, and group alerts by service so they land in one notification:
# alertmanager.yaml (excerpt)
route:
receiver: slack-triage
group_by: [namespace, service]
group_wait: 30s
group_interval: 5m
routes:
- matchers: [severity="page"]
receiver: pagerduty
continue: true
- matchers: [severity="ticket"]
receiver: slack-triage
inhibit_rules:
# Pods crash-looping explains 502s and errors for the same namespace
- source_matchers: [alertname="PodsRestartingUserImpact"]
target_matchers: [alertname=~"Gateway502Burn|AvailabilitySLOBurn"]
equal: [namespace]
# Saturated DB explains slow APIs; responders start at the DB playbook
- source_matchers: [alertname="DBCPUSaturated"]
target_matchers: [alertname="LatencySLOBurn"]
equal: [namespace]
Inhibit Carefully
Inhibition hides alerts. If the inhibition rule is wrong — the DB is fine but the API is slow for an unrelated reason — you have suppressed the page that would have told you. Only inhibit pairs where the causal link is near-certain, and always keep the inhibited alert visible in the triage channel. Routing and grouping are covered in depth in Escalation Policies.
The Triage Dashboard
Build one dashboard that mirrors the master table: one row per symptom, each row showing the primary signal, a status indicator for the paging alert, and a text panel linking the playbook. On-call engineers open it first when they do not yet know which symptom they are looking at.
{
"title": "Symptom Triage",
"uid": "symptom-triage",
"templating": {
"list": [
{ "name": "namespace", "type": "query", "query": "label_values(service:http_requests:rate5m, namespace)" },
{ "name": "service", "type": "query", "query": "label_values(service:http_requests:rate5m{namespace=\"$namespace\"}, service)" }
]
},
"panels": [
{ "type": "row", "title": "API slow → /debugging-playbooks/api-slow", "gridPos": { "x": 0, "y": 0, "w": 24, "h": 1 } },
{
"type": "timeseries", "title": "Share of requests > 500ms",
"gridPos": { "x": 0, "y": 1, "w": 16, "h": 6 },
"targets": [{ "expr": "service:http_slow:ratio_rate5m{namespace=\"$namespace\", service=\"$service\"}" }],
"fieldConfig": { "defaults": { "unit": "percentunit",
"thresholds": { "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 0.144 } ] } } },
"links": [{ "title": "API Is Slow playbook", "url": "https://docs.example.com/debugging-playbooks/api-slow" }]
},
{
"type": "stat", "title": "LatencySLOBurn",
"gridPos": { "x": 16, "y": 1, "w": 8, "h": 6 },
"targets": [{ "expr": "count(ALERTS{alertname=\"LatencySLOBurn\", alertstate=\"firing\", service=\"$service\"}) or vector(0)" }],
"fieldConfig": { "defaults": { "mappings": [
{ "type": "value", "options": { "0": { "text": "OK", "color": "green" } } } ],
"thresholds": { "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 1 } ] } } }
},
{ "type": "row", "title": "Errors → /debugging-playbooks/high-error-rate", "gridPos": { "x": 0, "y": 7, "w": 24, "h": 1 } },
{
"type": "timeseries", "title": "5xx ratio (app) vs 502 ratio (edge)",
"gridPos": { "x": 0, "y": 8, "w": 24, "h": 6 },
"targets": [
{ "expr": "service:http_errors:ratio_rate5m{namespace=\"$namespace\", service=\"$service\"}", "legendFormat": "app 5xx" },
{ "expr": "ingress:http_502:ratio_rate5m{exported_namespace=\"$namespace\", exported_service=\"$service\"}", "legendFormat": "edge 502" }
],
"fieldConfig": { "defaults": { "unit": "percentunit" } }
}
]
}
The excerpt shows two rows; the full dashboard repeats the pattern for DB CPU, memory, and pods. Provision it as code alongside the rules (see Provisioning Dashboards as Code) so a rule change and its panel change land in the same pull request.
Testing the Chain
Alert rules are code, and the detection chain can break silently: a renamed metric, a label that changed from status to code, a playbook that moved. Two checks catch most breakage.
Unit-Test Rules with promtool
# tests/symptom-memory_test.yaml
rule_files:
- ../rules/symptom-memory.yaml
evaluation_interval: 1m
tests:
- interval: 5m
input_series:
# 1 GiB limit; working set starts at 600 MiB and grows 5 MiB every 5 minutes (60 MiB/h).
# The 4h projection crosses the limit at ~3.1h; with for: 30m it fires by ~3.6h.
- series: 'container_memory_working_set_bytes{namespace="prod", pod="api-1", container="api"}'
values: '629145600+5242880x120'
- series: 'kube_pod_container_resource_limits{namespace="prod", pod="api-1", container="api", resource="memory"}'
values: '1073741824x120'
alert_rule_test:
- eval_time: 2h
alertname: MemoryExhaustionPredicted
exp_alerts: []
- eval_time: 5h
alertname: MemoryExhaustionPredicted
exp_alerts:
- exp_labels:
severity: page
symptom: memory-leak
namespace: prod
pod: api-1
container: api
exp_annotations:
summary: "api-1 will hit its memory limit within ~4h; capture a heap snapshot now"
dashboard_url: "https://grafana.example.com/d/svc-memory?var-pod=api-1"
runbook_url: "https://docs.example.com/debugging-playbooks/memory-leak"
playbook_step: "Step 1: confirm the leak is real; Step 3: take heap snapshots before OOM"
Run promtool test rules tests/*.yaml in CI. The Alert Testing Framework section covers testing burn-rate rules across windows.
Lint the Links
A small CI check that every paging rule has a runbook_url pointing at a playbook that actually exists:
#!/usr/bin/env python3
"""Fail CI if a paging alert lacks a dashboard or a valid playbook link."""
import pathlib
import sys
import yaml
PLAYBOOK_DIR = pathlib.Path("docs/debugging-playbooks")
PREFIX = "https://docs.example.com/debugging-playbooks/"
REQUIRED = ("summary", "dashboard_url", "runbook_url")
def check(rule_file: pathlib.Path) -> list[str]:
problems = []
for group in yaml.safe_load(rule_file.read_text()).get("groups", []):
for rule in group.get("rules", []):
if "alert" not in rule:
continue # recording rule
name = rule["alert"]
ann = rule.get("annotations", {})
severity = rule.get("labels", {}).get("severity")
if severity == "page":
missing = [k for k in REQUIRED if not ann.get(k)]
if missing:
problems.append(f"{rule_file}:{name} missing {', '.join(missing)}")
url = ann.get("runbook_url", "")
if url.startswith(PREFIX):
slug = url[len(PREFIX):].split("#")[0].strip("/")
if not (PLAYBOOK_DIR / f"{slug}.md").exists():
problems.append(f"{rule_file}:{name} runbook_url -> missing playbook {slug}")
return problems
if __name__ == "__main__":
files = [pathlib.Path(p) for p in sys.argv[1:]] or sorted(pathlib.Path("rules").glob("*.yaml"))
problems = [p for f in files for p in check(f)]
print("\n".join(problems) or f"OK: {len(files)} rule files")
sys.exit(1 if problems else 0)
Coverage Gaps to Watch For
| Gap | How it shows up | Fix |
|---|---|---|
| Symptom with no paging alert | Customers report the problem before the pager does | Add a row to the master table; every postmortem should ask "did an alert fire?" |
| Cause alert that pages | On-call woken for a busy-but-healthy DB at 3 AM | Downgrade to ticket; page only on the user-facing symptom |
Alert with no playbook_step | Responder opens the playbook at the top every time | Add the step; review annotations in the postmortem |
| Edge metrics missing | 502s invisible because the app never saw the request | Scrape the ingress controller or LB; see Symptom 3 |
| Low-traffic services | Burn-rate ratios swing wildly on a handful of requests | Keep the rate5m > 1 guard; use synthetic probes for low-traffic paths |
| Dashboards drift from rules | Panel shows a metric the alert no longer uses | Provision dashboards and rules from the same repository |
For broader pitfalls — dashboard rot, vanity metrics, cardinality explosions — see Monitoring Antipatterns.
War Story (illustrative)
A team had good alerts for every symptom in the master table, but their runbook_url annotations all pointed at a wiki page that had been reorganized a year earlier. Every page during a regional database failover led responders to a "page not found". The incident ran 40 minutes longer than it needed to, mostly spent finding the right playbook. The fix was the link-lint script above in CI, plus moving playbooks into the same repository as the rules so a rename broke the build instead of the incident.
Key Takeaways
- Treat detection as a chain — symptom, signal, alert, dashboard, playbook — and make every link explicit in configuration.
- Page on user-facing symptoms (SLO burn rates); route cause-level alerts to a ticket channel where they hint at the playbook branch.
- Put
dashboard_url,runbook_url, andplaybook_stepon every paging alert. - Measure 502s at the edge, memory as time-to-limit, and DB CPU together with session queuing.
- Use inhibition and grouping so one root cause produces one notification, but inhibit only near-certain causal pairs.
- Test rules with
promtooland lint playbook links in CI so the chain cannot rot silently.
Related Pages
- Monitoring Overview and Metrics Design — RED, USE, and SLO fundamentals
- Prometheus Deep Dive — PromQL, recording rules, Alertmanager
- Grafana Dashboards — panel design and production dashboard JSON
- Alerting Overview and Alert Design — burn-rate theory and alert quality
- Severity Levels and Runbook Templates — what
pageandticketmean, and how to write the linked runbooks - SLIs, SLOs, and SLAs and Error Budgets — choosing the targets these alerts burn against
- Distributed Tracing — trace propagation behind the latency and error investigations
- Debugging Playbooks — the decision trees every alert on this page links to