Kubernetes Resource Management
Every container in a cluster competes for the same finite pool of CPU and memory. Resource requests and limits are how you tell Kubernetes what a container needs and what it is allowed to take. Get them wrong and you pay for it in one of three ways: the scheduler packs pods onto nodes that cannot actually run them, the kernel throttles your latency-sensitive service into a p99 disaster, or the OOM killer terminates your process mid-request and you spend the afternoon in a CrashLoopBackOff investigation.
The basic definitions are covered in Pod Lifecycle — Resource Requests and Limits. This page goes deeper: what the kernel actually does with those numbers, why language runtimes fight container limits, how to diagnose an OOMKill properly, and how to enforce sane defaults across many teams sharing one cluster.
The Mental Model: Two Numbers, Two Consumers
Requests and limits are read by completely different components:
| Field | Consumed by | Effect |
|---|---|---|
requests.cpu | Scheduler + kubelet | Node placement; converted to cgroup cpu.weight (relative share under contention) |
requests.memory | Scheduler + kubelet | Node placement; influences eviction ranking and oom_score_adj |
limits.cpu | Kernel (CFS bandwidth control) | Hard cap: cpu.max quota per period; exceeding it means throttling |
limits.memory | Kernel (memory cgroup) | Hard cap: memory.max; exceeding it means OOM kill |
flowchart LR
SPEC["Pod spec<br/>requests + limits"] --> SCHED["kube-scheduler<br/>sums requests<br/>vs node allocatable"]
SCHED --> KUBELET["kubelet on node"]
KUBELET --> CRI["container runtime<br/>(containerd / CRI-O)"]
CRI --> CG["cgroup v2"]
CG --> W["cpu.weight<br/>(from requests.cpu)"]
CG --> Q["cpu.max<br/>(from limits.cpu)"]
CG --> M["memory.max<br/>(from limits.memory)"]
KUBELET --> EVICT["eviction manager<br/>(uses requests<br/>to rank pods)"]
The scheduler never looks at actual usage. It only sums requests. A node with 16 cores of allocatable CPU can host pods requesting 16 cores total, even if those pods are idle — and it will refuse a 17th core's worth of requests even if the node is at 5% utilization. This is why over-requesting wastes money and under-requesting causes noisy-neighbour incidents.
Node Allocatable
Pods do not get the whole node. The kubelet reserves capacity for the OS and itself:
Node capacity 16 CPU 64Gi
- kube-reserved -0.5 -1Gi (kubelet, container runtime)
- system-reserved -0.5 -1Gi (sshd, journald, systemd)
- eviction threshold 0 -500Mi (memory.available hard eviction)
─────────────────────────────────────
= Allocatable 15 CPU ~61.5Gi
# What the scheduler sees per node
kubectl describe node ip-10-0-12-34 | sed -n '/Allocatable/,/System Info/p'
# How much is already requested vs allocatable
kubectl describe node ip-10-0-12-34 | sed -n '/Allocated resources/,/Events/p'
TIP
Managed offerings (EKS, GKE, AKS) set kube-reserved automatically, scaled to instance size. On larger nodes the reservation can be several GiB. Always size against allocatable, not instance specs, when calculating how many replicas fit per node.
CPU: Compressible, Throttled
CPU is a compressible resource. When a container wants more CPU than it is allowed, the kernel makes it wait. Nothing crashes — things just get slow.
How Requests Become Shares
requests.cpu maps to cgroup cpu.weight. Weight only matters under contention: if two containers on a saturated node request 500m and 1000m, the second gets roughly twice the CPU time. On an idle node, a container with a 100m request can happily use 4 cores.
How Limits Become Quota (CFS Bandwidth Control)
limits.cpu is enforced by the Completely Fair Scheduler's bandwidth controller. Kubernetes configures a period (default 100ms) and a quota = limit × period:
limits.cpu: 500m → cpu.max = "50000 100000"
(50ms of CPU time every 100ms period)
limits.cpu: 2 → cpu.max = "200000 100000"
(200ms of CPU time every 100ms period, across all cores)
The quota is consumed across all threads on all cores. This is the source of the most surprising throttling behaviour:
Container with limits.cpu: 1 (100ms quota per 100ms period)
Running a multi-threaded runtime with 8 busy threads on an 8-core node
t=0ms 8 threads start, each burns 1 core
t=12.5ms quota exhausted (8 threads × 12.5ms = 100ms)
t=12.5ms → t=100ms ALL threads throttled — container frozen for 87.5ms
t=100ms new period, quota refilled
Average CPU usage looks like a modest "1 core", yet every request that lands in that window waits up to 87.5ms. Dashboards show 60% of limit utilization while p99 latency explodes. This is the classic "we're nowhere near our CPU limit but we're being throttled" incident.
Detecting Throttling
# cgroup v2: inside the container or on the node
cat /sys/fs/cgroup/cpu.stat
# usage_usec 9283741234
# nr_periods 412233
# nr_throttled 88120 ← periods where quota ran out
# throttled_usec 3121987332 ← total time spent frozen
The Prometheus metrics come from cAdvisor:
# Fraction of CFS periods in which the container was throttled
sum by (namespace, pod, container) (
rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m])
)
/
sum by (namespace, pod, container) (
rate(container_cpu_cfs_periods_total{container!=""}[5m])
)
A throttle ratio above ~5% on a latency-sensitive service is worth investigating; above 25% it is almost certainly hurting tail latency.
Should You Set CPU Limits at All?
This is one of the longest-running debates in Kubernetes operations.
| Approach | Pros | Cons | Use when |
|---|---|---|---|
| CPU limit = request (Guaranteed) | Predictable performance; enables CPU Manager static policy (pinned cores) | No burst headroom; idle capacity is wasted | Latency-critical, CPU-pinned workloads; strict multi-tenant chargeback |
| CPU limit > request | Some burst, bounded blast radius | Still throttles multi-threaded bursts | Shared clusters with untrusted tenants |
| No CPU limit, request only | Pods burst into idle CPU; no CFS throttling | A runaway pod can starve neighbours above their request (weights still protect requests) | Most trusted internal services with accurate requests |
The "no CPU limits" approach works because cpu.weight still guarantees each container its requested share under contention. What you lose is the hard ceiling. If your requests are honest, that is usually a good trade.
WARNING
Removing CPU limits does not remove the need for accurate CPU requests. With no limits, requests are the only thing protecting a service from its neighbours. A pod requesting 50m that actually needs 2 cores will be starved the moment the node gets busy.
If you keep limits, align runtime thread pools with them (see Runtime Memory and CPU Inside Containers) so a 1-core container is not running 32 GC threads that burn quota in a few milliseconds.
Memory: Incompressible, Killed
Memory cannot be "slowed down". When a container's cgroup reaches memory.max, the kernel first tries to reclaim page cache belonging to that cgroup. If it cannot free enough, the cgroup OOM killer picks a process inside the cgroup and sends SIGKILL. There is no graceful shutdown, no preStop hook, no flushed logs.
What Counts Toward the Limit
The memory cgroup charges more than your heap:
| Charged to cgroup | Notes |
|---|---|
| Anonymous memory (heap, stacks, malloc arenas) | The part your runtime reports |
| Page cache from files the container reads/writes | Reclaimable, but counts toward memory.current |
tmpfs / emptyDir: { medium: Memory } | Counts as memory, not disk |
| Kernel memory (socket buffers, dentries) | Charged under cgroup v2 |
| Memory-mapped files | Shared mappings count for whoever faulted them in first |
This is why container_memory_usage_bytes is misleading — it includes reclaimable cache. The metric the kubelet uses for eviction, and the one you should alert on, is working set:
# Working set = usage - inactive file cache (what actually threatens OOM)
container_memory_working_set_bytes{container!="", container!="POD"}
# As a ratio of the limit
max by (namespace, pod, container) (container_memory_working_set_bytes{container!=""})
/
max by (namespace, pod, container) (kube_pod_container_resource_limits{resource="memory"})
Two Different Kills
It is important to distinguish cgroup OOM kills from node-pressure evictions:
flowchart TD
A["Memory grows"] --> B{"Container hits<br/>its own limits.memory?"}
B -- Yes --> C["Kernel cgroup OOM killer<br/>SIGKILL inside container"]
C --> D["status: OOMKilled<br/>exit code 137<br/>container restarts in place"]
B -- No --> E{"Node memory.available<br/>below eviction threshold?"}
E -- Yes --> F["kubelet eviction manager<br/>ranks pods, evicts whole pod"]
F --> G["pod status: Failed<br/>reason: Evicted<br/>rescheduled elsewhere by controller"]
E -- "No, but kernel<br/>itself is out of memory" --> H["Global kernel OOM killer<br/>uses oom_score_adj"]
H --> D
| cgroup OOM kill | kubelet eviction | |
|---|---|---|
| Trigger | Container exceeds its own limits.memory | Node memory.available < threshold (default 100Mi hard) |
| Actor | Linux kernel | kubelet |
| Scope | One process in one container | Entire pod |
| Status | OOMKilled, exit 137, restartCount++ | Evicted, pod object left in Failed |
| Graceful? | No | Soft eviction honours grace period; hard eviction does not |
| Fix | Raise limit or fix the leak | Fix requests (overcommit), add node capacity |
QoS Classes and Eviction Order
Kubernetes derives a Quality of Service class from requests and limits. You do not set it directly.
| QoS class | Rule | oom_score_adj | Eviction priority |
|---|---|---|---|
| Guaranteed | Every container has CPU and memory requests == limits | -997 | Last |
| Burstable | At least one request or limit set, not Guaranteed | 2 to 999, scaled by request size | Middle |
| BestEffort | No requests or limits on any container | 1000 | First |
kubectl get pods -n payments \
-o custom-columns='NAME:.metadata.name,QOS:.status.qosClass'
For Burstable pods, the kernel's oom_score_adj is roughly 1000 - (1000 × memoryRequest / nodeCapacity): a pod requesting a large slice of the node's memory is less likely to be chosen by the global OOM killer than one requesting very little.
How the kubelet Ranks Pods for Eviction
When a node crosses an eviction threshold, the kubelet does not simply evict by QoS class. It ranks by:
- Whether usage exceeds requests — pods using more than they requested go first.
- Pod priority — lower
PriorityClassfirst (see Pod Priority and Preemption). - Usage above request — the pod furthest over its request goes first.
The practical consequence: a Guaranteed pod can never be "over its request" (request == limit), so it is effectively last. A Burstable pod that stays under its memory request is nearly as safe. A BestEffort pod has a request of zero, so any usage at all puts it at the front of the line.
# Kubelet eviction configuration (KubeletConfiguration)
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
evictionHard:
memory.available: "500Mi"
nodefs.available: "10%"
imagefs.available: "15%"
evictionSoft:
memory.available: "1Gi"
evictionSoftGracePeriod:
memory.available: "1m30s"
evictionMaxPodGracePeriod: 60
kubeReserved:
cpu: "500m"
memory: "1Gi"
systemReserved:
cpu: "500m"
memory: "1Gi"
TIP
Set a soft eviction threshold above the hard one. Soft eviction respects (a capped) termination grace period, so evicted pods can drain connections. Hard eviction is an instant kill — it should be the safety net, not the normal path.
Diagnosing OOMKilled
When a pod restarts with OOMKilled, resist the urge to double the limit and move on. Work out which kind of memory grew.
Step 1: Confirm It Was a cgroup OOM
kubectl get pod checkout-7d9f8b6c5-x2k4p -n shop \
-o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"\t"}{.restartCount}{"\n"}{end}'
# app OOMKilled 137 14
# envoy 0
kubectl describe pod checkout-7d9f8b6c5-x2k4p -n shop | grep -A6 "Last State"
Exit code 137 = 128 + 9 (SIGKILL). If the reason is OOMKilled, the kernel did it. If exit 137 appears without OOMKilled, something else sent SIGKILL — most often the kubelet after a failed liveness probe exceeded terminationGracePeriodSeconds.
# Node-level confirmation (on the node or via a debug pod)
kubectl debug node/ip-10-0-12-34 -it --image=busybox -- \
chroot /host dmesg -T | grep -i -E "oom|killed process"
# [Sat Nov 8 09:14:22 2026] Memory cgroup out of memory: Killed process 48211 (node)
# total-vm:3481232kB, anon-rss:1012844kB, file-rss:31244kB
Step 2: Look at the Shape of the Memory Curve
container_memory_working_set_bytes{namespace="shop", pod=~"checkout-.*", container="app"}
| Shape | Likely cause | Next step |
|---|---|---|
| Sawtooth that slowly climbs between restarts | Genuine leak | Heap snapshots; see the Memory Leak playbook |
| Flat, then a vertical spike right before the kill | One huge request (big upload, unbounded query, giant JSON) | Find the request in logs/traces at that timestamp; add size limits |
| Flat line sitting just under the limit | Limit too low for steady state, or runtime sized its heap to the node | Check runtime flags (next section) |
| Immediate kill at startup | Limit smaller than runtime baseline (JIT, class metadata, preallocated pools) | Raise limit; measure startup peak |
| Grows with traffic, drops when traffic drops | Not a leak — concurrency-driven; buffers per in-flight request | Bound concurrency, or scale horizontally (HPA) |
Step 3: Compare Runtime View vs cgroup View
The runtime's own heap metric and the cgroup's working set routinely disagree. The gap is native memory: thread stacks, JIT code, direct buffers, native modules, malloc fragmentation, page cache.
# What the cgroup thinks
kubectl exec -n shop checkout-7d9f8b6c5-x2k4p -c app -- cat /sys/fs/cgroup/memory.current
kubectl exec -n shop checkout-7d9f8b6c5-x2k4p -c app -- cat /sys/fs/cgroup/memory.stat | head -12
# What the process thinks (RSS)
kubectl exec -n shop checkout-7d9f8b6c5-x2k4p -c app -- \
sh -c 'grep -E "VmRSS|VmHWM" /proc/1/status'
If the heap is stable but RSS keeps climbing, you are looking at a native leak or allocator fragmentation, not a garbage-collected-object leak. For glibc-based images with many threads, MALLOC_ARENA_MAX=2 is a frequently effective mitigation.
War Story (illustrative)
A team's Java service was OOMKilled roughly every six hours. Heap dashboards showed a healthy 600MB of a 1GB -Xmx, inside a 1.5Gi container limit, so the team concluded "it can't be memory." memory.stat told a different story: anon memory was 1.45Gi. The culprit was Netty's direct buffer pool plus 400 platform threads with 1MB stacks each — none of it visible in heap metrics. Capping -XX:MaxDirectMemorySize, moving to a bounded thread pool, and alerting on working set instead of heap ended the restarts without raising the limit.
Runtime Memory and CPU Inside Containers
Language runtimes size their heaps and thread pools based on what they think the machine looks like. Old runtimes read /proc/meminfo and see the node's 64GB. Modern ones are container-aware, but the defaults are still rarely what you want.
The Budget Formula
limits.memory ≥ max heap
+ non-heap runtime overhead (JIT, metadata, GC structures)
+ thread stacks (threads × stack size)
+ off-heap buffers (direct/native)
+ headroom for page cache and spikes (10-25%)
JVM
Java 10+ (and 8u191+) detects cgroup limits. The default max heap is only 25% of the container limit, which wastes memory in single-process containers.
containers:
- name: api
image: registry.example.com/orders-api:4.2.1
resources:
requests:
cpu: "1"
memory: 2Gi
limits:
memory: 2Gi
env:
- name: JAVA_TOOL_OPTIONS
value: >-
-XX:MaxRAMPercentage=70.0
-XX:InitialRAMPercentage=70.0
-XX:MaxMetaspaceSize=256m
-XX:MaxDirectMemorySize=192m
-XX:ActiveProcessorCount=2
-XX:+ExitOnOutOfMemoryError
-Xss512k
MaxRAMPercentage=70leaves 30% for metaspace, code cache, threads, and direct buffers. Go to 75-80% only after measuring non-heap usage with Native Memory Tracking (-XX:NativeMemoryTracking=summary).ActiveProcessorCountmatters when there is no CPU limit — otherwise the JVM sees every node core and sizes GC and ForkJoin pools accordingly.ExitOnOutOfMemoryErrorturns a zombie JVM (heap exhausted but process alive) into a clean restart.
Node.js
V8 sets its old-space limit from available memory, and recent Node versions (20+) read the cgroup limit — but the heuristic still leaves either too much or too little room depending on version. Set it explicitly:
containers:
- name: web
image: registry.example.com/storefront:9.14.0
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
memory: 1Gi
env:
# ~75% of the limit for old space; the rest covers new space,
# Buffers (allocated outside the V8 heap), code, and native addons
- name: NODE_OPTIONS
value: "--max-old-space-size=768"
- name: UV_THREADPOOL_SIZE
value: "4"
Buffer and ArrayBuffer memory lives outside the V8 heap, so a service that streams files or handles large payloads needs more headroom than one that only juggles small JSON objects. See Node.js Performance for the full V8 heap model and how to read process.memoryUsage().
Go
Go has no fixed heap ceiling by default; the GC triggers at GOGC (100% growth over live heap). Without a limit, a burst of garbage can push RSS past the container limit before the next cycle. Since Go 1.19, GOMEMLIMIT gives the GC a soft target:
containers:
- name: ingest
image: registry.example.com/ingest-worker:2.8.0
resources:
requests:
cpu: "2"
memory: 512Mi
limits:
memory: 512Mi
env:
- name: GOMEMLIMIT
valueFrom:
resourceFieldRef:
resource: limits.memory
divisor: "1"
# GOMEMLIMIT above equals the hard limit; prefer ~90% via the app or
# automemlimit library so the GC reacts before the kernel does.
- name: GOMAXPROCS
valueFrom:
resourceFieldRef:
resource: requests.cpu
divisor: "1"
In code, the common approach is to set the limit to ~90% of the cgroup value at startup:
package main
import (
"log/slog"
"os"
"runtime/debug"
"strconv"
"strings"
)
// setMemoryLimit reads the cgroup v2 memory limit and sets the Go soft
// memory limit to 90% of it, leaving headroom for non-heap memory.
func setMemoryLimit() {
raw, err := os.ReadFile("/sys/fs/cgroup/memory.max")
if err != nil {
slog.Warn("cgroup memory.max not readable; leaving GOMEMLIMIT unset", "err", err)
return
}
s := strings.TrimSpace(string(raw))
if s == "max" {
return // no limit configured
}
limit, err := strconv.ParseInt(s, 10, 64)
if err != nil {
slog.Warn("unparseable memory.max", "value", s)
return
}
soft := limit * 9 / 10
debug.SetMemoryLimit(soft)
slog.Info("memory limit configured", "cgroup_bytes", limit, "gomemlimit_bytes", soft)
}
func main() {
setMemoryLimit()
// ... start server
}
Go 1.25+ sets GOMAXPROCS from the cgroup CPU limit automatically; for older versions use go.uber.org/automaxprocs. If there is no CPU limit, set GOMAXPROCS from the request, as above, so the scheduler does not spin up one P per node core.
Runtime Cheat Sheet
| Runtime | Heap control | Thread/CPU control | Common trap |
|---|---|---|---|
| JVM | -XX:MaxRAMPercentage | -XX:ActiveProcessorCount | Direct buffers and thread stacks outside -Xmx |
| Node.js | --max-old-space-size (MiB) | UV_THREADPOOL_SIZE | Buffer memory outside V8 heap |
| Go | GOMEMLIMIT | GOMAXPROCS | No heap ceiling by default; GC lag on bursts |
| Python | None (refcount + GC) | Worker count in gunicorn/uvicorn | Each worker is a full copy; fork after import to share pages |
| .NET | DOTNET_GCHeapHardLimitPercent | DOTNET_PROCESSOR_COUNT | Server GC allocates per-core heaps |
LimitRange: Defaults and Guardrails per Namespace
A LimitRange applies defaults to containers that do not declare resources and rejects pods outside allowed bounds. It is enforced by an admission plugin at pod creation time — it never changes running pods.
apiVersion: v1
kind: LimitRange
metadata:
name: container-defaults
namespace: team-checkout
spec:
limits:
- type: Container
# Applied when a container omits requests
defaultRequest:
cpu: 100m
memory: 128Mi
# Applied when a container omits limits
default:
memory: 512Mi
# Hard bounds — pod creation fails outside these
min:
cpu: 10m
memory: 32Mi
max:
cpu: "4"
memory: 8Gi
# Limit may be at most 4x the request (curbs extreme overcommit)
maxLimitRequestRatio:
memory: "4"
- type: PersistentVolumeClaim
min:
storage: 1Gi
max:
storage: 200Gi
WARNING
If a LimitRange sets default.cpu, every container that omits a CPU limit gets one — including sidecars injected by service meshes. Teams that have deliberately chosen "no CPU limits" are often surprised to find them reintroduced by a namespace default. Only set default for the resources you actually want capped.
Debug a rejection:
kubectl apply -f deploy.yaml
# Error from server (Forbidden): pods "checkout-..." is forbidden:
# maximum memory usage per Container is 8Gi, but limit is 12Gi
kubectl describe limitrange -n team-checkout
Note that Deployment creation succeeds even when its pods would be rejected — the failure appears on the ReplicaSet. Check kubectl describe rs or kubectl get events when replicas stay at zero.
ResourceQuota: Budgets per Namespace
A ResourceQuota caps the aggregate resources a namespace can consume. It is the primary tool for multi-tenant fairness and cost control.
apiVersion: v1
kind: ResourceQuota
metadata:
name: compute-budget
namespace: team-checkout
spec:
hard:
requests.cpu: "40"
requests.memory: 80Gi
limits.memory: 120Gi
pods: "200"
services.loadbalancers: "2"
persistentvolumeclaims: "30"
requests.storage: 2Ti
count/jobs.batch: "50"
---
# Separate, tighter budget for low-priority work in the same namespace
apiVersion: v1
kind: ResourceQuota
metadata:
name: batch-budget
namespace: team-checkout
spec:
hard:
requests.cpu: "8"
requests.memory: 16Gi
scopeSelector:
matchExpressions:
- operator: In
scopeName: PriorityClass
values: ["batch-low"]
Key behaviours:
- Once a quota covers
requests.cpuorrequests.memory, every new pod in the namespace must declare those requests, or it is rejected. Pair every ResourceQuota with a LimitRange that supplies defaults. - Quota is checked at admission. Scaling a Deployment past the quota leaves the extra pods un-created, with
exceeded quotaevents on the ReplicaSet. - Quotas count requests, not usage. A namespace can hit its quota while its nodes sit idle — that is the signal to right-size, not to raise the quota.
kubectl describe resourcequota -n team-checkout
# Name: compute-budget
# Resource Used Hard
# -------- ---- ----
# limits.memory 96Gi 120Gi
# pods 143 200
# requests.cpu 38500m 40
# requests.memory 71Gi 80Gi
# Alert before a team hits its quota
kube_resourcequota{type="used"} / ignoring(type) kube_resourcequota{type="hard"} > 0.9
Right-Sizing
Most clusters are dramatically over-requested. Industry surveys routinely find average CPU utilization of requested capacity in the 10-30% range. Right-sizing is the highest-leverage cost lever you have (see FinOps Cost Optimization).
The Data You Need
# CPU: p95 usage over 7 days, per container
quantile_over_time(0.95,
rate(container_cpu_usage_seconds_total{namespace="shop", container="app"}[5m])[7d:5m]
)
# Memory: max working set over 7 days, per container
max_over_time(
container_memory_working_set_bytes{namespace="shop", container="app"}[7d]
)
# Request efficiency: usage as a fraction of request
sum by (namespace) (rate(container_cpu_usage_seconds_total{container!=""}[1h]))
/
sum by (namespace) (kube_pod_container_resource_requests{resource="cpu"})
Sizing Heuristics
| Resource | Request | Limit |
|---|---|---|
| CPU | p90-p95 of usage under normal peak | None, or 2-4x request if your policy requires one |
| Memory | p99 / max working set + 10-20% | Equal to request (or close) — memory is not shareable |
Setting memory request == limit is the safest default: the scheduler reserves exactly what the container can use, so overcommit cannot trigger node-pressure eviction, and the cgroup kill point is predictable.
Use at least 7 days of data (14 is better) so weekly cycles are included, and re-check after major releases. Startup peaks (JIT warm-up, cache priming) count — a container that needs 1.2Gi for 30 seconds at boot needs a 1.2Gi limit.
VPA as a Recommender
The Vertical Pod Autoscaler computes recommendations from historical usage. Running it in Off mode gives you the numbers without the risk of it restarting pods. Full VPA configuration, including Auto and InPlaceOrRecreate modes and its interaction with HPA, is covered in HPA, VPA & KEDA.
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: checkout-recommender
namespace: shop
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: checkout
updatePolicy:
updateMode: "Off"
resourcePolicy:
containerPolicies:
- containerName: app
controlledResources: ["cpu", "memory"]
minAllowed:
cpu: 50m
memory: 128Mi
maxAllowed:
cpu: "4"
memory: 4Gi
- containerName: istio-proxy
mode: "Off"
kubectl get vpa checkout-recommender -n shop \
-o jsonpath='{.status.recommendation.containerRecommendations[*]}' | jq
# {
# "containerName": "app",
# "lowerBound": { "cpu": "180m", "memory": "410Mi" },
# "target": { "cpu": "250m", "memory": "520Mi" },
# "upperBound": { "cpu": "900m", "memory": "1100Mi" },
# "uncappedTarget":{ "cpu": "250m", "memory": "520Mi" }
# }
Use target for requests. Treat upperBound as a sanity check on memory limits. If uncappedTarget differs from target, your maxAllowed is clipping the recommendation — the container genuinely wants more.
WARNING
Do not let VPA (in Auto mode) and HPA both act on the same CPU or memory metric. HPA scales out when utilization relative to request rises; VPA raises the request, which lowers utilization; HPA scales back in. The two controllers chase each other. Use HPA on a custom metric (RPS, queue depth) if you need both.
In-Place Resize
Kubernetes 1.33 promoted in-place pod resize to beta (enabled by default): CPU and memory on a running container can be changed through the resize subresource without recreating the pod. Each container's resizePolicy declares whether a change needs a restart:
containers:
- name: app
image: registry.example.com/checkout:7.1.3
resizePolicy:
- resourceName: cpu
restartPolicy: NotRequired
- resourceName: memory
restartPolicy: RestartContainer # most runtimes cannot shrink heaps live
resources:
requests: { cpu: 250m, memory: 512Mi }
limits: { memory: 512Mi }
kubectl patch pod checkout-7d9f8b6c5-x2k4p -n shop --subresource resize \
--patch '{"spec":{"containers":[{"name":"app","resources":{"requests":{"cpu":"500m"}}}]}}'
CPU resizes are effectively free. Memory resizes usually still need a container restart because the runtime sized its heap at boot.
Namespace Multi-Tenancy
Resource controls are one layer of a soft multi-tenancy model where teams share a cluster but cannot hurt each other.
flowchart TB
subgraph Cluster
subgraph NS1["namespace: team-checkout"]
Q1["ResourceQuota<br/>40 CPU / 80Gi"]
L1["LimitRange<br/>defaults + max"]
P1["Pods"]
end
subgraph NS2["namespace: team-search"]
Q2["ResourceQuota<br/>24 CPU / 96Gi"]
L2["LimitRange"]
P2["Pods"]
end
PC["PriorityClasses<br/>prod-critical / prod / batch-low"]
ADM["Admission policy<br/>(require requests,<br/>forbid BestEffort in prod)"]
end
RBAC["RBAC: teams cannot edit<br/>their own quota"] --> NS1
RBAC --> NS2
PC --> P1
PC --> P2
ADM --> P1
ADM --> P2
| Layer | Mechanism | Page |
|---|---|---|
| Compute budget | ResourceQuota per namespace | This page |
| Per-container defaults and bounds | LimitRange | This page |
| Who wins under pressure | PriorityClass + preemption | Pod Lifecycle |
| Who can change budgets | RBAC (platform team owns quota objects) | RBAC |
| Network isolation | NetworkPolicy default-deny | Network Policies |
| Policy enforcement | Validating admission (Kyverno, Gatekeeper, ValidatingAdmissionPolicy) | Admission Webhooks |
| Cost attribution | Requests × price, per namespace label | Cost Allocation |
Enforcing "Every Container Declares Resources"
LimitRange defaults are a safety net, not a strategy — a default of 128Mi is wrong for nearly every real service. Native ValidatingAdmissionPolicy (GA since 1.30) can require explicit values without a webhook:
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: require-memory-requests-and-limits
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: ["apps"]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["deployments", "statefulsets", "daemonsets"]
validations:
- expression: >-
object.spec.template.spec.containers.all(c,
has(c.resources) &&
has(c.resources.requests) && 'memory' in c.resources.requests &&
has(c.resources.limits) && 'memory' in c.resources.limits)
message: "Every container must declare memory requests and limits."
- expression: >-
object.spec.template.spec.containers.all(c,
has(c.resources.requests) && 'cpu' in c.resources.requests)
message: "Every container must declare a CPU request."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: require-memory-requests-and-limits
spec:
policyName: require-memory-requests-and-limits
validationActions: ["Deny"]
matchResources:
namespaceSelector:
matchLabels:
tenancy: team
Roll it out with validationActions: ["Audit"] first, review the audit annotations, then switch to Deny.
Overcommit Policy
Decide deliberately how much overcommit each tier tolerates:
| Tier | CPU | Memory | Rationale |
|---|---|---|---|
| Production, latency-critical | Request = p95, no limit | Request = limit | Predictable, never node-evicted for overcommit |
| Production, standard | Request = p90, no limit | Request = limit | Same, slightly denser |
| Batch / CI | Small request, no limit | Limit ≤ 2x request | Burst into idle capacity; acceptable to be evicted |
| Dev / preview | Small request | Limit ≤ 4x request, enforced by LimitRange | Density over stability |
Keep batch and production on separate node pools (taints and tolerations) if batch overcommit is aggressive — it keeps eviction storms away from customer-facing pods.
Production Checklist
| Item | Why |
|---|---|
| Every container has CPU and memory requests | Scheduler accuracy; HPA utilization math; quota admission |
| Memory limit set, usually equal to request | Predictable OOM point, no overcommit-driven eviction |
| CPU limits are a deliberate decision, not a default | Avoid CFS throttling on multi-threaded runtimes |
| Runtime heap flags derived from the container limit | Prevent heap sized to node memory |
Thread pools / GOMAXPROCS aligned with CPU allocation | Avoid quota burn and context switching |
| Alerts on working set / limit > 90% | Warning before OOMKill |
| Alerts on CFS throttle ratio > 25% for latency services | Detect throttling hidden by average CPU |
Alerts on OOMKilled restarts (kube_pod_container_status_last_terminated_reason) | Catch kills that recover silently |
VPA in Off mode on every Deployment | Continuous right-sizing data |
| LimitRange + ResourceQuota on every tenant namespace | Defaults, bounds, budgets |
| Admission policy forbids BestEffort in production | BestEffort pods are evicted first and destabilize nodes |
kube-reserved / system-reserved configured | Protect kubelet and OS from pod pressure |
The broader cluster readiness list lives in the Kubernetes Production Checklist.
# PrometheusRule: resource alerts
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: container-resources
namespace: monitoring
spec:
groups:
- name: container-resources
rules:
- alert: ContainerMemoryNearLimit
expr: |
max by (namespace, pod, container) (container_memory_working_set_bytes{container!=""})
/
max by (namespace, pod, container) (kube_pod_container_resource_limits{resource="memory"})
> 0.9
for: 10m
labels:
severity: warning
annotations:
summary: "Container memory working set above 90% of limit"
- alert: ContainerOOMKilled
expr: |
increase(kube_pod_container_status_restarts_total[15m]) > 0
and on (namespace, pod, container)
kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
labels:
severity: warning
annotations:
summary: "Container restarted after OOMKill"
- alert: ContainerCPUThrottlingHigh
expr: |
sum by (namespace, pod, container) (rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m]))
/
sum by (namespace, pod, container) (rate(container_cpu_cfs_periods_total{container!=""}[5m]))
> 0.25
for: 15m
labels:
severity: info
annotations:
summary: "Container throttled in more than 25% of CFS periods"
Cross-References
- Pod Lifecycle — requests/limits basics, QoS, priority and preemption
- HPA, VPA & KEDA — autoscaling on top of accurate requests
- Kubernetes Troubleshooting — broader pod failure diagnosis
- Pods Keep Restarting — OOMKill and CrashLoopBackOff playbook
- Memory Leak — finding the leak once you know it is one
- Linux Memory Management — page cache, RSS, the OOM killer
- Node.js Performance — V8 heap sizing for Node in containers
- Capacity Planning — turning requests into node counts