On this page

Kubernetes Resource Management

Every container in a cluster competes for the same finite pool of CPU and memory. Resource requests and limits are how you tell Kubernetes what a container needs and what it is allowed to take. Get them wrong and you pay for it in one of three ways: the scheduler packs pods onto nodes that cannot actually run them, the kernel throttles your latency-sensitive service into a p99 disaster, or the OOM killer terminates your process mid-request and you spend the afternoon in a CrashLoopBackOff investigation.

The basic definitions are covered in Pod Lifecycle — Resource Requests and Limits. This page goes deeper: what the kernel actually does with those numbers, why language runtimes fight container limits, how to diagnose an OOMKill properly, and how to enforce sane defaults across many teams sharing one cluster.

The Mental Model: Two Numbers, Two Consumers

Requests and limits are read by completely different components:

FieldConsumed byEffect
requests.cpuScheduler + kubeletNode placement; converted to cgroup cpu.weight (relative share under contention)
requests.memoryScheduler + kubeletNode placement; influences eviction ranking and oom_score_adj
limits.cpuKernel (CFS bandwidth control)Hard cap: cpu.max quota per period; exceeding it means throttling
limits.memoryKernel (memory cgroup)Hard cap: memory.max; exceeding it means OOM kill
flowchart LR
    SPEC["Pod spec<br/>requests + limits"] --> SCHED["kube-scheduler<br/>sums requests<br/>vs node allocatable"]
    SCHED --> KUBELET["kubelet on node"]
    KUBELET --> CRI["container runtime<br/>(containerd / CRI-O)"]
    CRI --> CG["cgroup v2"]
    CG --> W["cpu.weight<br/>(from requests.cpu)"]
    CG --> Q["cpu.max<br/>(from limits.cpu)"]
    CG --> M["memory.max<br/>(from limits.memory)"]
    KUBELET --> EVICT["eviction manager<br/>(uses requests<br/>to rank pods)"]

The scheduler never looks at actual usage. It only sums requests. A node with 16 cores of allocatable CPU can host pods requesting 16 cores total, even if those pods are idle — and it will refuse a 17th core's worth of requests even if the node is at 5% utilization. This is why over-requesting wastes money and under-requesting causes noisy-neighbour incidents.

Node Allocatable

Pods do not get the whole node. The kubelet reserves capacity for the OS and itself:

Node capacity          16 CPU   64Gi
 - kube-reserved       -0.5     -1Gi    (kubelet, container runtime)
 - system-reserved     -0.5     -1Gi    (sshd, journald, systemd)
 - eviction threshold   0       -500Mi  (memory.available hard eviction)
 ─────────────────────────────────────
 = Allocatable         15 CPU   ~61.5Gi
# What the scheduler sees per node
kubectl describe node ip-10-0-12-34 | sed -n '/Allocatable/,/System Info/p'

# How much is already requested vs allocatable
kubectl describe node ip-10-0-12-34 | sed -n '/Allocated resources/,/Events/p'

TIP

Managed offerings (EKS, GKE, AKS) set kube-reserved automatically, scaled to instance size. On larger nodes the reservation can be several GiB. Always size against allocatable, not instance specs, when calculating how many replicas fit per node.

CPU: Compressible, Throttled

CPU is a compressible resource. When a container wants more CPU than it is allowed, the kernel makes it wait. Nothing crashes — things just get slow.

How Requests Become Shares

requests.cpu maps to cgroup cpu.weight. Weight only matters under contention: if two containers on a saturated node request 500m and 1000m, the second gets roughly twice the CPU time. On an idle node, a container with a 100m request can happily use 4 cores.

How Limits Become Quota (CFS Bandwidth Control)

limits.cpu is enforced by the Completely Fair Scheduler's bandwidth controller. Kubernetes configures a period (default 100ms) and a quota = limit × period:

limits.cpu: 500m   →  cpu.max = "50000 100000"
                      (50ms of CPU time every 100ms period)

limits.cpu: 2      →  cpu.max = "200000 100000"
                      (200ms of CPU time every 100ms period, across all cores)

The quota is consumed across all threads on all cores. This is the source of the most surprising throttling behaviour:

Container with limits.cpu: 1 (100ms quota per 100ms period)
Running a multi-threaded runtime with 8 busy threads on an 8-core node

t=0ms    8 threads start, each burns 1 core
t=12.5ms quota exhausted (8 threads × 12.5ms = 100ms)
t=12.5ms → t=100ms   ALL threads throttled — container frozen for 87.5ms
t=100ms  new period, quota refilled

Average CPU usage looks like a modest "1 core", yet every request that lands in that window waits up to 87.5ms. Dashboards show 60% of limit utilization while p99 latency explodes. This is the classic "we're nowhere near our CPU limit but we're being throttled" incident.

Detecting Throttling

# cgroup v2: inside the container or on the node
cat /sys/fs/cgroup/cpu.stat
# usage_usec 9283741234
# nr_periods 412233
# nr_throttled 88120          ← periods where quota ran out
# throttled_usec 3121987332   ← total time spent frozen

The Prometheus metrics come from cAdvisor:

# Fraction of CFS periods in which the container was throttled
sum by (namespace, pod, container) (
  rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m])
)
/
sum by (namespace, pod, container) (
  rate(container_cpu_cfs_periods_total{container!=""}[5m])
)

A throttle ratio above ~5% on a latency-sensitive service is worth investigating; above 25% it is almost certainly hurting tail latency.

Should You Set CPU Limits at All?

This is one of the longest-running debates in Kubernetes operations.

ApproachProsConsUse when
CPU limit = request (Guaranteed)Predictable performance; enables CPU Manager static policy (pinned cores)No burst headroom; idle capacity is wastedLatency-critical, CPU-pinned workloads; strict multi-tenant chargeback
CPU limit > requestSome burst, bounded blast radiusStill throttles multi-threaded burstsShared clusters with untrusted tenants
No CPU limit, request onlyPods burst into idle CPU; no CFS throttlingA runaway pod can starve neighbours above their request (weights still protect requests)Most trusted internal services with accurate requests

The "no CPU limits" approach works because cpu.weight still guarantees each container its requested share under contention. What you lose is the hard ceiling. If your requests are honest, that is usually a good trade.

WARNING

Removing CPU limits does not remove the need for accurate CPU requests. With no limits, requests are the only thing protecting a service from its neighbours. A pod requesting 50m that actually needs 2 cores will be starved the moment the node gets busy.

If you keep limits, align runtime thread pools with them (see Runtime Memory and CPU Inside Containers) so a 1-core container is not running 32 GC threads that burn quota in a few milliseconds.

Memory: Incompressible, Killed

Memory cannot be "slowed down". When a container's cgroup reaches memory.max, the kernel first tries to reclaim page cache belonging to that cgroup. If it cannot free enough, the cgroup OOM killer picks a process inside the cgroup and sends SIGKILL. There is no graceful shutdown, no preStop hook, no flushed logs.

What Counts Toward the Limit

The memory cgroup charges more than your heap:

Charged to cgroupNotes
Anonymous memory (heap, stacks, malloc arenas)The part your runtime reports
Page cache from files the container reads/writesReclaimable, but counts toward memory.current
tmpfs / emptyDir: { medium: Memory }Counts as memory, not disk
Kernel memory (socket buffers, dentries)Charged under cgroup v2
Memory-mapped filesShared mappings count for whoever faulted them in first

This is why container_memory_usage_bytes is misleading — it includes reclaimable cache. The metric the kubelet uses for eviction, and the one you should alert on, is working set:

# Working set = usage - inactive file cache (what actually threatens OOM)
container_memory_working_set_bytes{container!="", container!="POD"}

# As a ratio of the limit
max by (namespace, pod, container) (container_memory_working_set_bytes{container!=""})
/
max by (namespace, pod, container) (kube_pod_container_resource_limits{resource="memory"})

Two Different Kills

It is important to distinguish cgroup OOM kills from node-pressure evictions:

flowchart TD
    A["Memory grows"] --> B{"Container hits<br/>its own limits.memory?"}
    B -- Yes --> C["Kernel cgroup OOM killer<br/>SIGKILL inside container"]
    C --> D["status: OOMKilled<br/>exit code 137<br/>container restarts in place"]
    B -- No --> E{"Node memory.available<br/>below eviction threshold?"}
    E -- Yes --> F["kubelet eviction manager<br/>ranks pods, evicts whole pod"]
    F --> G["pod status: Failed<br/>reason: Evicted<br/>rescheduled elsewhere by controller"]
    E -- "No, but kernel<br/>itself is out of memory" --> H["Global kernel OOM killer<br/>uses oom_score_adj"]
    H --> D
cgroup OOM killkubelet eviction
TriggerContainer exceeds its own limits.memoryNode memory.available < threshold (default 100Mi hard)
ActorLinux kernelkubelet
ScopeOne process in one containerEntire pod
StatusOOMKilled, exit 137, restartCount++Evicted, pod object left in Failed
Graceful?NoSoft eviction honours grace period; hard eviction does not
FixRaise limit or fix the leakFix requests (overcommit), add node capacity

QoS Classes and Eviction Order

Kubernetes derives a Quality of Service class from requests and limits. You do not set it directly.

QoS classRuleoom_score_adjEviction priority
GuaranteedEvery container has CPU and memory requests == limits-997Last
BurstableAt least one request or limit set, not Guaranteed2 to 999, scaled by request sizeMiddle
BestEffortNo requests or limits on any container1000First
kubectl get pods -n payments \
  -o custom-columns='NAME:.metadata.name,QOS:.status.qosClass'

For Burstable pods, the kernel's oom_score_adj is roughly 1000 - (1000 × memoryRequest / nodeCapacity): a pod requesting a large slice of the node's memory is less likely to be chosen by the global OOM killer than one requesting very little.

How the kubelet Ranks Pods for Eviction

When a node crosses an eviction threshold, the kubelet does not simply evict by QoS class. It ranks by:

  1. Whether usage exceeds requests — pods using more than they requested go first.
  2. Pod priority — lower PriorityClass first (see Pod Priority and Preemption).
  3. Usage above request — the pod furthest over its request goes first.

The practical consequence: a Guaranteed pod can never be "over its request" (request == limit), so it is effectively last. A Burstable pod that stays under its memory request is nearly as safe. A BestEffort pod has a request of zero, so any usage at all puts it at the front of the line.

# Kubelet eviction configuration (KubeletConfiguration)
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
evictionHard:
  memory.available: "500Mi"
  nodefs.available: "10%"
  imagefs.available: "15%"
evictionSoft:
  memory.available: "1Gi"
evictionSoftGracePeriod:
  memory.available: "1m30s"
evictionMaxPodGracePeriod: 60
kubeReserved:
  cpu: "500m"
  memory: "1Gi"
systemReserved:
  cpu: "500m"
  memory: "1Gi"

TIP

Set a soft eviction threshold above the hard one. Soft eviction respects (a capped) termination grace period, so evicted pods can drain connections. Hard eviction is an instant kill — it should be the safety net, not the normal path.

Diagnosing OOMKilled

When a pod restarts with OOMKilled, resist the urge to double the limit and move on. Work out which kind of memory grew.

Step 1: Confirm It Was a cgroup OOM

kubectl get pod checkout-7d9f8b6c5-x2k4p -n shop \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"\t"}{.restartCount}{"\n"}{end}'
# app     OOMKilled   137   14
# envoy               0

kubectl describe pod checkout-7d9f8b6c5-x2k4p -n shop | grep -A6 "Last State"

Exit code 137 = 128 + 9 (SIGKILL). If the reason is OOMKilled, the kernel did it. If exit 137 appears without OOMKilled, something else sent SIGKILL — most often the kubelet after a failed liveness probe exceeded terminationGracePeriodSeconds.

# Node-level confirmation (on the node or via a debug pod)
kubectl debug node/ip-10-0-12-34 -it --image=busybox -- \
  chroot /host dmesg -T | grep -i -E "oom|killed process"
# [Sat Nov  8 09:14:22 2026] Memory cgroup out of memory: Killed process 48211 (node)
#   total-vm:3481232kB, anon-rss:1012844kB, file-rss:31244kB

Step 2: Look at the Shape of the Memory Curve

container_memory_working_set_bytes{namespace="shop", pod=~"checkout-.*", container="app"}
ShapeLikely causeNext step
Sawtooth that slowly climbs between restartsGenuine leakHeap snapshots; see the Memory Leak playbook
Flat, then a vertical spike right before the killOne huge request (big upload, unbounded query, giant JSON)Find the request in logs/traces at that timestamp; add size limits
Flat line sitting just under the limitLimit too low for steady state, or runtime sized its heap to the nodeCheck runtime flags (next section)
Immediate kill at startupLimit smaller than runtime baseline (JIT, class metadata, preallocated pools)Raise limit; measure startup peak
Grows with traffic, drops when traffic dropsNot a leak — concurrency-driven; buffers per in-flight requestBound concurrency, or scale horizontally (HPA)

Step 3: Compare Runtime View vs cgroup View

The runtime's own heap metric and the cgroup's working set routinely disagree. The gap is native memory: thread stacks, JIT code, direct buffers, native modules, malloc fragmentation, page cache.

# What the cgroup thinks
kubectl exec -n shop checkout-7d9f8b6c5-x2k4p -c app -- cat /sys/fs/cgroup/memory.current
kubectl exec -n shop checkout-7d9f8b6c5-x2k4p -c app -- cat /sys/fs/cgroup/memory.stat | head -12

# What the process thinks (RSS)
kubectl exec -n shop checkout-7d9f8b6c5-x2k4p -c app -- \
  sh -c 'grep -E "VmRSS|VmHWM" /proc/1/status'

If the heap is stable but RSS keeps climbing, you are looking at a native leak or allocator fragmentation, not a garbage-collected-object leak. For glibc-based images with many threads, MALLOC_ARENA_MAX=2 is a frequently effective mitigation.

War Story (illustrative)

A team's Java service was OOMKilled roughly every six hours. Heap dashboards showed a healthy 600MB of a 1GB -Xmx, inside a 1.5Gi container limit, so the team concluded "it can't be memory." memory.stat told a different story: anon memory was 1.45Gi. The culprit was Netty's direct buffer pool plus 400 platform threads with 1MB stacks each — none of it visible in heap metrics. Capping -XX:MaxDirectMemorySize, moving to a bounded thread pool, and alerting on working set instead of heap ended the restarts without raising the limit.

Runtime Memory and CPU Inside Containers

Language runtimes size their heaps and thread pools based on what they think the machine looks like. Old runtimes read /proc/meminfo and see the node's 64GB. Modern ones are container-aware, but the defaults are still rarely what you want.

The Budget Formula

limits.memory ≥ max heap
              + non-heap runtime overhead (JIT, metadata, GC structures)
              + thread stacks (threads × stack size)
              + off-heap buffers (direct/native)
              + headroom for page cache and spikes (10-25%)

JVM

Java 10+ (and 8u191+) detects cgroup limits. The default max heap is only 25% of the container limit, which wastes memory in single-process containers.

containers:
  - name: api
    image: registry.example.com/orders-api:4.2.1
    resources:
      requests:
        cpu: "1"
        memory: 2Gi
      limits:
        memory: 2Gi
    env:
      - name: JAVA_TOOL_OPTIONS
        value: >-
          -XX:MaxRAMPercentage=70.0
          -XX:InitialRAMPercentage=70.0
          -XX:MaxMetaspaceSize=256m
          -XX:MaxDirectMemorySize=192m
          -XX:ActiveProcessorCount=2
          -XX:+ExitOnOutOfMemoryError
          -Xss512k
  • MaxRAMPercentage=70 leaves 30% for metaspace, code cache, threads, and direct buffers. Go to 75-80% only after measuring non-heap usage with Native Memory Tracking (-XX:NativeMemoryTracking=summary).
  • ActiveProcessorCount matters when there is no CPU limit — otherwise the JVM sees every node core and sizes GC and ForkJoin pools accordingly.
  • ExitOnOutOfMemoryError turns a zombie JVM (heap exhausted but process alive) into a clean restart.

Node.js

V8 sets its old-space limit from available memory, and recent Node versions (20+) read the cgroup limit — but the heuristic still leaves either too much or too little room depending on version. Set it explicitly:

containers:
  - name: web
    image: registry.example.com/storefront:9.14.0
    resources:
      requests:
        cpu: 500m
        memory: 1Gi
      limits:
        memory: 1Gi
    env:
      # ~75% of the limit for old space; the rest covers new space,
      # Buffers (allocated outside the V8 heap), code, and native addons
      - name: NODE_OPTIONS
        value: "--max-old-space-size=768"
      - name: UV_THREADPOOL_SIZE
        value: "4"

Buffer and ArrayBuffer memory lives outside the V8 heap, so a service that streams files or handles large payloads needs more headroom than one that only juggles small JSON objects. See Node.js Performance for the full V8 heap model and how to read process.memoryUsage().

Go

Go has no fixed heap ceiling by default; the GC triggers at GOGC (100% growth over live heap). Without a limit, a burst of garbage can push RSS past the container limit before the next cycle. Since Go 1.19, GOMEMLIMIT gives the GC a soft target:

containers:
  - name: ingest
    image: registry.example.com/ingest-worker:2.8.0
    resources:
      requests:
        cpu: "2"
        memory: 512Mi
      limits:
        memory: 512Mi
    env:
      - name: GOMEMLIMIT
        valueFrom:
          resourceFieldRef:
            resource: limits.memory
            divisor: "1"
      # GOMEMLIMIT above equals the hard limit; prefer ~90% via the app or
      # automemlimit library so the GC reacts before the kernel does.
      - name: GOMAXPROCS
        valueFrom:
          resourceFieldRef:
            resource: requests.cpu
            divisor: "1"

In code, the common approach is to set the limit to ~90% of the cgroup value at startup:

package main

import (
	"log/slog"
	"os"
	"runtime/debug"
	"strconv"
	"strings"
)

// setMemoryLimit reads the cgroup v2 memory limit and sets the Go soft
// memory limit to 90% of it, leaving headroom for non-heap memory.
func setMemoryLimit() {
	raw, err := os.ReadFile("/sys/fs/cgroup/memory.max")
	if err != nil {
		slog.Warn("cgroup memory.max not readable; leaving GOMEMLIMIT unset", "err", err)
		return
	}
	s := strings.TrimSpace(string(raw))
	if s == "max" {
		return // no limit configured
	}
	limit, err := strconv.ParseInt(s, 10, 64)
	if err != nil {
		slog.Warn("unparseable memory.max", "value", s)
		return
	}
	soft := limit * 9 / 10
	debug.SetMemoryLimit(soft)
	slog.Info("memory limit configured", "cgroup_bytes", limit, "gomemlimit_bytes", soft)
}

func main() {
	setMemoryLimit()
	// ... start server
}

Go 1.25+ sets GOMAXPROCS from the cgroup CPU limit automatically; for older versions use go.uber.org/automaxprocs. If there is no CPU limit, set GOMAXPROCS from the request, as above, so the scheduler does not spin up one P per node core.

Runtime Cheat Sheet

RuntimeHeap controlThread/CPU controlCommon trap
JVM-XX:MaxRAMPercentage-XX:ActiveProcessorCountDirect buffers and thread stacks outside -Xmx
Node.js--max-old-space-size (MiB)UV_THREADPOOL_SIZEBuffer memory outside V8 heap
GoGOMEMLIMITGOMAXPROCSNo heap ceiling by default; GC lag on bursts
PythonNone (refcount + GC)Worker count in gunicorn/uvicornEach worker is a full copy; fork after import to share pages
.NETDOTNET_GCHeapHardLimitPercentDOTNET_PROCESSOR_COUNTServer GC allocates per-core heaps

LimitRange: Defaults and Guardrails per Namespace

A LimitRange applies defaults to containers that do not declare resources and rejects pods outside allowed bounds. It is enforced by an admission plugin at pod creation time — it never changes running pods.

apiVersion: v1
kind: LimitRange
metadata:
  name: container-defaults
  namespace: team-checkout
spec:
  limits:
    - type: Container
      # Applied when a container omits requests
      defaultRequest:
        cpu: 100m
        memory: 128Mi
      # Applied when a container omits limits
      default:
        memory: 512Mi
      # Hard bounds — pod creation fails outside these
      min:
        cpu: 10m
        memory: 32Mi
      max:
        cpu: "4"
        memory: 8Gi
      # Limit may be at most 4x the request (curbs extreme overcommit)
      maxLimitRequestRatio:
        memory: "4"
    - type: PersistentVolumeClaim
      min:
        storage: 1Gi
      max:
        storage: 200Gi

WARNING

If a LimitRange sets default.cpu, every container that omits a CPU limit gets one — including sidecars injected by service meshes. Teams that have deliberately chosen "no CPU limits" are often surprised to find them reintroduced by a namespace default. Only set default for the resources you actually want capped.

Debug a rejection:

kubectl apply -f deploy.yaml
# Error from server (Forbidden): pods "checkout-..." is forbidden:
#   maximum memory usage per Container is 8Gi, but limit is 12Gi

kubectl describe limitrange -n team-checkout

Note that Deployment creation succeeds even when its pods would be rejected — the failure appears on the ReplicaSet. Check kubectl describe rs or kubectl get events when replicas stay at zero.

ResourceQuota: Budgets per Namespace

A ResourceQuota caps the aggregate resources a namespace can consume. It is the primary tool for multi-tenant fairness and cost control.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-budget
  namespace: team-checkout
spec:
  hard:
    requests.cpu: "40"
    requests.memory: 80Gi
    limits.memory: 120Gi
    pods: "200"
    services.loadbalancers: "2"
    persistentvolumeclaims: "30"
    requests.storage: 2Ti
    count/jobs.batch: "50"
---
# Separate, tighter budget for low-priority work in the same namespace
apiVersion: v1
kind: ResourceQuota
metadata:
  name: batch-budget
  namespace: team-checkout
spec:
  hard:
    requests.cpu: "8"
    requests.memory: 16Gi
  scopeSelector:
    matchExpressions:
      - operator: In
        scopeName: PriorityClass
        values: ["batch-low"]

Key behaviours:

  • Once a quota covers requests.cpu or requests.memory, every new pod in the namespace must declare those requests, or it is rejected. Pair every ResourceQuota with a LimitRange that supplies defaults.
  • Quota is checked at admission. Scaling a Deployment past the quota leaves the extra pods un-created, with exceeded quota events on the ReplicaSet.
  • Quotas count requests, not usage. A namespace can hit its quota while its nodes sit idle — that is the signal to right-size, not to raise the quota.
kubectl describe resourcequota -n team-checkout
# Name:            compute-budget
# Resource         Used    Hard
# --------         ----    ----
# limits.memory    96Gi    120Gi
# pods             143     200
# requests.cpu     38500m  40
# requests.memory  71Gi    80Gi
# Alert before a team hits its quota
kube_resourcequota{type="used"} / ignoring(type) kube_resourcequota{type="hard"} > 0.9

Right-Sizing

Most clusters are dramatically over-requested. Industry surveys routinely find average CPU utilization of requested capacity in the 10-30% range. Right-sizing is the highest-leverage cost lever you have (see FinOps Cost Optimization).

The Data You Need

# CPU: p95 usage over 7 days, per container
quantile_over_time(0.95,
  rate(container_cpu_usage_seconds_total{namespace="shop", container="app"}[5m])[7d:5m]
)

# Memory: max working set over 7 days, per container
max_over_time(
  container_memory_working_set_bytes{namespace="shop", container="app"}[7d]
)

# Request efficiency: usage as a fraction of request
sum by (namespace) (rate(container_cpu_usage_seconds_total{container!=""}[1h]))
/
sum by (namespace) (kube_pod_container_resource_requests{resource="cpu"})

Sizing Heuristics

ResourceRequestLimit
CPUp90-p95 of usage under normal peakNone, or 2-4x request if your policy requires one
Memoryp99 / max working set + 10-20%Equal to request (or close) — memory is not shareable

Setting memory request == limit is the safest default: the scheduler reserves exactly what the container can use, so overcommit cannot trigger node-pressure eviction, and the cgroup kill point is predictable.

Use at least 7 days of data (14 is better) so weekly cycles are included, and re-check after major releases. Startup peaks (JIT warm-up, cache priming) count — a container that needs 1.2Gi for 30 seconds at boot needs a 1.2Gi limit.

VPA as a Recommender

The Vertical Pod Autoscaler computes recommendations from historical usage. Running it in Off mode gives you the numbers without the risk of it restarting pods. Full VPA configuration, including Auto and InPlaceOrRecreate modes and its interaction with HPA, is covered in HPA, VPA & KEDA.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: checkout-recommender
  namespace: shop
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout
  updatePolicy:
    updateMode: "Off"
  resourcePolicy:
    containerPolicies:
      - containerName: app
        controlledResources: ["cpu", "memory"]
        minAllowed:
          cpu: 50m
          memory: 128Mi
        maxAllowed:
          cpu: "4"
          memory: 4Gi
      - containerName: istio-proxy
        mode: "Off"
kubectl get vpa checkout-recommender -n shop \
  -o jsonpath='{.status.recommendation.containerRecommendations[*]}' | jq
# {
#   "containerName": "app",
#   "lowerBound":    { "cpu": "180m", "memory": "410Mi" },
#   "target":        { "cpu": "250m", "memory": "520Mi" },
#   "upperBound":    { "cpu": "900m", "memory": "1100Mi" },
#   "uncappedTarget":{ "cpu": "250m", "memory": "520Mi" }
# }

Use target for requests. Treat upperBound as a sanity check on memory limits. If uncappedTarget differs from target, your maxAllowed is clipping the recommendation — the container genuinely wants more.

WARNING

Do not let VPA (in Auto mode) and HPA both act on the same CPU or memory metric. HPA scales out when utilization relative to request rises; VPA raises the request, which lowers utilization; HPA scales back in. The two controllers chase each other. Use HPA on a custom metric (RPS, queue depth) if you need both.

In-Place Resize

Kubernetes 1.33 promoted in-place pod resize to beta (enabled by default): CPU and memory on a running container can be changed through the resize subresource without recreating the pod. Each container's resizePolicy declares whether a change needs a restart:

containers:
  - name: app
    image: registry.example.com/checkout:7.1.3
    resizePolicy:
      - resourceName: cpu
        restartPolicy: NotRequired
      - resourceName: memory
        restartPolicy: RestartContainer   # most runtimes cannot shrink heaps live
    resources:
      requests: { cpu: 250m, memory: 512Mi }
      limits:   { memory: 512Mi }
kubectl patch pod checkout-7d9f8b6c5-x2k4p -n shop --subresource resize \
  --patch '{"spec":{"containers":[{"name":"app","resources":{"requests":{"cpu":"500m"}}}]}}'

CPU resizes are effectively free. Memory resizes usually still need a container restart because the runtime sized its heap at boot.

Namespace Multi-Tenancy

Resource controls are one layer of a soft multi-tenancy model where teams share a cluster but cannot hurt each other.

flowchart TB
    subgraph Cluster
        subgraph NS1["namespace: team-checkout"]
            Q1["ResourceQuota<br/>40 CPU / 80Gi"]
            L1["LimitRange<br/>defaults + max"]
            P1["Pods"]
        end
        subgraph NS2["namespace: team-search"]
            Q2["ResourceQuota<br/>24 CPU / 96Gi"]
            L2["LimitRange"]
            P2["Pods"]
        end
        PC["PriorityClasses<br/>prod-critical / prod / batch-low"]
        ADM["Admission policy<br/>(require requests,<br/>forbid BestEffort in prod)"]
    end
    RBAC["RBAC: teams cannot edit<br/>their own quota"] --> NS1
    RBAC --> NS2
    PC --> P1
    PC --> P2
    ADM --> P1
    ADM --> P2
LayerMechanismPage
Compute budgetResourceQuota per namespaceThis page
Per-container defaults and boundsLimitRangeThis page
Who wins under pressurePriorityClass + preemptionPod Lifecycle
Who can change budgetsRBAC (platform team owns quota objects)RBAC
Network isolationNetworkPolicy default-denyNetwork Policies
Policy enforcementValidating admission (Kyverno, Gatekeeper, ValidatingAdmissionPolicy)Admission Webhooks
Cost attributionRequests × price, per namespace labelCost Allocation

Enforcing "Every Container Declares Resources"

LimitRange defaults are a safety net, not a strategy — a default of 128Mi is wrong for nearly every real service. Native ValidatingAdmissionPolicy (GA since 1.30) can require explicit values without a webhook:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: require-memory-requests-and-limits
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: ["apps"]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["deployments", "statefulsets", "daemonsets"]
  validations:
    - expression: >-
        object.spec.template.spec.containers.all(c,
          has(c.resources) &&
          has(c.resources.requests) && 'memory' in c.resources.requests &&
          has(c.resources.limits) && 'memory' in c.resources.limits)
      message: "Every container must declare memory requests and limits."
    - expression: >-
        object.spec.template.spec.containers.all(c,
          has(c.resources.requests) && 'cpu' in c.resources.requests)
      message: "Every container must declare a CPU request."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: require-memory-requests-and-limits
spec:
  policyName: require-memory-requests-and-limits
  validationActions: ["Deny"]
  matchResources:
    namespaceSelector:
      matchLabels:
        tenancy: team

Roll it out with validationActions: ["Audit"] first, review the audit annotations, then switch to Deny.

Overcommit Policy

Decide deliberately how much overcommit each tier tolerates:

TierCPUMemoryRationale
Production, latency-criticalRequest = p95, no limitRequest = limitPredictable, never node-evicted for overcommit
Production, standardRequest = p90, no limitRequest = limitSame, slightly denser
Batch / CISmall request, no limitLimit ≤ 2x requestBurst into idle capacity; acceptable to be evicted
Dev / previewSmall requestLimit ≤ 4x request, enforced by LimitRangeDensity over stability

Keep batch and production on separate node pools (taints and tolerations) if batch overcommit is aggressive — it keeps eviction storms away from customer-facing pods.

Production Checklist

ItemWhy
Every container has CPU and memory requestsScheduler accuracy; HPA utilization math; quota admission
Memory limit set, usually equal to requestPredictable OOM point, no overcommit-driven eviction
CPU limits are a deliberate decision, not a defaultAvoid CFS throttling on multi-threaded runtimes
Runtime heap flags derived from the container limitPrevent heap sized to node memory
Thread pools / GOMAXPROCS aligned with CPU allocationAvoid quota burn and context switching
Alerts on working set / limit > 90%Warning before OOMKill
Alerts on CFS throttle ratio > 25% for latency servicesDetect throttling hidden by average CPU
Alerts on OOMKilled restarts (kube_pod_container_status_last_terminated_reason)Catch kills that recover silently
VPA in Off mode on every DeploymentContinuous right-sizing data
LimitRange + ResourceQuota on every tenant namespaceDefaults, bounds, budgets
Admission policy forbids BestEffort in productionBestEffort pods are evicted first and destabilize nodes
kube-reserved / system-reserved configuredProtect kubelet and OS from pod pressure

The broader cluster readiness list lives in the Kubernetes Production Checklist.

# PrometheusRule: resource alerts
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: container-resources
  namespace: monitoring
spec:
  groups:
    - name: container-resources
      rules:
        - alert: ContainerMemoryNearLimit
          expr: |
            max by (namespace, pod, container) (container_memory_working_set_bytes{container!=""})
            /
            max by (namespace, pod, container) (kube_pod_container_resource_limits{resource="memory"})
            > 0.9
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "Container memory working set above 90% of limit"
        - alert: ContainerOOMKilled
          expr: |
            increase(kube_pod_container_status_restarts_total[15m]) > 0
            and on (namespace, pod, container)
            kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
          labels:
            severity: warning
          annotations:
            summary: "Container restarted after OOMKill"
        - alert: ContainerCPUThrottlingHigh
          expr: |
            sum by (namespace, pod, container) (rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m]))
            /
            sum by (namespace, pod, container) (rate(container_cpu_cfs_periods_total{container!=""}[5m]))
            > 0.25
          for: 15m
          labels:
            severity: info
          annotations:
            summary: "Container throttled in more than 25% of CFS periods"

Cross-References