Cloudflare BYOIP Outage (February 2026)
On February 20, 2026, about a quarter of the IP prefixes that Cloudflare customers had brought to its network with Bring Your Own IP (BYOIP) disappeared from the Internet's routing tables. Nothing was attacked and no hardware failed. A new scheduled cleanup task was meant to delete only the prefixes that customers had asked to remove. It called an internal API with a query flag that had no value: ?pending_delete instead of ?pending_delete=true. The server read the empty value as "flag not set", fell through to its default branch, and returned every BYOIP prefix. The task then treated each one as due for deletion.
Within 50 minutes the task had withdrawn about 1,100 prefixes from BGP. For some of them it had also deleted the service bindings that tell Cloudflare's edge what to do with traffic for those addresses. Total customer impact lasted 6 hours and 7 minutes. Most of that time went into rebuilding deleted state, not into finding the bug.
The awkward part is the motive. The task was written for Code Orange: Fail Small, Cloudflare's own reliability programme, to replace a risky manual process with automation. A change made to reduce risk caused the outage.
Primary source: Cloudflare's own post-incident report, Cloudflare outage on February 20, 2026. Unless a claim cites something else, it comes from that report.
TL;DR
| What | About 1,100 of 4,306 BYOIP prefixes (about 25%) withdrawn from BGP, and some had their edge service bindings deleted |
| When | 2026-02-20, 17:56 to 23:03 UTC (impact window in Cloudflare's timeline) |
| Duration | 6 h 7 min of customer impact |
| Trigger | A new recurring cleanup sub-task in the Addressing API, deployed at 17:46 UTC |
| Root cause | The client sent pending_delete with an empty value. The server only filtered when the value was non-empty, so it returned all BYOIP prefixes, and the task deleted them |
| Why it was slow to fix | Prefixes ended up in three different broken states. The worst (about 300 prefixes) needed a global configuration rollout to every edge machine |
| Not affected | Customers on Cloudflare-owned IP space. 1.1.1.1 DNS resolution, including DoH, kept working |
| Attack? | No. Cloudflare states explicitly it was not a cyberattack |
| Fixes promised | Strictly typed API flags, snapshotted operational state with health-mediated rollout and fast rollback, circuit breakers on large or fast withdrawals |
Background: What BYOIP Is
Most Cloudflare customers use IP addresses that Cloudflare owns. Some large customers, such as banks, carriers and enterprises with allow-listed addresses, want to keep using their own IP space, but behind Cloudflare. BYOIP lets them do that. The customer authorises Cloudflare to announce their prefix, for example 203.0.113.0/24, and Cloudflare advertises it over BGP from its anycast network. Traffic for those addresses lands on Cloudflare's edge, where it is handled by CDN and security products, Spectrum (TCP/UDP proxying), Magic Transit (network-layer DDoS protection) or Dedicated Egress.
Two pieces of state matter for every BYOIP prefix:
- Advertisement state. Is the prefix announced to Cloudflare's BGP peers? If not, the Internet has no route to Cloudflare for those addresses.
- Service bindings. The mapping on each edge machine that says "traffic to this prefix belongs to customer X, product Y". If bindings are missing, a packet can arrive at the edge and still have nowhere to go.
Both are managed through Cloudflare's internal Addressing API, which the report calls the authoritative source of truth for addresses on the network. Changes made through it propagate to the edge immediately. That design detail explains both the speed of the damage and the difficulty of undoing it.
flowchart LR
Cust["Customer dashboard / API"] --> AAPI["Addressing API<br/>(source of truth)"]
Tasks["Scheduled tasks<br/>(new: cleanup sub-task)"] --> AAPI
AAPI -->|"advertise / withdraw"| BGPd["BGP speakers"]
AAPI -->|"service bindings"| Edge["Every edge machine"]
BGPd -->|"announcements"| Peers["Peers and transit<br/>(the Internet)"]
Peers -->|"traffic for customer prefix"| Edge
The Alert
The first signal did not come from a monitor on BYOIP prefixes. Cloudflare's timeline says the company was "engaged" at 18:13 UTC because of failures on one.one.one.one, the website for Cloudflare's public DNS resolver. That site's addresses were among the withdrawn BYOIP prefixes. Visitors received HTTP 403 "Edge IP Restricted" errors.
That was 17 minutes after impact started. An internal incident was declared at 18:18, and the Addressing API team was paged at 18:21. The cleanup task kept running in the background and withdrawing prefixes until an engineer terminated and disabled it at 18:46, 50 minutes after it started.
The detection gap
The system that could see the problem directly was the one causing it. The Addressing API knew how many prefixes were being withdrawn per minute, and nothing in the report suggests it alarmed on that rate. Detection came from a symptom on one well-known site. Customers whose prefixes were withdrawn first were already down before Cloudflare engaged.
Impact
Scale. Cloudflare gives these figures:
- One of its BGP peers was receiving about 6,500 prefixes, of which 4,306 were BYOIP. That BYOIP number is every BYOIP prefix Cloudflare advertises globally.
- About 1,100 prefixes (about 25% of BYOIP) were withdrawn between 17:56 and 18:46 UTC.
- About 800 were restored around 20:20 UTC.
- About 300 had their service configuration removed from the edge and needed a global rollout that finished at 23:03 UTC.
- Not every BYOIP customer was hit. The task worked iteratively and was stopped part-way through, so the damage depended on how far it got before it was killed.
What customers saw.
| Product | Effect |
|---|---|
| Core CDN and security services | Traffic never reached Cloudflare, so connections failed |
| Spectrum | BYOIP applications could not proxy traffic |
| Dedicated Egress (Gateway and CDN dedicated IPs on BYOIP) | Outbound traffic could not be sent |
| Magic Transit | Protected applications were unreachable and connections timed out |
one.one.one.one website | HTTP 403 "Edge IP Restricted" |
| 1.1.1.1 DNS resolution (including DoH) | Not affected |
The report describes the network-level symptom as BGP path hunting. When an origin withdraws a prefix, routers do not learn "this is gone" all at once. They try progressively longer alternative paths learned from other neighbours, and those paths are withdrawn one after another as the withdrawal propagates. Connections in flight wander between networks and time out instead of failing fast with a clean "no route". Users see hangs rather than errors.
Even after a prefix was re-advertised, some customers still saw latency and failures, because the configuration for their addresses had to be pushed back out to the edge. A route that delivers packets to machines with no binding for them does not help.
What is not public
Cloudflare does not say how many customers were affected, which customers, or what any of this cost. Secondary coverage repeats the 1,100 and 4,306 figures from the report and adds nothing first-hand. This page uses only the published figures.
Timeline
All times are UTC and come from Cloudflare's report.
| Time (UTC) | Event |
|---|---|
| Feb 5, 21:53 | Change containing the faulty cleanup sub-process merged into the code base |
| Feb 20, 17:46 | Addressing API release containing the sub-process completes deployment |
| Feb 20, 17:56 | Sub-process runs. BYOIP prefixes start being withdrawn. Impact starts |
| 18:13 | Cloudflare engaged because of failures on one.one.one.one |
| 18:18 | Internal incident declared |
| 18:21 | Addressing API team paged |
| 18:46 | An engineer terminates the sub-process and disables it. Remediation begins |
| 19:11 | Mitigation starts on withdrawn and removed prefixes |
| 19:19 | Customers start re-advertising prefixes themselves from the dashboard (impact downgraded) |
| 19:44 | Database recovery starts for prefixes whose records were removed |
| ~20:20 to 20:30 | About 800 prefixes restored. A release at 20:30 restores withdrawn prefixes that still had service bindings (impact downgraded) |
| 21:08 | Global configuration rollout to all edge machines begins, to reapply removed bindings (impact downgraded) |
| 23:03 | Global rollout complete. Impact ends |
Note on the 17:48 vs 17:56 start time
The report's opening paragraph says the issue began at 17:48 UTC. Its detailed timeline says withdrawals began, and impact started, at 17:56 UTC, after the release completed at 17:46. Secondary coverage quotes both. This page uses 17:56 for the impact window because that is the timestamp in the structured timeline, and because the 6 h 7 min total (17:56 to 23:03) matches it. The 17:48 figure may refer to the first observed effect, but the report does not say.
sequenceDiagram
participant Dev as Addressing API team
participant API as Addressing API
participant Task as Cleanup sub-task
participant BGP as BGP speakers and edge
participant Net as The Internet
Dev->>API: Feb 5 21:53 — merge cleanup sub-task
Dev->>API: Feb 20 17:46 — release completes
Task->>API: 17:56 — GET /v1/prefixes?pending_delete
API-->>Task: All BYOIP prefixes (filter skipped)
loop Every returned prefix, iteratively
Task->>API: Delete prefix and dependents
API->>BGP: Withdraw route, remove bindings
BGP->>Net: BGP withdrawal, path hunting
end
Note over Net: one.one.one.one returns 403
Dev->>Dev: 18:13 engaged — 18:18 incident — 18:21 page
Dev->>Task: 18:46 — terminate and disable
Dev->>API: 19:11 to 20:30 — dashboard re-advertise, DB recovery, release
Dev->>BGP: 21:08 to 23:03 — global config rollout
Root Cause
The change: automating a risky manual task
Removing a BYOIP prefix when a customer asks for it was a manual operation. Cloudflare's reliability programme, Code Orange: Fail Small, has three workstreams according to the report: controlled rollouts for all configuration changes, revised break-glass procedures with circular dependencies removed, and well-defined behaviour under failure. Replacing hand-run operational steps with tested automation fits that programme. So a new recurring cleanup sub-task was added to the Addressing API. Each run would:
- Ask the Addressing API for prefixes marked pending deletion.
- For each one, delete the prefix and its dependent objects, such as service bindings, which withdraws the route and removes edge configuration.
The design is reasonable. The bug was in the contract between step 1's caller and step 1's server.
The bug: a flag with no value
The client called:
GET /v1/prefixes?pending_delete
The intent was "only prefixes pending deletion". The server only took the filtered path when reading the parameter produced a non-empty value. Cloudflare describes the server check in terms of Query().Get("pending_delete"). In Go's net/url, a key present with no value parses to an empty string, and Get also returns an empty string for a key that is absent. To that check, "present but empty" and "not sent at all" looked identical. So the server took the unfiltered branch and returned every BYOIP prefix.
The snippet below shows the shape of the bug. It is illustrative, written to match the report's description, and is not Cloudflare's code.
// Illustrative reconstruction of the failure mode — not Cloudflare's code.
func listPrefixes(w http.ResponseWriter, r *http.Request) {
q := r.URL.Query()
// "?pending_delete" -> Get returns "" (key present, empty value)
// "?pending_delete=true" -> Get returns "true"
// (no parameter at all) -> Get returns "" (indistinguishable from the first case)
if v := q.Get("pending_delete"); v != "" {
writeJSON(w, store.PrefixesPendingDelete())
return
}
// Default branch: the full list. Fine for the dashboard,
// catastrophic for a caller that is about to delete everything it gets back.
writeJSON(w, store.AllPrefixes())
}
The cleanup task trusted the response. It had no reason to doubt that the server had applied the filter, and nothing in the response said it had not. So it started deleting.
Cloudflare's own account of why this was hard to catch is that the flag "was interpreted as a string, making it difficult for both client and server to rationalize the value" (as quoted by Lorin Hochstein). The flag was a boolean in meaning and a free-form string in implementation.
flowchart TD
A["Cleanup task runs<br/>(17:56 UTC)"] --> B["GET /v1/prefixes?pending_delete"]
B --> C{"Server: is pending_delete<br/>value non-empty?"}
C -->|"yes (e.g. =true)"| D["Return only pending-delete prefixes<br/>(intended)"]
C -->|"no: value is empty string"| E["Fall through to default:<br/>return ALL BYOIP prefixes"]
E --> F["Task treats every prefix<br/>as queued for deletion"]
F --> G["Delete prefix + dependents<br/>(service bindings)"]
G --> H["Addressing API propagates<br/>immediately to edge"]
H --> I["BGP withdrawal of ~1,100 prefixes"]
H --> J["Bindings removed on edge<br/>for some prefixes"]
I --> K["Customer traffic cannot reach Cloudflare"]
J --> L["Traffic that does arrive has nowhere to go"]
style E fill:#ff6b6b,color:#fff
style F fill:#ff6b6b,color:#fff
style I fill:#cc5500,color:#fff
style J fill:#cc5500,color:#fff
Why the system amplified a one-character bug
A missing =true became a six-hour outage because several properties lined up:
| Property | Effect |
|---|---|
| Fail-open default | The unfiltered branch was the fallback. When the filter was not recognised, the result was "everything", the most dangerous possible answer for a deletion caller |
| Destructive consumer of a read API | The same list endpoint served read-only callers (where "all" is harmless) and a deleting caller (where "all" is catastrophic) |
| Immediate propagation | Addressing API writes reach the edge directly. There was no staged, health-checked rollout between "row deleted" and "route withdrawn worldwide" |
| No blast-radius limit | Nothing stopped a single task run from withdrawing hundreds of prefixes. Cloudflare's own fix list includes adding exactly this |
| Cascading deletes | Deleting a prefix also deleted dependent objects. Recovery then needed data reconstruction, not just a toggle |
| No rollback of operational state | Cloudflare says a snapshot and rollback system for this state was being built but was not yet in production |
Why testing missed it
The report gives two reasons:
- Staging and mocks did not represent production. The test data did not show what a production-sized prefix list would do when the filter was skipped.
- Test coverage focused on the wrong flow. Testing covered the customer self-service path, where a person removes a prefix through the API with an explicit target. It did not cover a task runner changing data on its own with no explicit input. The new behaviour, "the system decides by itself what to delete", is exactly the one that went untested.
A useful mental test for any automated destructive job: what does this job do if its selector returns everything? What does it do if its selector returns nothing? Both are common failure modes of list APIs. Only one of them is usually harmless.
Why Recovery Took Six Hours
Stopping the task took 50 minutes. Undoing what it did took over four more hours, because the task was killed part-way through an iterative run. Prefixes ended up in three different states, and each needed a different repair.
stateDiagram-v2
[*] --> Healthy
Healthy --> WithdrawnOnly: task withdrew route
Healthy --> PartialBindings: route withdrawn and some bindings deleted
Healthy --> NoBindings: route withdrawn and all bindings deleted
WithdrawnOnly --> Healthy: customer re-toggles advertisement in dashboard from 19.19 UTC
PartialBindings --> Healthy: dashboard for some, DB recovery and release at 20.30 UTC for the rest
NoBindings --> Healthy: global config rollout to every edge machine, 21.08 to 23.03 UTC
| State | What was lost | Repair |
|---|---|---|
| 1. Withdrawn only | Advertisement | Customer could re-enable advertisement from the dashboard (available from 19:19) |
| 2. Withdrawn, some bindings removed | Advertisement and part of the edge config | Only some of these could be toggled back by the customer. The rest needed database recovery (from 19:44) and a release (20:30) |
| 3. Withdrawn, all bindings removed | Advertisement and all edge config | Could not be toggled at all. Required a global configuration rollout to every edge machine, about 300 prefixes, finished 23:03 |
This is the core lesson of the recovery. Withdrawal is cheap to undo. Deletion is not. A route that is merely withdrawn can be re-announced in seconds. A service binding that has been deleted has to be reconstructed from some other record and redistributed to a global fleet, and that fleet-wide distribution is itself a risky operation that must be done carefully.
Self-service as a recovery tool
The fastest part of the recovery was customers fixing their own prefixes from the dashboard at 19:19. A self-service "re-advertise" button already existed for normal operations, and it doubled as an incident tool. Lorin Hochstein's analysis notes that the report does not say how much of the later recovery tooling already existed and how much engineers improvised on the day (Surfing Complexity). That detail is unknown.
The Fix
Immediate response
- Stop the bleeding. At 18:46 an engineer terminated the cleanup sub-process and disabled it, so it could not run again on its schedule.
- Re-advertise what could be re-advertised. From 19:19, customers whose prefixes were only withdrawn could restore advertisement themselves.
- Recover deleted records. Database recovery for removed prefixes started at 19:44.
- Release a fix for partially damaged prefixes. At 20:30 a release restored withdrawn prefixes that still had bindings.
- Push configuration to the whole fleet. From 21:08 a global configuration rollout reapplied service bindings to every edge machine. It completed at 23:03.
Remediation Cloudflare committed to
From the report's remediation section:
| Commitment | What it changes |
|---|---|
| API schema standardisation | Stop treating flags such as pending_delete as loosely interpreted strings, so that malformed calls are detected instead of silently changing meaning |
| Separate operational state from configured state | Snapshot the data that is applied to production and deploy it through health-mediated rollouts. This allows near-instant rollback for one customer or for all. The trade-off Cloudflare names: direct API updates may be temporarily blocked during an outage |
| Circuit breakers for large withdrawals | Detect withdrawals or deletions that are happening too fast or too broadly, and halt them. Customer service-health monitoring will also be able to trip the breaker |
These are the right three controls, because they map to three different layers. Input validation stops the bug at the interface. A breaker limits the blast radius when validation fails. Snapshots and rollback shorten recovery when the breaker fails too.
What Would Have Prevented It: Engineering Controls
The controls below are generic and apply to any system where an automated job can delete or withdraw production state. The code is illustrative and not Cloudflare's.
1. Parse flags strictly and reject what you do not understand
A boolean should be a boolean. Distinguish "absent" from "present", and reject a present-but-invalid value with 400 Bad Request instead of falling through to a default.
package prefixes
import (
"fmt"
"net/url"
"strconv"
)
// ParseBoolFlag returns (value, present, error).
// Absent -> (false, false, nil)
// "?f=true" -> (true, true, nil)
// "?f=false" -> (false, true, nil)
// "?f" or "?f=" -> error: a present flag must carry an explicit value
// "?f=yes" -> error
func ParseBoolFlag(q url.Values, name string) (bool, bool, error) {
if !q.Has(name) { // url.Values.Has exists since Go 1.17
return false, false, nil
}
raw := q.Get(name)
if raw == "" {
return false, true, fmt.Errorf("%s: present but empty, send %s=true or %s=false", name, name, name)
}
v, err := strconv.ParseBool(raw)
if err != nil {
return false, true, fmt.Errorf("%s: invalid boolean %q", name, raw)
}
return v, true, nil
}
package prefixes
import (
"net/url"
"testing"
)
func TestParseBoolFlag(t *testing.T) {
cases := []struct {
query string
want, present, e bool
}{
{"", false, false, false},
{"pending_delete=true", true, true, false},
{"pending_delete=false", false, true, false},
{"pending_delete", false, true, true}, // the Feb 20 call shape: must be an error
{"pending_delete=", false, true, true},
{"pending_delete=yes", false, true, true},
}
for _, c := range cases {
q, _ := url.ParseQuery(c.query)
got, present, err := ParseBoolFlag(q, "pending_delete")
if got != c.want || present != c.present || (err != nil) != c.e {
t.Errorf("%q: got (%v,%v,%v)", c.query, got, present, err)
}
}
}
Also reject unknown query parameters on internal APIs. A typo such as ?pendng_delete=true should be an error, not a silent "return everything".
2. Put the contract in a schema, and make filters required for destructive callers
Declaring the parameter as a typed boolean in an OpenAPI spec lets both generated clients and server validators enforce it. Better still, give the destructive caller its own endpoint whose filter is not optional.
# openapi.yaml (excerpt)
paths:
/v1/prefixes:
get:
operationId: listPrefixes
parameters:
- name: pending_delete
in: query
required: false
schema:
type: boolean # generated clients send =true/=false, validators reject ""
allowEmptyValue: false
/v1/prefixes/pending-deletion:
get:
operationId: listPrefixesPendingDeletion
description: >
Used only by the cleanup task. There is no unfiltered mode, so a
malformed request cannot widen the result set.
The design principle: an endpoint that feeds a deletion must not have a "return everything" mode at all. The safest default for a filter that is not understood is an error, and the second-safest is an empty list. "All rows" is the worst choice.
3. Make the destructive job check its own input
Defence in depth: even if the API misbehaves, the job can refuse to act on input that cannot be right.
// Before deleting, re-check every item against the job's own predicate,
// and refuse batches that are implausibly large.
func planDeletions(prefixes []Prefix, totalActive int) ([]Prefix, error) {
var plan []Prefix
for _, p := range prefixes {
if !p.PendingDelete || p.DeleteRequestedAt.IsZero() {
// The API returned something that is not ours to delete.
return nil, fmt.Errorf("prefix %s not marked pending_delete, aborting run", p.CIDR)
}
plan = append(plan, p)
}
const maxAbs = 20 // tune to the normal daily volume of removals
const maxFrac = 0.01 // never touch more than 1% of active prefixes in one run
if len(plan) > maxAbs || float64(len(plan)) > maxFrac*float64(totalActive) {
return nil, fmt.Errorf("plan deletes %d of %d prefixes, exceeds limit, needs human approval",
len(plan), totalActive)
}
return plan, nil
}
With either check in place, the February 20 run would have stopped before its first deletion: the returned prefixes were not marked pending_delete, and there were far more of them than any normal cleanup run would handle.
4. A rate-and-volume circuit breaker on withdrawals
This is Cloudflare's third commitment, and it belongs in the component that performs the withdrawal, not in each caller. Every path to "withdraw a prefix" (task, API, dashboard, operator CLI) then passes through one guard.
// Sliding-window breaker: trips if too many withdrawals happen too fast.
// Once open, it stays open until a human resets it.
type WithdrawalBreaker struct {
mu sync.Mutex
window time.Duration
maxInWin int
events []time.Time
open bool
}
func (b *WithdrawalBreaker) Allow(now time.Time) error {
b.mu.Lock()
defer b.mu.Unlock()
if b.open {
return errors.New("withdrawal breaker open: manual reset required")
}
cutoff := now.Add(-b.window)
kept := b.events[:0]
for _, t := range b.events {
if t.After(cutoff) {
kept = append(kept, t)
}
}
b.events = kept
if len(b.events) >= b.maxInWin {
b.open = true
alertOncall("withdrawal breaker tripped", len(b.events), b.window)
return errors.New("withdrawal rate limit exceeded")
}
b.events = append(b.events, now)
return nil
}
A breaker like this needs per-region or per-fleet state in a real deployment, and possibly a separate limit for deletions as well as withdrawals. The Circuit Breaker Pattern page covers the general idea. Here it protects a dependency from you rather than you from a dependency.
5. Prefer soft-delete and separate "withdraw" from "destroy"
The difference between a 20-minute and a 6-hour outage was cascade deletion. Make destruction a two-step process with time in between:
-- Step 1 (automated): mark and withdraw. Fully reversible.
UPDATE prefixes
SET advertised = false,
state = 'withdrawn_pending_destroy',
destroy_after = now() + interval '7 days'
WHERE id = $1
AND state = 'pending_delete'; -- predicate in the write, not only in the read
-- Step 2 (separate job, days later): only then remove bindings and rows.
DELETE FROM service_bindings
WHERE prefix_id IN (
SELECT id FROM prefixes
WHERE state = 'withdrawn_pending_destroy' AND destroy_after < now()
);
Note the AND state = 'pending_delete' in step 1. Putting the safety predicate in the write means a bad read cannot widen the write.
6. Snapshot operational state and roll it out like code
Cloudflare's second commitment is to stop treating the Addressing API's live database as the thing the edge reads directly. Instead, produce a versioned snapshot of the desired state and push it through the same staged, health-checked pipeline used for software. Rollback then means "redeploy snapshot N-1", which is fast and well-practised.
flowchart LR
API["Addressing API<br/>(configured state)"] --> Snap["Versioned snapshot N"]
Snap --> Diff{"Diff vs N-1:<br/>withdrawals > limit?"}
Diff -->|"yes"| Hold["Hold, page a human"]
Diff -->|"no"| Canary["Canary PoPs"]
Canary --> Health{"Customer health OK?"}
Health -->|"no"| Roll["Auto-rollback to N-1"]
Health -->|"yes"| Wave["Staged waves to all PoPs"]
Wave --> Health2{"Health OK?"}
Health2 -->|"no"| Roll
Health2 -->|"yes"| Done["Snapshot N live"]
This is the same reasoning as canary deployments and GitOps: configuration and data that change production behaviour should flow through a pipeline with gates and a one-step rollback. Cloudflare is explicit about the cost: during an outage, direct API changes may have to wait for the pipeline.
7. Alert on the thing that matters: announced prefixes
Detection came 17 minutes late, from a symptom. A direct signal was available.
# Prometheus alert rules (illustrative metric names)
groups:
- name: prefix-advertisement
rules:
- alert: AnnouncedPrefixesDropped
expr: |
(sum(bgp_announced_prefixes{type="byoip"})
/ sum(bgp_announced_prefixes{type="byoip"} offset 15m)) < 0.98
for: 2m
labels: { severity: page }
annotations:
summary: "BYOIP announced prefixes fell >2% in 15 minutes"
- alert: WithdrawalRateHigh
expr: sum(rate(prefix_withdrawals_total[5m])) * 300 > 10
for: 1m
labels: { severity: page }
annotations:
summary: "More than 10 prefix withdrawals in 5 minutes"
Pair internal metrics with external checks: BGP collectors such as RIPE RIS or RouteViews, or a synthetic probe to a canary address inside each customer prefix. The external view catches failures that internal state does not report.
8. Ship destructive automation dark first
A new job that deletes things should run in dry-run mode in production for a while, logging what it would have done. Behind a feature flag, it can then be enabled for one prefix, then a few, then all.
if cfg.CleanupDryRun {
log.Info("cleanup dry-run", "would_delete", len(plan), "cidrs", cidrs(plan))
metrics.CleanupWouldDelete.Set(float64(len(plan)))
return nil
}
On February 20 a dry-run log line saying "would delete 4,306 prefixes" would have been an unmissable signal at zero cost.
Lessons for System Design
1. A read API's default is a safety property. "No filter means all rows" is a sensible default for a dashboard and a dangerous one for a job that deletes what it reads. When one endpoint serves both kinds of caller, design the default for the most dangerous one, or split the endpoint.
2. Absent, empty and false are three different values. Many frameworks collapse them. Query strings, environment variables, YAML (key: with no value), JSON null and form fields all have this ambiguity. Decide explicitly what each means at every trust boundary, even internal ones.
3. Put the predicate in the write. DELETE ... WHERE state = 'pending_delete' cannot delete live rows, whatever list the caller passes. A safety check performed only in a preceding read is a check the write does not enforce.
4. Limit blast radius where the effect happens. One guard in the withdrawal path protects every caller: today's task, tomorrow's task, and an operator at 3 AM. Per-caller guards will be forgotten by the next caller. This is the same argument as cell-style isolation for fleet-wide pushes.
5. Cheap-to-undo first, expensive-to-undo later. Withdraw, then destroy days later. Disable, then delete. Every system with cascade deletes should ask how it would rebuild what the cascade removed, and how long that would take.
6. Immediate global propagation is a design choice with a cost. A source of truth that writes straight to every edge machine is fast in normal operation and has no brakes in an incident. Staged rollout of data deserves the same care as staged rollout of code. Cloudflare learned this from configuration files in 2019. This incident shows that API-driven state needs it too.
7. Reliability work is change, and change carries risk. This was a reliability improvement that caused an outage. That is not an argument against automation, since the manual process carried its own risks. It is an argument for treating "make it safer" changes with the same rollout discipline as any other. As Lorin Hochstein put it, cleanup is inherently hazardous, but "not cleaning up is also risky" (Surfing Complexity).
8. Test the autonomous path, with production-shaped data. The tests covered the human-initiated flow. The automation was the new risk and it went untested. A property-style test along the lines of "the cleanup task never deletes a prefix not marked pending" would have caught this, whatever the API returned.
9. Detect from the system's own view of itself. The Addressing API knew how many prefixes it had withdrawn. Alerting on the rate of destructive operations is cheap and catches this whole class of bug.
What Would You Do?
Scenario 1: You are on call. At 18:15 you learn that a public site is returning 403s, and a teammate notices that BYOIP prefix announcements are falling. You do not yet know why. A new cleanup task shipped 30 minutes ago. Do you (A) investigate the 403s on the affected site first, (B) disable the newest change that touches prefix state, even though you have not proven it is the cause, or (C) start re-announcing withdrawn prefixes by hand?
Best answer: (B). When state is actively being destroyed, stop the most likely writer first and diagnose second. A recent deploy to the system that owns the damaged state is the prime suspect, and disabling a cleanup task is low-risk because nothing breaks if cleanup waits a day. (C) loses a race against a task that is still running: you re-announce and it withdraws again. (A) treats a symptom. In the real incident the task ran until 18:46, 50 minutes after it started, and every extra minute meant more prefixes in the expensive "bindings deleted" state.
Scenario 2: You are reviewing the pull request for the cleanup task. The tests mock the list API to return three prefixes marked pending deletion, and the task deletes all three. The author says it is covered. What do you ask for?
Ask what happens when the API returns something unexpected. Add tests where the mock returns (1) prefixes that are not marked pending, (2) a list far larger than normal, and (3) an empty list. The task should refuse (1) and (2) and do nothing on (3). Ask for a dry-run mode and a per-run deletion cap. Ask for the client to use a typed generated client or an explicit =true. Most importantly, ask that the deletion SQL itself carry the state = 'pending_delete' predicate. A test that only mocks the happy path checks the code the author imagined, not the contract the server implements.
Scenario 3: After the incident, leadership asks for one change that would most reduce the impact of the next bug of this kind, whatever its trigger. Options: (A) strict schema validation on all internal APIs, (B) a circuit breaker on prefix withdrawals and deletions, enforced in the component that applies them, (C) more end-to-end tests in staging.
Best single answer: (B). Validation (A) prevents this bug, and you should do it, but the next bug may be a wrong SQL predicate, an operator script or a different caller. A breaker in the component that applies withdrawals bounds the damage from all of them. Staging tests (C) help, but Cloudflare's report says its staging data did not represent production, which is a common and persistent problem. Bounding blast radius is the control that still works when the bug is one you did not predict.
Key Lessons
- A missing
=truewithdrew about 1,100 prefixes. The server treated an empty flag as absent and returned everything, and the deleting caller trusted the result. - Fail closed on filters that feed deletes. An unrecognised filter must produce an error or an empty list, never "all rows".
- Put the safety predicate in the write, and have destructive jobs check their own input and refuse implausibly large batches.
- Breakers belong where the effect happens. One guard on withdrawals protects every caller.
- Deletion turned a quick fix into a six-hour one. Withdrawn prefixes came back with a dashboard toggle. Deleted bindings needed a global fleet rollout.
- Roll out data like code. Snapshots, health-gated waves and one-step rollback are what Cloudflare committed to build.
- Reliability changes are still changes. The task was part of Code Orange: Fail Small.
Quiz
Quiz
Q1: What exactly was wrong with the request GET /v1/prefixes?pending_delete?
The flag was present with no value. The server only applied the "pending deletion" filter when the value was non-empty. An empty value looked the same as an absent parameter, so the server took its default branch and returned every BYOIP prefix. The client intended pending_delete=true.
Q2: Why did some prefixes recover around 19:19 while others needed until 23:03? The task was stopped part-way through an iterative run, leaving prefixes in three states. Prefixes that were only withdrawn could be re-advertised by customers from the dashboard. Prefixes with some bindings removed needed database recovery and a release. About 300 prefixes with all bindings removed needed a global configuration rollout to every edge machine.
Q3: What is BGP path hunting, and why does it make outages look like hangs instead of errors? When a prefix is withdrawn, routers do not learn at once that it is gone. They fall back to alternative paths learned from other neighbours, which are themselves withdrawn as the change propagates. Packets are forwarded along paths that are about to disappear, so connections time out instead of failing fast with "no route to host".
Q4: Name the three remediation commitments in Cloudflare's report and the layer each protects. (1) Standardised API schemas with strictly typed flags, which protects the interface by rejecting malformed calls. (2) Separating operational from configured state, with snapshots, health-mediated rollouts and fast rollback, which protects recovery. (3) Circuit breakers on withdrawals or deletions that are too fast or too broad, which bounds the blast radius.
Q5: Why is "put the predicate in the write" stronger than checking the list before deleting?
A check done only on a preceding read depends on that read being correct, and here the read was wrong. If the delete statement itself says WHERE state = 'pending_delete', it cannot remove live rows whatever list the caller passes in. The guarantee then sits where the damage would happen.
Q6: Why was this incident's root cause not an unscoped query?
It looks like a DELETE with no WHERE clause, but the scope existed: the server had a filtered branch. The failure was in how the flag was encoded and parsed. Client and server disagreed about what a valueless flag meant, and the disagreement resolved to the most dangerous interpretation.
Related Pages
- BGP — Border Gateway Protocol — how announcements, withdrawals and anycast work
- Facebook October 2021 — another self-inflicted BGP withdrawal, by a maintenance command, with a much larger blast radius
- Cloudflare July 2019 — Cloudflare's earlier global outage from a configuration push without staged rollout
- CrowdStrike July 2024 — fleet-wide content push without staged rollout
- GitLab Database Deletion 2017 — destructive command against the wrong target, and why recovery was hard
- AWS S3 February 2017 — an operator command that removed more capacity than intended, and AWS's response of adding rate limits on removal
- Circuit Breaker Pattern
- REST API Best Practices
- Canary Deployment and Rollback Procedures
- Feature Flags Deep Dive
- GitOps
- Postmortem Framework
One-Liner Summary
A reliability-motivated cleanup job sent a boolean flag without a value, the API read it as "no filter" and returned every BYOIP prefix, and the job withdrew about 1,100 of them from the Internet. Stopping it took 50 minutes, and rebuilding what it had deleted took more than four hours after that.
Sources
- Cloudflare, Cloudflare outage on February 20, 2026 — the primary post-incident report: timeline, prefix counts, root cause, the three recovery states and the remediation commitments
- Lorin Hochstein, Quick takes on Feb 20 Cloudflare outage — independent analysis by a resilience-engineering practitioner, including the "not cleaning up is also risky" observation and the open questions about recovery tooling
- 365i, Cloudflare Outage: Automation Script Deleted 1,100 BGP Prefixes — secondary coverage of the report's figures
- The Breach, The Cleanup Script That Cleaned Too Much — secondary coverage
- Go standard library,
net/urlValues.Get and Values.Has — documents thatGetreturns""for a missing key, and thatHasreports presence