On this page

Cloudflare BYOIP Outage (February 2026)

On February 20, 2026, about a quarter of the IP prefixes that Cloudflare customers had brought to its network with Bring Your Own IP (BYOIP) disappeared from the Internet's routing tables. Nothing was attacked and no hardware failed. A new scheduled cleanup task was meant to delete only the prefixes that customers had asked to remove. It called an internal API with a query flag that had no value: ?pending_delete instead of ?pending_delete=true. The server read the empty value as "flag not set", fell through to its default branch, and returned every BYOIP prefix. The task then treated each one as due for deletion.

Within 50 minutes the task had withdrawn about 1,100 prefixes from BGP. For some of them it had also deleted the service bindings that tell Cloudflare's edge what to do with traffic for those addresses. Total customer impact lasted 6 hours and 7 minutes. Most of that time went into rebuilding deleted state, not into finding the bug.

The awkward part is the motive. The task was written for Code Orange: Fail Small, Cloudflare's own reliability programme, to replace a risky manual process with automation. A change made to reduce risk caused the outage.

Primary source: Cloudflare's own post-incident report, Cloudflare outage on February 20, 2026. Unless a claim cites something else, it comes from that report.

TL;DR

WhatAbout 1,100 of 4,306 BYOIP prefixes (about 25%) withdrawn from BGP, and some had their edge service bindings deleted
When2026-02-20, 17:56 to 23:03 UTC (impact window in Cloudflare's timeline)
Duration6 h 7 min of customer impact
TriggerA new recurring cleanup sub-task in the Addressing API, deployed at 17:46 UTC
Root causeThe client sent pending_delete with an empty value. The server only filtered when the value was non-empty, so it returned all BYOIP prefixes, and the task deleted them
Why it was slow to fixPrefixes ended up in three different broken states. The worst (about 300 prefixes) needed a global configuration rollout to every edge machine
Not affectedCustomers on Cloudflare-owned IP space. 1.1.1.1 DNS resolution, including DoH, kept working
Attack?No. Cloudflare states explicitly it was not a cyberattack
Fixes promisedStrictly typed API flags, snapshotted operational state with health-mediated rollout and fast rollback, circuit breakers on large or fast withdrawals

Background: What BYOIP Is

Most Cloudflare customers use IP addresses that Cloudflare owns. Some large customers, such as banks, carriers and enterprises with allow-listed addresses, want to keep using their own IP space, but behind Cloudflare. BYOIP lets them do that. The customer authorises Cloudflare to announce their prefix, for example 203.0.113.0/24, and Cloudflare advertises it over BGP from its anycast network. Traffic for those addresses lands on Cloudflare's edge, where it is handled by CDN and security products, Spectrum (TCP/UDP proxying), Magic Transit (network-layer DDoS protection) or Dedicated Egress.

Two pieces of state matter for every BYOIP prefix:

  1. Advertisement state. Is the prefix announced to Cloudflare's BGP peers? If not, the Internet has no route to Cloudflare for those addresses.
  2. Service bindings. The mapping on each edge machine that says "traffic to this prefix belongs to customer X, product Y". If bindings are missing, a packet can arrive at the edge and still have nowhere to go.

Both are managed through Cloudflare's internal Addressing API, which the report calls the authoritative source of truth for addresses on the network. Changes made through it propagate to the edge immediately. That design detail explains both the speed of the damage and the difficulty of undoing it.

flowchart LR
    Cust["Customer dashboard / API"] --> AAPI["Addressing API<br/>(source of truth)"]
    Tasks["Scheduled tasks<br/>(new: cleanup sub-task)"] --> AAPI
    AAPI -->|"advertise / withdraw"| BGPd["BGP speakers"]
    AAPI -->|"service bindings"| Edge["Every edge machine"]
    BGPd -->|"announcements"| Peers["Peers and transit<br/>(the Internet)"]
    Peers -->|"traffic for customer prefix"| Edge

The Alert

The first signal did not come from a monitor on BYOIP prefixes. Cloudflare's timeline says the company was "engaged" at 18:13 UTC because of failures on one.one.one.one, the website for Cloudflare's public DNS resolver. That site's addresses were among the withdrawn BYOIP prefixes. Visitors received HTTP 403 "Edge IP Restricted" errors.

That was 17 minutes after impact started. An internal incident was declared at 18:18, and the Addressing API team was paged at 18:21. The cleanup task kept running in the background and withdrawing prefixes until an engineer terminated and disabled it at 18:46, 50 minutes after it started.

The detection gap

The system that could see the problem directly was the one causing it. The Addressing API knew how many prefixes were being withdrawn per minute, and nothing in the report suggests it alarmed on that rate. Detection came from a symptom on one well-known site. Customers whose prefixes were withdrawn first were already down before Cloudflare engaged.

Impact

Scale. Cloudflare gives these figures:

  • One of its BGP peers was receiving about 6,500 prefixes, of which 4,306 were BYOIP. That BYOIP number is every BYOIP prefix Cloudflare advertises globally.
  • About 1,100 prefixes (about 25% of BYOIP) were withdrawn between 17:56 and 18:46 UTC.
  • About 800 were restored around 20:20 UTC.
  • About 300 had their service configuration removed from the edge and needed a global rollout that finished at 23:03 UTC.
  • Not every BYOIP customer was hit. The task worked iteratively and was stopped part-way through, so the damage depended on how far it got before it was killed.

What customers saw.

ProductEffect
Core CDN and security servicesTraffic never reached Cloudflare, so connections failed
SpectrumBYOIP applications could not proxy traffic
Dedicated Egress (Gateway and CDN dedicated IPs on BYOIP)Outbound traffic could not be sent
Magic TransitProtected applications were unreachable and connections timed out
one.one.one.one websiteHTTP 403 "Edge IP Restricted"
1.1.1.1 DNS resolution (including DoH)Not affected

The report describes the network-level symptom as BGP path hunting. When an origin withdraws a prefix, routers do not learn "this is gone" all at once. They try progressively longer alternative paths learned from other neighbours, and those paths are withdrawn one after another as the withdrawal propagates. Connections in flight wander between networks and time out instead of failing fast with a clean "no route". Users see hangs rather than errors.

Even after a prefix was re-advertised, some customers still saw latency and failures, because the configuration for their addresses had to be pushed back out to the edge. A route that delivers packets to machines with no binding for them does not help.

What is not public

Cloudflare does not say how many customers were affected, which customers, or what any of this cost. Secondary coverage repeats the 1,100 and 4,306 figures from the report and adds nothing first-hand. This page uses only the published figures.

Timeline

All times are UTC and come from Cloudflare's report.

Time (UTC)Event
Feb 5, 21:53Change containing the faulty cleanup sub-process merged into the code base
Feb 20, 17:46Addressing API release containing the sub-process completes deployment
Feb 20, 17:56Sub-process runs. BYOIP prefixes start being withdrawn. Impact starts
18:13Cloudflare engaged because of failures on one.one.one.one
18:18Internal incident declared
18:21Addressing API team paged
18:46An engineer terminates the sub-process and disables it. Remediation begins
19:11Mitigation starts on withdrawn and removed prefixes
19:19Customers start re-advertising prefixes themselves from the dashboard (impact downgraded)
19:44Database recovery starts for prefixes whose records were removed
~20:20 to 20:30About 800 prefixes restored. A release at 20:30 restores withdrawn prefixes that still had service bindings (impact downgraded)
21:08Global configuration rollout to all edge machines begins, to reapply removed bindings (impact downgraded)
23:03Global rollout complete. Impact ends
Note on the 17:48 vs 17:56 start time

The report's opening paragraph says the issue began at 17:48 UTC. Its detailed timeline says withdrawals began, and impact started, at 17:56 UTC, after the release completed at 17:46. Secondary coverage quotes both. This page uses 17:56 for the impact window because that is the timestamp in the structured timeline, and because the 6 h 7 min total (17:56 to 23:03) matches it. The 17:48 figure may refer to the first observed effect, but the report does not say.

sequenceDiagram
    participant Dev as Addressing API team
    participant API as Addressing API
    participant Task as Cleanup sub-task
    participant BGP as BGP speakers and edge
    participant Net as The Internet

    Dev->>API: Feb 5 21:53 — merge cleanup sub-task
    Dev->>API: Feb 20 17:46 — release completes
    Task->>API: 17:56 — GET /v1/prefixes?pending_delete
    API-->>Task: All BYOIP prefixes (filter skipped)
    loop Every returned prefix, iteratively
        Task->>API: Delete prefix and dependents
        API->>BGP: Withdraw route, remove bindings
        BGP->>Net: BGP withdrawal, path hunting
    end
    Note over Net: one.one.one.one returns 403
    Dev->>Dev: 18:13 engaged — 18:18 incident — 18:21 page
    Dev->>Task: 18:46 — terminate and disable
    Dev->>API: 19:11 to 20:30 — dashboard re-advertise, DB recovery, release
    Dev->>BGP: 21:08 to 23:03 — global config rollout

Root Cause

The change: automating a risky manual task

Removing a BYOIP prefix when a customer asks for it was a manual operation. Cloudflare's reliability programme, Code Orange: Fail Small, has three workstreams according to the report: controlled rollouts for all configuration changes, revised break-glass procedures with circular dependencies removed, and well-defined behaviour under failure. Replacing hand-run operational steps with tested automation fits that programme. So a new recurring cleanup sub-task was added to the Addressing API. Each run would:

  1. Ask the Addressing API for prefixes marked pending deletion.
  2. For each one, delete the prefix and its dependent objects, such as service bindings, which withdraws the route and removes edge configuration.

The design is reasonable. The bug was in the contract between step 1's caller and step 1's server.

The bug: a flag with no value

The client called:

GET /v1/prefixes?pending_delete

The intent was "only prefixes pending deletion". The server only took the filtered path when reading the parameter produced a non-empty value. Cloudflare describes the server check in terms of Query().Get("pending_delete"). In Go's net/url, a key present with no value parses to an empty string, and Get also returns an empty string for a key that is absent. To that check, "present but empty" and "not sent at all" looked identical. So the server took the unfiltered branch and returned every BYOIP prefix.

The snippet below shows the shape of the bug. It is illustrative, written to match the report's description, and is not Cloudflare's code.

// Illustrative reconstruction of the failure mode — not Cloudflare's code.
func listPrefixes(w http.ResponseWriter, r *http.Request) {
    q := r.URL.Query()

    // "?pending_delete"      -> Get returns ""   (key present, empty value)
    // "?pending_delete=true" -> Get returns "true"
    // (no parameter at all)  -> Get returns ""   (indistinguishable from the first case)
    if v := q.Get("pending_delete"); v != "" {
        writeJSON(w, store.PrefixesPendingDelete())
        return
    }

    // Default branch: the full list. Fine for the dashboard,
    // catastrophic for a caller that is about to delete everything it gets back.
    writeJSON(w, store.AllPrefixes())
}

The cleanup task trusted the response. It had no reason to doubt that the server had applied the filter, and nothing in the response said it had not. So it started deleting.

Cloudflare's own account of why this was hard to catch is that the flag "was interpreted as a string, making it difficult for both client and server to rationalize the value" (as quoted by Lorin Hochstein). The flag was a boolean in meaning and a free-form string in implementation.

flowchart TD
    A["Cleanup task runs<br/>(17:56 UTC)"] --> B["GET /v1/prefixes?pending_delete"]
    B --> C{"Server: is pending_delete<br/>value non-empty?"}
    C -->|"yes (e.g. =true)"| D["Return only pending-delete prefixes<br/>(intended)"]
    C -->|"no: value is empty string"| E["Fall through to default:<br/>return ALL BYOIP prefixes"]
    E --> F["Task treats every prefix<br/>as queued for deletion"]
    F --> G["Delete prefix + dependents<br/>(service bindings)"]
    G --> H["Addressing API propagates<br/>immediately to edge"]
    H --> I["BGP withdrawal of ~1,100 prefixes"]
    H --> J["Bindings removed on edge<br/>for some prefixes"]
    I --> K["Customer traffic cannot reach Cloudflare"]
    J --> L["Traffic that does arrive has nowhere to go"]

    style E fill:#ff6b6b,color:#fff
    style F fill:#ff6b6b,color:#fff
    style I fill:#cc5500,color:#fff
    style J fill:#cc5500,color:#fff

Why the system amplified a one-character bug

A missing =true became a six-hour outage because several properties lined up:

PropertyEffect
Fail-open defaultThe unfiltered branch was the fallback. When the filter was not recognised, the result was "everything", the most dangerous possible answer for a deletion caller
Destructive consumer of a read APIThe same list endpoint served read-only callers (where "all" is harmless) and a deleting caller (where "all" is catastrophic)
Immediate propagationAddressing API writes reach the edge directly. There was no staged, health-checked rollout between "row deleted" and "route withdrawn worldwide"
No blast-radius limitNothing stopped a single task run from withdrawing hundreds of prefixes. Cloudflare's own fix list includes adding exactly this
Cascading deletesDeleting a prefix also deleted dependent objects. Recovery then needed data reconstruction, not just a toggle
No rollback of operational stateCloudflare says a snapshot and rollback system for this state was being built but was not yet in production

Why testing missed it

The report gives two reasons:

  1. Staging and mocks did not represent production. The test data did not show what a production-sized prefix list would do when the filter was skipped.
  2. Test coverage focused on the wrong flow. Testing covered the customer self-service path, where a person removes a prefix through the API with an explicit target. It did not cover a task runner changing data on its own with no explicit input. The new behaviour, "the system decides by itself what to delete", is exactly the one that went untested.

A useful mental test for any automated destructive job: what does this job do if its selector returns everything? What does it do if its selector returns nothing? Both are common failure modes of list APIs. Only one of them is usually harmless.

Why Recovery Took Six Hours

Stopping the task took 50 minutes. Undoing what it did took over four more hours, because the task was killed part-way through an iterative run. Prefixes ended up in three different states, and each needed a different repair.

stateDiagram-v2
    [*] --> Healthy
    Healthy --> WithdrawnOnly: task withdrew route
    Healthy --> PartialBindings: route withdrawn and some bindings deleted
    Healthy --> NoBindings: route withdrawn and all bindings deleted
    WithdrawnOnly --> Healthy: customer re-toggles advertisement in dashboard from 19.19 UTC
    PartialBindings --> Healthy: dashboard for some, DB recovery and release at 20.30 UTC for the rest
    NoBindings --> Healthy: global config rollout to every edge machine, 21.08 to 23.03 UTC
StateWhat was lostRepair
1. Withdrawn onlyAdvertisementCustomer could re-enable advertisement from the dashboard (available from 19:19)
2. Withdrawn, some bindings removedAdvertisement and part of the edge configOnly some of these could be toggled back by the customer. The rest needed database recovery (from 19:44) and a release (20:30)
3. Withdrawn, all bindings removedAdvertisement and all edge configCould not be toggled at all. Required a global configuration rollout to every edge machine, about 300 prefixes, finished 23:03

This is the core lesson of the recovery. Withdrawal is cheap to undo. Deletion is not. A route that is merely withdrawn can be re-announced in seconds. A service binding that has been deleted has to be reconstructed from some other record and redistributed to a global fleet, and that fleet-wide distribution is itself a risky operation that must be done carefully.

Self-service as a recovery tool

The fastest part of the recovery was customers fixing their own prefixes from the dashboard at 19:19. A self-service "re-advertise" button already existed for normal operations, and it doubled as an incident tool. Lorin Hochstein's analysis notes that the report does not say how much of the later recovery tooling already existed and how much engineers improvised on the day (Surfing Complexity). That detail is unknown.

The Fix

Immediate response

  1. Stop the bleeding. At 18:46 an engineer terminated the cleanup sub-process and disabled it, so it could not run again on its schedule.
  2. Re-advertise what could be re-advertised. From 19:19, customers whose prefixes were only withdrawn could restore advertisement themselves.
  3. Recover deleted records. Database recovery for removed prefixes started at 19:44.
  4. Release a fix for partially damaged prefixes. At 20:30 a release restored withdrawn prefixes that still had bindings.
  5. Push configuration to the whole fleet. From 21:08 a global configuration rollout reapplied service bindings to every edge machine. It completed at 23:03.

Remediation Cloudflare committed to

From the report's remediation section:

CommitmentWhat it changes
API schema standardisationStop treating flags such as pending_delete as loosely interpreted strings, so that malformed calls are detected instead of silently changing meaning
Separate operational state from configured stateSnapshot the data that is applied to production and deploy it through health-mediated rollouts. This allows near-instant rollback for one customer or for all. The trade-off Cloudflare names: direct API updates may be temporarily blocked during an outage
Circuit breakers for large withdrawalsDetect withdrawals or deletions that are happening too fast or too broadly, and halt them. Customer service-health monitoring will also be able to trip the breaker

These are the right three controls, because they map to three different layers. Input validation stops the bug at the interface. A breaker limits the blast radius when validation fails. Snapshots and rollback shorten recovery when the breaker fails too.

What Would Have Prevented It: Engineering Controls

The controls below are generic and apply to any system where an automated job can delete or withdraw production state. The code is illustrative and not Cloudflare's.

1. Parse flags strictly and reject what you do not understand

A boolean should be a boolean. Distinguish "absent" from "present", and reject a present-but-invalid value with 400 Bad Request instead of falling through to a default.

package prefixes

import (
    "fmt"
    "net/url"
    "strconv"
)

// ParseBoolFlag returns (value, present, error).
// Absent        -> (false, false, nil)
// "?f=true"     -> (true,  true,  nil)
// "?f=false"    -> (false, true,  nil)
// "?f" or "?f=" -> error: a present flag must carry an explicit value
// "?f=yes"      -> error
func ParseBoolFlag(q url.Values, name string) (bool, bool, error) {
    if !q.Has(name) { // url.Values.Has exists since Go 1.17
        return false, false, nil
    }
    raw := q.Get(name)
    if raw == "" {
        return false, true, fmt.Errorf("%s: present but empty, send %s=true or %s=false", name, name, name)
    }
    v, err := strconv.ParseBool(raw)
    if err != nil {
        return false, true, fmt.Errorf("%s: invalid boolean %q", name, raw)
    }
    return v, true, nil
}
package prefixes

import (
    "net/url"
    "testing"
)

func TestParseBoolFlag(t *testing.T) {
    cases := []struct {
        query            string
        want, present, e bool
    }{
        {"", false, false, false},
        {"pending_delete=true", true, true, false},
        {"pending_delete=false", false, true, false},
        {"pending_delete", false, true, true}, // the Feb 20 call shape: must be an error
        {"pending_delete=", false, true, true},
        {"pending_delete=yes", false, true, true},
    }
    for _, c := range cases {
        q, _ := url.ParseQuery(c.query)
        got, present, err := ParseBoolFlag(q, "pending_delete")
        if got != c.want || present != c.present || (err != nil) != c.e {
            t.Errorf("%q: got (%v,%v,%v)", c.query, got, present, err)
        }
    }
}

Also reject unknown query parameters on internal APIs. A typo such as ?pendng_delete=true should be an error, not a silent "return everything".

2. Put the contract in a schema, and make filters required for destructive callers

Declaring the parameter as a typed boolean in an OpenAPI spec lets both generated clients and server validators enforce it. Better still, give the destructive caller its own endpoint whose filter is not optional.

# openapi.yaml (excerpt)
paths:
  /v1/prefixes:
    get:
      operationId: listPrefixes
      parameters:
        - name: pending_delete
          in: query
          required: false
          schema:
            type: boolean        # generated clients send =true/=false, validators reject ""
          allowEmptyValue: false
  /v1/prefixes/pending-deletion:
    get:
      operationId: listPrefixesPendingDeletion
      description: >
        Used only by the cleanup task. There is no unfiltered mode, so a
        malformed request cannot widen the result set.

The design principle: an endpoint that feeds a deletion must not have a "return everything" mode at all. The safest default for a filter that is not understood is an error, and the second-safest is an empty list. "All rows" is the worst choice.

3. Make the destructive job check its own input

Defence in depth: even if the API misbehaves, the job can refuse to act on input that cannot be right.

// Before deleting, re-check every item against the job's own predicate,
// and refuse batches that are implausibly large.
func planDeletions(prefixes []Prefix, totalActive int) ([]Prefix, error) {
    var plan []Prefix
    for _, p := range prefixes {
        if !p.PendingDelete || p.DeleteRequestedAt.IsZero() {
            // The API returned something that is not ours to delete.
            return nil, fmt.Errorf("prefix %s not marked pending_delete, aborting run", p.CIDR)
        }
        plan = append(plan, p)
    }
    const maxAbs = 20       // tune to the normal daily volume of removals
    const maxFrac = 0.01    // never touch more than 1% of active prefixes in one run
    if len(plan) > maxAbs || float64(len(plan)) > maxFrac*float64(totalActive) {
        return nil, fmt.Errorf("plan deletes %d of %d prefixes, exceeds limit, needs human approval",
            len(plan), totalActive)
    }
    return plan, nil
}

With either check in place, the February 20 run would have stopped before its first deletion: the returned prefixes were not marked pending_delete, and there were far more of them than any normal cleanup run would handle.

4. A rate-and-volume circuit breaker on withdrawals

This is Cloudflare's third commitment, and it belongs in the component that performs the withdrawal, not in each caller. Every path to "withdraw a prefix" (task, API, dashboard, operator CLI) then passes through one guard.

// Sliding-window breaker: trips if too many withdrawals happen too fast.
// Once open, it stays open until a human resets it.
type WithdrawalBreaker struct {
    mu        sync.Mutex
    window    time.Duration
    maxInWin  int
    events    []time.Time
    open      bool
}

func (b *WithdrawalBreaker) Allow(now time.Time) error {
    b.mu.Lock()
    defer b.mu.Unlock()
    if b.open {
        return errors.New("withdrawal breaker open: manual reset required")
    }
    cutoff := now.Add(-b.window)
    kept := b.events[:0]
    for _, t := range b.events {
        if t.After(cutoff) {
            kept = append(kept, t)
        }
    }
    b.events = kept
    if len(b.events) >= b.maxInWin {
        b.open = true
        alertOncall("withdrawal breaker tripped", len(b.events), b.window)
        return errors.New("withdrawal rate limit exceeded")
    }
    b.events = append(b.events, now)
    return nil
}

A breaker like this needs per-region or per-fleet state in a real deployment, and possibly a separate limit for deletions as well as withdrawals. The Circuit Breaker Pattern page covers the general idea. Here it protects a dependency from you rather than you from a dependency.

5. Prefer soft-delete and separate "withdraw" from "destroy"

The difference between a 20-minute and a 6-hour outage was cascade deletion. Make destruction a two-step process with time in between:

-- Step 1 (automated): mark and withdraw. Fully reversible.
UPDATE prefixes
SET    advertised = false,
       state = 'withdrawn_pending_destroy',
       destroy_after = now() + interval '7 days'
WHERE  id = $1
  AND  state = 'pending_delete';   -- predicate in the write, not only in the read

-- Step 2 (separate job, days later): only then remove bindings and rows.
DELETE FROM service_bindings
WHERE  prefix_id IN (
  SELECT id FROM prefixes
  WHERE  state = 'withdrawn_pending_destroy' AND destroy_after < now()
);

Note the AND state = 'pending_delete' in step 1. Putting the safety predicate in the write means a bad read cannot widen the write.

6. Snapshot operational state and roll it out like code

Cloudflare's second commitment is to stop treating the Addressing API's live database as the thing the edge reads directly. Instead, produce a versioned snapshot of the desired state and push it through the same staged, health-checked pipeline used for software. Rollback then means "redeploy snapshot N-1", which is fast and well-practised.

flowchart LR
    API["Addressing API<br/>(configured state)"] --> Snap["Versioned snapshot N"]
    Snap --> Diff{"Diff vs N-1:<br/>withdrawals > limit?"}
    Diff -->|"yes"| Hold["Hold, page a human"]
    Diff -->|"no"| Canary["Canary PoPs"]
    Canary --> Health{"Customer health OK?"}
    Health -->|"no"| Roll["Auto-rollback to N-1"]
    Health -->|"yes"| Wave["Staged waves to all PoPs"]
    Wave --> Health2{"Health OK?"}
    Health2 -->|"no"| Roll
    Health2 -->|"yes"| Done["Snapshot N live"]

This is the same reasoning as canary deployments and GitOps: configuration and data that change production behaviour should flow through a pipeline with gates and a one-step rollback. Cloudflare is explicit about the cost: during an outage, direct API changes may have to wait for the pipeline.

7. Alert on the thing that matters: announced prefixes

Detection came 17 minutes late, from a symptom. A direct signal was available.

# Prometheus alert rules (illustrative metric names)
groups:
  - name: prefix-advertisement
    rules:
      - alert: AnnouncedPrefixesDropped
        expr: |
          (sum(bgp_announced_prefixes{type="byoip"})
             / sum(bgp_announced_prefixes{type="byoip"} offset 15m)) < 0.98
        for: 2m
        labels: { severity: page }
        annotations:
          summary: "BYOIP announced prefixes fell >2% in 15 minutes"

      - alert: WithdrawalRateHigh
        expr: sum(rate(prefix_withdrawals_total[5m])) * 300 > 10
        for: 1m
        labels: { severity: page }
        annotations:
          summary: "More than 10 prefix withdrawals in 5 minutes"

Pair internal metrics with external checks: BGP collectors such as RIPE RIS or RouteViews, or a synthetic probe to a canary address inside each customer prefix. The external view catches failures that internal state does not report.

8. Ship destructive automation dark first

A new job that deletes things should run in dry-run mode in production for a while, logging what it would have done. Behind a feature flag, it can then be enabled for one prefix, then a few, then all.

if cfg.CleanupDryRun {
    log.Info("cleanup dry-run", "would_delete", len(plan), "cidrs", cidrs(plan))
    metrics.CleanupWouldDelete.Set(float64(len(plan)))
    return nil
}

On February 20 a dry-run log line saying "would delete 4,306 prefixes" would have been an unmissable signal at zero cost.

Lessons for System Design

1. A read API's default is a safety property. "No filter means all rows" is a sensible default for a dashboard and a dangerous one for a job that deletes what it reads. When one endpoint serves both kinds of caller, design the default for the most dangerous one, or split the endpoint.

2. Absent, empty and false are three different values. Many frameworks collapse them. Query strings, environment variables, YAML (key: with no value), JSON null and form fields all have this ambiguity. Decide explicitly what each means at every trust boundary, even internal ones.

3. Put the predicate in the write. DELETE ... WHERE state = 'pending_delete' cannot delete live rows, whatever list the caller passes. A safety check performed only in a preceding read is a check the write does not enforce.

4. Limit blast radius where the effect happens. One guard in the withdrawal path protects every caller: today's task, tomorrow's task, and an operator at 3 AM. Per-caller guards will be forgotten by the next caller. This is the same argument as cell-style isolation for fleet-wide pushes.

5. Cheap-to-undo first, expensive-to-undo later. Withdraw, then destroy days later. Disable, then delete. Every system with cascade deletes should ask how it would rebuild what the cascade removed, and how long that would take.

6. Immediate global propagation is a design choice with a cost. A source of truth that writes straight to every edge machine is fast in normal operation and has no brakes in an incident. Staged rollout of data deserves the same care as staged rollout of code. Cloudflare learned this from configuration files in 2019. This incident shows that API-driven state needs it too.

7. Reliability work is change, and change carries risk. This was a reliability improvement that caused an outage. That is not an argument against automation, since the manual process carried its own risks. It is an argument for treating "make it safer" changes with the same rollout discipline as any other. As Lorin Hochstein put it, cleanup is inherently hazardous, but "not cleaning up is also risky" (Surfing Complexity).

8. Test the autonomous path, with production-shaped data. The tests covered the human-initiated flow. The automation was the new risk and it went untested. A property-style test along the lines of "the cleanup task never deletes a prefix not marked pending" would have caught this, whatever the API returned.

9. Detect from the system's own view of itself. The Addressing API knew how many prefixes it had withdrawn. Alerting on the rate of destructive operations is cheap and catches this whole class of bug.

What Would You Do?

Scenario 1: You are on call. At 18:15 you learn that a public site is returning 403s, and a teammate notices that BYOIP prefix announcements are falling. You do not yet know why. A new cleanup task shipped 30 minutes ago. Do you (A) investigate the 403s on the affected site first, (B) disable the newest change that touches prefix state, even though you have not proven it is the cause, or (C) start re-announcing withdrawn prefixes by hand?

Best answer: (B). When state is actively being destroyed, stop the most likely writer first and diagnose second. A recent deploy to the system that owns the damaged state is the prime suspect, and disabling a cleanup task is low-risk because nothing breaks if cleanup waits a day. (C) loses a race against a task that is still running: you re-announce and it withdraws again. (A) treats a symptom. In the real incident the task ran until 18:46, 50 minutes after it started, and every extra minute meant more prefixes in the expensive "bindings deleted" state.

Scenario 2: You are reviewing the pull request for the cleanup task. The tests mock the list API to return three prefixes marked pending deletion, and the task deletes all three. The author says it is covered. What do you ask for?

Ask what happens when the API returns something unexpected. Add tests where the mock returns (1) prefixes that are not marked pending, (2) a list far larger than normal, and (3) an empty list. The task should refuse (1) and (2) and do nothing on (3). Ask for a dry-run mode and a per-run deletion cap. Ask for the client to use a typed generated client or an explicit =true. Most importantly, ask that the deletion SQL itself carry the state = 'pending_delete' predicate. A test that only mocks the happy path checks the code the author imagined, not the contract the server implements.

Scenario 3: After the incident, leadership asks for one change that would most reduce the impact of the next bug of this kind, whatever its trigger. Options: (A) strict schema validation on all internal APIs, (B) a circuit breaker on prefix withdrawals and deletions, enforced in the component that applies them, (C) more end-to-end tests in staging.

Best single answer: (B). Validation (A) prevents this bug, and you should do it, but the next bug may be a wrong SQL predicate, an operator script or a different caller. A breaker in the component that applies withdrawals bounds the damage from all of them. Staging tests (C) help, but Cloudflare's report says its staging data did not represent production, which is a common and persistent problem. Bounding blast radius is the control that still works when the bug is one you did not predict.

Key Lessons

  • A missing =true withdrew about 1,100 prefixes. The server treated an empty flag as absent and returned everything, and the deleting caller trusted the result.
  • Fail closed on filters that feed deletes. An unrecognised filter must produce an error or an empty list, never "all rows".
  • Put the safety predicate in the write, and have destructive jobs check their own input and refuse implausibly large batches.
  • Breakers belong where the effect happens. One guard on withdrawals protects every caller.
  • Deletion turned a quick fix into a six-hour one. Withdrawn prefixes came back with a dashboard toggle. Deleted bindings needed a global fleet rollout.
  • Roll out data like code. Snapshots, health-gated waves and one-step rollback are what Cloudflare committed to build.
  • Reliability changes are still changes. The task was part of Code Orange: Fail Small.

Quiz

Quiz

Q1: What exactly was wrong with the request GET /v1/prefixes?pending_delete? The flag was present with no value. The server only applied the "pending deletion" filter when the value was non-empty. An empty value looked the same as an absent parameter, so the server took its default branch and returned every BYOIP prefix. The client intended pending_delete=true.

Q2: Why did some prefixes recover around 19:19 while others needed until 23:03? The task was stopped part-way through an iterative run, leaving prefixes in three states. Prefixes that were only withdrawn could be re-advertised by customers from the dashboard. Prefixes with some bindings removed needed database recovery and a release. About 300 prefixes with all bindings removed needed a global configuration rollout to every edge machine.

Q3: What is BGP path hunting, and why does it make outages look like hangs instead of errors? When a prefix is withdrawn, routers do not learn at once that it is gone. They fall back to alternative paths learned from other neighbours, which are themselves withdrawn as the change propagates. Packets are forwarded along paths that are about to disappear, so connections time out instead of failing fast with "no route to host".

Q4: Name the three remediation commitments in Cloudflare's report and the layer each protects. (1) Standardised API schemas with strictly typed flags, which protects the interface by rejecting malformed calls. (2) Separating operational from configured state, with snapshots, health-mediated rollouts and fast rollback, which protects recovery. (3) Circuit breakers on withdrawals or deletions that are too fast or too broad, which bounds the blast radius.

Q5: Why is "put the predicate in the write" stronger than checking the list before deleting? A check done only on a preceding read depends on that read being correct, and here the read was wrong. If the delete statement itself says WHERE state = 'pending_delete', it cannot remove live rows whatever list the caller passes in. The guarantee then sits where the damage would happen.

Q6: Why was this incident's root cause not an unscoped query? It looks like a DELETE with no WHERE clause, but the scope existed: the server had a filtered branch. The failure was in how the flag was encoded and parsed. Client and server disagreed about what a valueless flag meant, and the disagreement resolved to the most dangerous interpretation.

One-Liner Summary

A reliability-motivated cleanup job sent a boolean flag without a value, the API read it as "no filter" and returned every BYOIP prefix, and the job withdrew about 1,100 of them from the Internet. Stopping it took 50 minutes, and rebuilding what it had deleted took more than four hours after that.

Sources