On this page

Google Cloud europe-west4-a Power and Cooling Failure (July 2026)

On the evening of 15 July 2026 (Pacific time), which was the early morning of 16 July in the Netherlands, the power feed into a data center serving Google Cloud's europe-west4-a zone dipped for 3 milliseconds. That dip should have been routine. A data center facility is built to ride through it: utility breakers trip, rotary UPS units with diesel engines pick up the load, and the chillers keep running.

That night, one of the two backup power systems failed to take the load. One row of racks lost both of its feeds. The chiller controller dropped offline in the same event, the cooling pumps did not restart, and the redundant cooling source was unavailable because of construction work at the site. Within two hours the data hall reached 44°C and Google began shutting machines down to stop them cooking.

Three products went dark: Google Cloud VMware Engine (GCVE), Google Cloud NetApp Volumes (GCNV) and Bare Metal Solution (BMS). According to Google's full incident report, total disruption lasted 14 hours and 55 minutes. Most other Google Cloud services in the region were not named in the incident.

This case study is not about a dramatic bug. It is about correlated failures in physical infrastructure, redundancy that quietly was not there, and a product-design fact many cloud customers never think about: some cloud services live in one zone, sometimes one building, and the zonal risk is yours to carry.

Sourcing note

Almost everything below comes from Google's own incident report on the Google Cloud Service Health page (preliminary report 17 July, full report 25 July 2026), with context from The Register, W.Media and DCD. Google's report gives times in US/Pacific (PDT, UTC−7). This page shows PDT and UTC side by side. Where sources disagree on numbers, the page says so instead of picking one silently.

TL;DR

  • What: A utility-side voltage transient tripped both utility feeds into a data center in europe-west4-a. Feed B's diesel rotary UPS (DRUPS) took over cleanly. Feed A's DRUPS failed because of electrical component failures. Rows 1 and 2 moved to feed B, but Row 3 hit an overload breaker trip (Google attributes it to a "load deployment discrepancy") and lost all power.
  • Cooling: The chiller controller went offline during the same transient, so the chilled-water distribution pumps did not restart and chiller A shut down. The redundant cooling source was unavailable because of ongoing construction at the facility. The hall reached 44°C.
  • Impact: GCVE (24 private clouds, 20 customers), BMS (9 customers) and NetApp Volumes (all 6 clusters in the zone; Standard, Premium and Extreme tiers) were unavailable for between roughly 8 and 13 hours each. Total disruption: 14 h 55 min.
  • Detection: Automated telemetry alerted at 16:30–16:39 PDT for power. The temperature alert came at 17:44 PDT, more than an hour after the event began.
  • Response: Pumps switched to hand mode, a portable UPS was put on the chiller controllers, a swing DRUPS restored feed-A redundancy, Row 3 breakers were reset, and then a controlled restart: network fabric, then storage, then hosts.
  • Lessons: N+1 redundancy that is under construction is N. Power and cooling share failure modes. Products deployed in a single zone need you to design the second zone. Graceful thermal shutdown protects data but needs automation and staffing that is ready when the hall heats fast.

The Alert

At 16:30 PDT (23:30 UTC) on 15 July, GCVE engineers received a power alert. At 16:32 the NetApp Volumes team got a high-temperature and switch failure alert. By 16:41, GCVE and BMS servers were reporting power feed redundancy failure: a server with two power supplies that has lost one of them.

On its own, "lost one feed" is not an emergency. Every enterprise server in a tier-III-style facility has two power supplies fed from independent A and B paths for exactly this reason. What turned the alert into an incident was what happened next: the temperature kept climbing.

The quiet part

At 17:05 PDT GCVE's network switches were already above 60°C. Google's own preventive action list includes "add earlier alerts for power and cooling events," which is an admission that the cooling signal arrived later than it should have. Engineers were alerted to the temperature rise at 17:44 PDT, about 80 minutes after the grid fault.

Impact

ProductCustomer impactImpact window (PDT, preliminary report)Duration (preliminary report)
Google Cloud NetApp VolumesStandard, Premium and Extreme volumes inaccessible. Control-plane operations (creating storage pools, volumes, backups) failed15 Jul 16:39 – 16 Jul 01:108 h 31 min
Google Cloud VMware EngineLost connectivity to private clouds and their workloads. 24 private clouds, 20 customers15 Jul 17:09 – 16 Jul 02:339 h 24 min
Bare Metal SolutionDatabase servers and storage appliances unreachable. 9 customers15 Jul 18:37 – 16 Jul 07:3412 h 57 min

The full report frames the overall incident as 15 Jul 16:39 to 16 Jul 07:34 PDT, which is the 14 h 55 min figure (Google incident report).

Where the numbers disagree

Different sources, and even different parts of Google's own status site, give different durations. None of these are invented; they are just measured differently.

ItemValue AValue BNotes
Overall start16:24 PDT (grid fault, full report data-center timeline)16:39 PDT (first customer impact, full report)The status page header shows 16:57 PDT as incident start.
BMS duration12 h 57 min (preliminary report table)12 h 28 min (status summary list)W.Media says "nearly 13 hours".
GCVE duration9 h 24 min (preliminary report)9 h 34 min (status summary list)
NetApp Volumes duration8 h 31 min (preliminary report)8 h 03 min (status summary list)
BMS end07:34 PDT (impact end in the report)08:42 PDT ("mitigated" in the BMS timeline)The report's own BMS timeline marks mitigation after its stated impact end.
Total14 h 55 min (Google full report, DCD, W.Media)"about 15 hours" (The Register)Consistent.

The likely explanation is that the summary list measures from the status-page posting time rather than from first impact, but Google does not say so. Treat the preliminary and full report figures as authoritative.

What was not affected. The incident lists only GCVE, NetApp Volumes and BMS. Compute Engine, GKE, Cloud SQL and other mainstream services in europe-west4-a were not named. The Register reports that Google runs these three services from a separate data center that lost power. W.Media notes Google's reference to a "third-party facility provider" and that it is unclear exactly which facility in or around the Eemshaven campus lost power (W.Media). Google did not answer The Register's question about whether it discloses which services depend on a single data center.

Timeline

All times from Google's full incident report unless noted. PDT is UTC−7. Local time in the Netherlands (CEST) is UTC+2.

PDT (15–16 Jul)UTCEventSource
15 Jul 16:2423:24Utility grid fault: 3 ms voltage drop, utility breakers trip on both feeds. Side-A DRUPS fails. Chiller controller goes offlineGoogle full report
16:3023:30GCVE receives power alertGoogle full report
16:3223:32NetApp Volumes receives high-temperature and switch-failure alertGoogle full report
16:3723:37Row 3 fails to transfer and loses all power (overload breaker trip)Google full report
16:3923:39First customer impact (NetApp Volumes). Automated telemetry alerting from this point (preliminary report)Google preliminary and full reports
16:4123:41GCVE and BMS servers report power-feed redundancy failureGoogle full report
17:0500:05 (16 Jul)GCVE network switches exceed 60°CGoogle full report
17:0900:09GCVE customer impact beginsGoogle full report
17:3200:32All 6 NetApp Volumes clusters shut down automatically due to heatGoogle full report
17:4400:44Engineers alerted to the temperature riseGoogle full report
17:5200:52BMS detects rising temperaturesGoogle full report
18:2901:29Data hall reaches 44°CGoogle full report
18:3701:37Four NetApp SAN nodes serving BMS fail. BMS customer impact beginsGoogle full report
18:5901:59BMS reserve and buffer servers begin powering offGoogle full report
19:00–19:5502:00–02:55GCVE devices shut down. Status update says "full shutdown" began 19:00 PDT (04:00 local)Google status updates and full report
19:5502:55Machine shutdown procedures implemented across the hallGoogle full report
20:3003:30Earliest public status update shown on the incident page, advising GCVE customers to shut down workload VMs "so that recovery is safer"Google status page
20:4303:43BMS sees temperature dropGoogle full report
20:5203:52BMS SAN nodes recoveredGoogle full report
21:0504:05Cooling fully recoveredGoogle full report
21:24 / 21:4604:24 / 04:46Facility provider sends cooling, then power, recovery noticesGoogle full report
21:5604:56GCVE network restoration startsGoogle full report
22:0805:08First NetApp Volumes cluster recoveredGoogle full report
22:4005:40GCVE onsite technician arrivesGoogle full report
23:1006:10GCVE console servers recoveredGoogle full report
23:2306:23First BMS customer server rebootGoogle full report
23:5506:55All 6 NetApp Volumes clusters and customers recoveredGoogle full report
16 Jul 00:3307:33GCVE network recovery complete for all placement groupsGoogle full report
02:33–02:3609:33–09:36GCVE mitigatedGoogle preliminary and full reports
05:2512:25Final status update: GCVE and NetApp Volumes resolved, BMS "believed to be affecting a very small number of customers"Google status page
06:3213:32Last BMS customer server reboot startedGoogle full report
07:3414:34End of impact window (overall)Google full report
08:4215:42BMS marked mitigated in its product timelineGoogle full report
17 Jul 17:2218 Jul 00:22Preliminary incident report publishedGoogle status page
25 Jul 06:1613:16Full incident report publishedGoogle status page
sequenceDiagram
    participant Grid as Utility grid
    participant Power as Facility power (A and B)
    participant Cool as Chillers and pumps
    participant Hall as Data hall
    participant Svc as GCVE, GCNV, BMS
    participant Eng as Google engineers

    Grid->>Power: 16:24 PDT — 3 ms voltage drop, both utility breakers trip
    Power->>Power: Side B DRUPS takes load, side A DRUPS fails
    Power->>Hall: 16:37 — Row 3 overload trip, row loses all power
    Grid->>Cool: Chiller controller drops offline, pumps do not restart
    Svc->>Eng: 16:30 to 16:41 — power and redundancy alerts
    Hall->>Svc: 17:05 — switches above 60°C
    Svc->>Svc: 17:32 — NetApp clusters thermal auto-shutdown
    Eng->>Eng: 17:44 — temperature-rise alert reaches engineers
    Hall->>Hall: 18:29 — hall reaches 44°C
    Eng->>Svc: 19:00 to 19:55 — controlled shutdown of remaining devices
    Eng->>Cool: Pumps to hand mode, portable UPS on chiller controllers
    Cool->>Hall: 21:05 — cooling fully recovered
    Eng->>Svc: 21:56 onwards — network, then storage, then hosts
    Svc->>Eng: 16 Jul 07:34 — impact ends, BMS last to recover

Root Cause

Google's full report names the root cause as "a 3ms voltage drop in the power feed from the utility provider and subsequent utility breaker protective action." That is the trigger. The reason a 3 ms dip became a 15-hour outage is a chain of four independent weaknesses that lined up.

1. The power path: A/B feeds with rotary UPS

A typical hyperscale-adjacent facility feeds each row of racks from two independent electrical paths, A and B. Each server has two power supplies, one on each path. Each path has its own utility connection and its own backup.

The backup here was a DRUPS (Diesel Rotary Uninterruptible Power Supply). A DRUPS couples a motor-generator and a flywheel to a diesel engine. When utility power drops, the flywheel's stored kinetic energy keeps the output alive for the seconds it takes the diesel engine to start and clutch in. No battery room is involved. When it works, the load sees nothing.

flowchart TB
    U["Utility grid"] --> BA["Utility breaker A"]
    U --> BB["Utility breaker B"]
    BA --> DA["DRUPS A (flywheel + diesel)"]
    BB --> DB["DRUPS B (flywheel + diesel)"]
    DA --> PA["Distribution A"]
    DB --> PB["Distribution B"]
    PA --> R1["Row 1"]
    PA --> R2["Row 2"]
    PA --> R3["Row 3"]
    PB --> R1
    PB --> R2
    PB --> R3
    U --> CC["Chiller controller"]
    CC --> PU["Chilled-water pumps"]
    PU --> CA["Chiller A"]
    CB["Redundant cooling source"] -.->|"unavailable: construction"| PU
    CA --> HALL["Data hall air"]

2. Side A's DRUPS failed

On 15 July, the voltage transient tripped the utility breakers on both feeds. That is expected behaviour: protective breakers open on a disturbance and the DRUPS take over.

  • Side B: "The transition of side B to DRUPS system was successful without any power interruption" (Google, quoted by The Register).
  • Side A: "The DRUPS back-up power system for side A failed to take over the facility load due to electrical component failures."

Google has not published which components failed. One of the preventive actions due in August 2026 is to investigate why the load did not transfer; as of this page's review date we found no public follow-up.

3. Row 3 could not survive on one feed

With side A gone, every row should have run on side B alone. That is the whole point of dual feeds. Rows 1 and 2 did. Row 3 did not. Its side-B breaker tripped on overload, and the row lost both feeds. Google attributes this to a "load deployment discrepancy" and said it was still investigating.

The report does not explain the term. The usual meaning in data-center operations is that the equipment actually installed on a circuit draws more than the circuit was planned for. In a dual-fed design, each feed must be able to carry the whole row's load alone, which means each feed should normally run at no more than about 50% of its rating. If a row has crept above that (more racks, denser servers, a planning record that does not match reality), everything looks fine every day, because both feeds share the load. The problem is invisible until one feed fails and the survivor has to carry 100%.

The latent-failure pattern

A dual-fed row loaded at 60% per feed shows no symptoms in normal operation. It fails only at the exact moment you need the redundancy. This is the same class of bug as a database replica that cannot actually keep up with primary write volume: it reports healthy until failover, then falls over. Note: Google has not confirmed that this was the mechanism; it is the textbook reading of the phrase.

The Register's summary of the full report quotes the outcome: "This resulted in loss of both redundant power feeds to the single row."

4. Cooling failed in the same event

Losing one row of power is bad but bounded. What made it hall-wide was cooling.

  • "The chiller controller dropped offline during the voltage transient event." The controller was apparently not on protected power, or its protection did not ride through the event.
  • Because the controller was offline, the chilled-water distribution pumps never received the restart signal, and chiller system A shut down.
  • "The redundant source was not available due to known ongoing construction work at the facility."

So the cooling system had redundancy on paper, but on this night it was a single point of failure, and that single point was controlled by a device that shared the triggering event.

flowchart TD
    T["Utility voltage transient (3 ms)"] --> B["Utility breakers trip on A and B"]
    B --> DA{"Side A DRUPS takes load?"}
    DA -->|"no: component failures"| LA["Feed A lost"]
    B --> DB{"Side B DRUPS takes load?"}
    DB -->|"yes"| OKB["Feed B carries rows 1 to 3"]
    LA --> R3{"Row 3 fits on feed B alone?"}
    OKB --> R3
    R3 -->|"no: overload trip"| R3X["Row 3 loses all power"]
    T --> CC["Chiller controller offline"]
    CC --> PUMP["Pumps do not restart, chiller A stops"]
    PUMP --> RED{"Redundant cooling available?"}
    RED -->|"no: construction"| HOT["Hall heats to 44°C"]
    HOT --> SHUT["Thermal auto-shutdowns + manual shutdown"]
    R3X --> OUT["GCVE, NetApp Volumes, BMS outage"]
    SHUT --> OUT

Why heat, not power, set the duration

Row 3 lost power at 16:37. But the NetApp clusters shut themselves down at 17:32 due to heat, and the BMS SAN nodes failed at 18:37, again in a hot hall. Cooling was not fully back until 21:05 PDT, and only then could the controlled restart begin. Then recovery itself took hours: the last BMS customer server reboot did not start until 06:32 PDT the next morning.

This is the general shape of a data-center thermal event:

  1. Minutes: temperature rises fast once airflow or chilled water stops. Modern dense racks have very little thermal mass.
  2. Tens of minutes: devices hit thermal limits and either throttle, shut themselves down uncleanly, or must be shut down by operators.
  3. Hours: cooling has to be restored and the hall brought back to safe temperature before anything restarts.
  4. Many hours: a careful restart, in dependency order, with validation, because a hard thermal shutdown can leave storage in states that need checking.

Detection

What worked. Power alerts reached engineers within six minutes of the grid fault (16:30 PDT). The NetApp clusters had automatic thermal protection and shut down at 17:32 rather than running until damaged.

What did not.

  • The temperature-rise alert reached engineers at 17:44, about 80 minutes after the controller dropped offline and nearly 40 minutes after switches crossed 60°C. Google's preventive action list includes "add earlier alerts for power and cooling events."
  • The signals were split across teams. The power alert went to GCVE, the temperature and switch alert went to NetApp, and BMS saw a redundancy failure. Three product teams each saw part of one facility event.
  • The public status page lagged. The earliest update visible on the incident page is 20:30 PDT, nearly four hours after first impact. (We cannot rule out that earlier updates were edited or merged, but the page as published starts there.)
flowchart LR
    F["Facility BMS/SCADA telemetry"] -->|"power events"| P1["GCVE power alert 16:30"]
    F -->|"temperature"| P2["NetApp temp alert 16:32"]
    F -->|"PSU redundancy"| P3["GCVE + BMS 16:41"]
    F -->|"hall temperature"| P4["Engineer temp alert 17:44"]
    P1 --> IC["Incident commander"]
    P2 --> IC
    P3 --> IC
    P4 --> IC
    IC --> SP["Public status page 20:30"]

The detection lesson for everyone

A single physical root cause showed up as three different symptoms on three products. If your alerting routes by product, you need correlation by location too: "every alert in building X in the last 15 minutes" is often the fastest way to see a facility event.

Response and Recovery

The full report lists these mitigation steps:

  1. Pumps switched to hand mode to restore chilled-water circulation without waiting for the controller.
  2. A portable UPS was connected to the chiller controllers, so a repeat voltage dip could not knock them out again.
  3. Two remote power panels needed manual verification before power could be restored to the affected row.
  4. Row 3's tripped breakers were reset once the fault was isolated.
  5. A swing DRUPS unit restored redundancy on feed A while the failed DRUPS was repaired.
  6. Controlled restart by layer. For GCVE: network restoration started at 21:56, an onsite technician arrived at 22:40, console servers were recovered at 23:10, and network recovery for all placement groups completed at 00:33. NetApp clusters came back one by one from 22:08 to 23:55. BMS servers were rebooted customer by customer from 23:23 to 06:32.

Google also told GCVE customers, in a status update, to shut down their VMware workload VMs "for precaution so that recovery is safer." That is good advice for any customer during a thermal event: a VM shut down cleanly by its owner is easier to bring back than one that lost its storage mid-write.

flowchart LR
    C["Cooling restored (21:05)"] --> N["Network fabric (21:56 to 00:33)"]
    C --> S["Storage clusters (22:08 to 23:55)"]
    N --> H["Hosts and private clouds"]
    S --> H
    H --> B["BMS servers, one by one (23:23 to 06:32)"]
    B --> V["Validation and mitigation"]

Preventive actions Google committed to

From the full report (Google):

ActionOwner areaTarget
Investigate why load did not transfer to DRUPSFacilityAug 2026
Resolve the cause of the Row 3 overloadFacilityAug 2026
Review chiller pump control redundancyFacilitySep 2026
Add earlier alerts for power and cooling eventsFacility / monitoringAug 2026
Improve automated shutdown during fast thermal runawayGCVEOct 2026
Update incident classification and SLOs so staff are available during major eventsBMSSep 2026
Review SLAs and runbooks, run monthly joint drills, look at faster cluster recoveryNetApp VolumesAug 2026

We did not find a public follow-up confirming completion of these actions as of 9 November 2026.

What Would Have Prevented It

Split this into two halves: what the facility operator controls, and what you, the customer, control. Most readers are in the second group, and that half is where the actionable lessons are.

Facility-side controls

These are standard data-center engineering practices. We list them because each one maps directly to a link in the failure chain.

Weak linkControl
Side-A DRUPS failed to pick up loadRegular integrated systems tests (pull-the-plug tests that open the utility breaker under real load), not just engine-start tests. Component-level maintenance on transfer gear
Row 3 overloaded on a single feedPer-feed load cap (commonly ≤ 50% of breaker rating on each side of a 2N row, with margin), continuous branch-circuit metering, and capacity planning that blocks new deployments that break the cap
Chiller controller dropped offlinePut cooling controls on the same protected (UPS-backed) power as IT load. Google's own fix was a portable UPS on the controllers
Pumps did not restartFail-safe pump logic: pumps default to running on loss of controller signal
Redundant cooling under constructionTreat construction or maintenance that removes redundancy as a declared risk state: extra staffing, pre-positioned temporary cooling, and a freeze on risky work elsewhere until redundancy is back
Temperature alert at 17:44Rate-of-change alerts on hall and inlet temperature, not just absolute thresholds

A rate-of-change alert is simple and catches thermal runaway long before an absolute threshold does. In Prometheus terms (the same idea works in any monitoring stack):

# Alert on fast temperature rise, not just on a high absolute value.
groups:
  - name: facility-thermal
    rules:
      - alert: DataHallTemperatureRisingFast
        # More than 3°C rise in 10 minutes at any inlet sensor in the hall
        expr: delta(dc_inlet_temperature_celsius[10m]) > 3
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "Inlet temperature rising fast in {{ $labels.hall }}"
      - alert: CoolingControllerUnreachable
        expr: up{job="chiller-controller"} == 0
        for: 1m
        labels:
          severity: page
      - alert: RowSingleFeedOverCapacity
        # Each feed in a 2N row must be able to carry the whole row alone.
        expr: |
          (sum by (row) (pdu_feed_power_watts))
            / on(row) group_left min by (row) (pdu_feed_rating_watts) > 0.8
        for: 15m
        labels:
          severity: ticket
        annotations:
          summary: "Row {{ $labels.row }} could not survive on one feed"

The third rule is the one that would have caught the latent Row 3 problem weeks earlier: it asks whether the total row load fits on the smaller feed, which is the only question that matters when one feed fails.

Customer-side: you own the zonal risk

Google's region documentation is explicit that a zone is a failure domain and that you should spread workloads across zones for high availability (Google Cloud regions and zones). Many Google Cloud services do that for you: regional managed instance groups, regional GKE clusters, Cloud SQL high availability, regional Persistent Disk, multi-region Cloud Storage.

But GCVE, BMS and most NetApp Volumes service levels are zonal products. A GCVE private cloud lives in one zone. A BMS server is a physical machine in one facility. A Standard/Premium/Extreme NetApp Volumes pool sits in one zone. When the zone goes, they go, and no amount of in-zone redundancy (more ESXi hosts, a bigger cluster) helps.

flowchart TB
    subgraph REG["Region europe-west4"]
        subgraph ZA["Zone a"]
            GCVE["GCVE private cloud"]
            BMS["BMS servers"]
            NV["NetApp Volumes pool"]
            MIGA["MIG instances"]
        end
        subgraph ZB["Zone b"]
            MIGB["MIG instances"]
        end
        subgraph ZC["Zone c"]
            MIGC["MIG instances"]
        end
    end
    LB["Regional load balancer"] --> MIGA
    LB --> MIGB
    LB --> MIGC
    APP["Your VMware workload"] --> GCVE
    DB["Your Oracle on BMS"] --> BMS

In this diagram, the managed instance group survives the loss of zone a. The GCVE workload, the BMS database and the NetApp volume do not, unless you built their second copy yourself.

Control 1: Regional by default for things Google can spread

For ordinary compute, make the regional option the default in your Terraform modules so nobody accidentally ships a zonal workload.

# Regional managed instance group: instances spread over three zones,
# and the autohealer recreates them in surviving zones if one zone fails.
resource "google_compute_region_instance_group_manager" "web" {
  name               = "web-rmig"
  region             = "europe-west4"
  base_instance_name = "web"

  distribution_policy_zones = [
    "europe-west4-a",
    "europe-west4-b",
    "europe-west4-c",
  ]
  # EVEN keeps capacity balanced, so losing one zone loses about a third.
  distribution_policy_target_shape = "EVEN"

  version {
    instance_template = google_compute_instance_template.web.id
  }

  # Size for N-1 zones: if you need 6 instances to serve peak,
  # run 9 so that 6 remain when a zone disappears.
  target_size = 9

  auto_healing_policies {
    health_check      = google_compute_health_check.web.id
    initial_delay_sec = 120
  }

  update_policy {
    type                  = "PROACTIVE"
    minimal_action        = "REPLACE"
    max_surge_fixed       = 3
    max_unavailable_fixed = 0
  }
}

# Regional persistent disk: synchronously replicated across two zones.
resource "google_compute_region_disk" "data" {
  name          = "data-rpd"
  region        = "europe-west4"
  replica_zones = ["europe-west4-a", "europe-west4-b"]
  type          = "pd-balanced"
  size          = 500
}

# Cloud SQL with high availability: standby in another zone, automatic failover.
resource "google_sql_database_instance" "main" {
  name             = "main-pg"
  region           = "europe-west4"
  database_version = "POSTGRES_16"

  settings {
    tier              = "db-custom-4-16384"
    availability_type = "REGIONAL"
    backup_configuration {
      enabled                        = true
      point_in_time_recovery_enabled = true
    }
  }
}

Note the N-1 sizing. Spreading over three zones does nothing if the two surviving zones cannot handle the full load. See Redundancy and Replication for the arithmetic.

For Kubernetes, use a regional cluster and force replicas across zones:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 6
  selector:
    matchLabels: { app: api }
  template:
    metadata:
      labels: { app: api }
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels: { app: api }
      containers:
        - name: api
          image: europe-west4-docker.pkg.dev/acme/api/api:1.42.0
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  minAvailable: 4
  selector:
    matchLabels: { app: api }

Control 2: Build the second zone yourself for zonal products

For products like the three that failed, the provider gives you building blocks and you assemble the redundancy.

VMware Engine. Options, roughly from cheapest to strongest:

  • Second private cloud in another zone or region, with VM-level replication (for example VMware HCX or a third-party DR tool) and a tested runbook to fail over. Accept an RPO of minutes and an RTO of an hour or more.
  • Stretched private cloud across two zones where Google offers it, which gives vSAN-level synchronous replication between zones. Check current Google documentation for regional availability and constraints. Note that a separate 14 July 2026 status entry reports GCVE stretched cluster customers experiencing zonal network connectivity problems (status summary), so stretched does not mean immune: test it.
  • Move stateless tiers off GCVE to regional compute and keep only what truly needs VMware there.

Bare Metal Solution. A BMS server is one machine in one place. Typical database HA on BMS is application-level:

  • Oracle Data Guard (or the equivalent for your database) with a standby in a different zone or region, and Fast-Start Failover or a scripted switchover.
  • Keep backups outside the zone (Cloud Storage dual-region or multi-region bucket).
  • Measure the real RTO: how long does it take your team to promote the standby and repoint the application at 2 a.m.?

NetApp Volumes. Google's status update said only the Standard, Premium and Extreme service levels were affected. Google documents the Flex service level as supporting regional storage pools that span two zones, and NetApp Volumes supports volume replication to another region. Check the current documentation for which features apply to your service level before relying on them.

Control 3: Prove it with a zone-failure game day

Architecture diagrams lie. The only way to know you survive a zone loss is to remove a zone and watch. A simple, reversible GKE drill:

#!/usr/bin/env bash
# Zone-loss drill for a regional GKE cluster.
# Cordons and drains every node in one zone, then checks the service still answers.
# Run in a staging cluster first. Reverse with the uncordon command at the end.
set -euo pipefail

ZONE="${1:-europe-west4-a}"
URL="${2:-https://staging.example.com/healthz}"

echo "Cordoning all nodes in $ZONE"
kubectl get nodes -l "topology.kubernetes.io/zone=$ZONE" -o name \
  | xargs -r kubectl cordon

echo "Draining nodes in $ZONE"
kubectl get nodes -l "topology.kubernetes.io/zone=$ZONE" -o name \
  | xargs -r -n1 kubectl drain --ignore-daemonsets --delete-emptydir-data --timeout=10m

echo "Probing $URL for 10 minutes"
fail=0
for i in $(seq 1 120); do
  code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 "$URL" || echo 000)
  [ "$code" = "200" ] || { fail=$((fail+1)); echo "probe $i -> $code"; }
  sleep 5
done
echo "Failed probes: $fail of 120"

echo "To restore: kubectl get nodes -l topology.kubernetes.io/zone=$ZONE -o name | xargs kubectl uncordon"

For zonal products the drill is a DR failover exercise: shut down the primary VMs or database in zone a, promote the standby elsewhere, measure RTO and RPO, and fail back. Do it on a schedule. Chaos Engineering covers how to run this safely.

Control 4: Audit what is still zonal

You cannot fix single-zone dependencies you do not know about. A small script that lists zonal resources is worth more than a slide that says "we are multi-zone."

#!/usr/bin/env python3
"""List Compute Engine instances and disks per zone in one region.

Flags any instance group or label-defined service whose instances
all sit in a single zone. Uses the gcloud CLI's JSON output so there
are no extra dependencies.
"""
import collections
import json
import subprocess
import sys


def gcloud_json(*args):
    out = subprocess.run(
        ["gcloud", *args, "--format=json"],
        check=True, capture_output=True, text=True,
    ).stdout
    return json.loads(out)


def zones_by_service(instances, label="service"):
    zones = collections.defaultdict(set)
    for inst in instances:
        svc = inst.get("labels", {}).get(label, "<unlabelled>")
        zones[svc].add(inst["zone"].rsplit("/", 1)[-1])
    return zones


def single_zone(zones_map):
    return sorted(svc for svc, zs in zones_map.items() if len(zs) < 2)


def main(region):
    instances = gcloud_json(
        "compute", "instances", "list", f"--filter=zone~^{region}-"
    )
    zones = zones_by_service(instances)
    for svc, zs in sorted(zones.items()):
        print(f"{svc:30} {sorted(zs)}")
    bad = single_zone(zones)
    if bad:
        print("\nSINGLE-ZONE services:", ", ".join(bad))
        return 1
    return 0


if __name__ == "__main__":
    # Self-check of the pure logic, no gcloud needed: python zonal_audit.py --test
    if sys.argv[1:] == ["--test"]:
        fake = [
            {"zone": "z/europe-west4-a", "labels": {"service": "api"}},
            {"zone": "z/europe-west4-b", "labels": {"service": "api"}},
            {"zone": "z/europe-west4-a", "labels": {"service": "db"}},
        ]
        assert single_zone(zones_by_service(fake)) == ["db"]
        print("ok")
        sys.exit(0)
    sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else "europe-west4"))

Extend the same idea to the zonal products you use (GCVE private clouds, NetApp storage pools, BMS servers) by keeping them in an inventory with a zone and second_copy field, and failing the review if second_copy is empty for anything tier-1.

Control 5: Graceful shutdown on thermal events

Google's GCVE team committed to "improve automated shutdown during fast thermal runaway." Customers can do their part: when the provider announces a thermal or power event in your zone, shut your own workloads down cleanly (databases first flush and stop, then app VMs) instead of waiting for the hardware to cut power. That turns an unclean crash into a clean restart and shortens your own recovery. Wire Google Cloud Service Health notifications into your paging system so you see the announcement at the same time Google posts it.

Lessons for System Design

1. Redundancy you cannot use is not redundancy

The facility had two power feeds, two DRUPS units and a redundant cooling source. On the night, one DRUPS failed, one row could not run on its surviving feed, and the redundant cooling was taken out by construction. Count the redundancy you actually have right now, not the redundancy in the design document. Planned maintenance, construction and degraded components all reduce N+1 to N, and N is where the next small fault becomes an outage. The AWS S3 2017 outage is a software version of the same idea: subsystems that had not been restarted at scale for years.

2. Redundant paths fail together when they share a trigger

The voltage transient tripped both feeds, killed a DRUPS, and took the chiller controller offline. These were supposedly independent systems with a common upstream event. When you design for failure, ask "what single event touches both copies?" For software, the equivalents are a shared config push, a shared DNS dependency, a shared certificate, or a shared control plane. Facebook October 2021 shows how a single control-plane event removed every redundant path at once.

3. Latent capacity bugs hide behind redundancy

Row 3 ran fine on two feeds and failed on one. Any system where each redundant copy must be able to carry the whole load needs a check that it can: per-feed load caps, database replicas sized for primary write load, a cluster that can absorb a third of its nodes vanishing. Load-test the failover state, not just the steady state. Availability Patterns goes into active-active sizing.

4. A zone is a failure domain, and some products have only one

Cloud documentation tells you to use multiple zones. That advice only works for services that can span zones. Products that pin you to one zone, or even one building, leave the zonal risk on your side of the shared-responsibility line. As Gartner's Adrian Wong told The Register, "It is very hard to figure out how an individual region is architected," and customers "are often surprised by that" (The Register). The practical response: classify every dependency as zonal, regional or multi-regional, and treat zonal ones as needing a second copy you build. See Multi-Region Architecture Patterns and Failover Strategies.

5. Recovery is slower than failure, and ordered

Power was back by 21:46 PDT. The last BMS server reboot started at 06:32 the next morning. Recovery had a dependency order (cooling, then network fabric, then storage, then hosts), needed hands on site (the GCVE technician arrived at 22:40), and was serialised per customer for BMS. Your own recovery plans should model that: write down the order, estimate each step, and multiply the total by the number of things that need restarting. Disaster Recovery covers RTO and RPO planning.

6. Staffing is part of the architecture

Two of the product-level preventive actions are about people, not hardware: BMS will update incident classification "so staff are available during major events," and NetApp Volumes will run monthly joint drills. An incident that starts at 01:24 local time needs an on-call model that can put engineers on site fast. If your RTO assumes a human, it also assumes that human can be reached and can get there.

7. Rate of change beats thresholds for fast failures

Thermal runaway in a dense hall is fast. The difference between a 16:39 alert on "temperature rising fast" and a 17:44 alert on "temperature high" was about an hour of reaction time. The same applies to error rates, queue depth and disk fill: alert on the derivative when the failure accelerates.

What Would You Do?

Scenario 1: You run a customer-facing Oracle database on Bare Metal Solution in europe-west4-a. Your RTO target is 1 hour. What is the minimum architecture that meets it during an event like this one?

A standby database outside zone a (another zone if available for BMS in your region, otherwise another region), kept current with Data Guard or equivalent, plus a tested, scripted switchover and application connection strings that can be repointed quickly (DNS with short TTL or a connection manager). Backups alone do not meet a 1-hour RTO for a large database, and a second server in the same zone fails in the same hot room. Test the switchover quarterly and time it.

Scenario 2: Google posts that your GCVE zone has a cooling failure and recommends shutting down VMs. Your app is still answering. Do you shut down?

Usually yes, in a controlled order: stop writes, flush and stop databases cleanly, then shut down app VMs. The hall is heating, and a hardware-level power cut later will turn a clean stop into a crash with possible storage recovery work. If you have a standby elsewhere, this is also the moment to fail over deliberately instead of waiting to be forced. The exception is if you have strong evidence your rack is unaffected, which you will rarely have.

Scenario 3: Your facility team says maintenance will take the redundant chiller offline for six weeks. What do you ask for?

A risk acceptance with explicit compensating controls: temporary cooling on standby, extra on-site staffing during the window, a freeze on other risky changes (new rack deployments, electrical work), tighter rate-of-change temperature alerting, and verified UPS protection for cooling controls. On the cloud-customer side, if you depend on that facility, it is a good window to pre-stage your failover.

Key Lessons

  • A 3 ms voltage drop became a 15-hour outage because four weaknesses lined up: a failed DRUPS, an overloaded row, an unprotected chiller controller, and redundant cooling removed by construction.
  • Each feed of a dual-fed system must carry the whole load alone. Monitor for it, or you find out during the failure.
  • Zonal products leave zonal risk with you. GCVE, BMS and most NetApp Volumes tiers needed customer-built cross-zone or cross-region copies to stay up.
  • Rate-of-change alerts catch thermal runaway much earlier than absolute thresholds.
  • Recovery takes far longer than failure and follows a dependency order. Plan and drill it.

Quiz

Quiz

Q1: What was the triggering event, and why should it normally have been harmless? A 3 ms voltage drop on the utility supply caused the utility breakers to trip on both feeds. Normally the diesel rotary UPS units on each side take over the load seamlessly using flywheel energy while the diesel engines start, so the IT load sees no interruption.

Q2: Rows 1 and 2 survived on feed B alone. Why did Row 3 not? Row 3's breaker on feed B tripped on overload. Google attributed this to a "load deployment discrepancy," which in practice suggests the row's load was more than a single feed could carry. Dual feeds only provide redundancy if each feed can carry the whole row by itself.

Q3: Why did the cooling fail, and why was there no backup cooling? The chiller controller dropped offline during the voltage transient, so the chilled-water pumps never got a restart signal and chiller A shut down. The redundant cooling source was unavailable because of known ongoing construction at the facility.

Q4: Why did mainstream services like Compute Engine with regional managed instance groups not appear in this incident, while GCVE, BMS and NetApp Volumes did? The affected products were zonal and, according to The Register, ran from a separate data center that lost power. Regional products such as regional MIGs, regional GKE clusters and Cloud SQL HA spread capacity across zones, so a single facility or zone failure does not take them down. Zonal products need the customer to build cross-zone or cross-region redundancy.

Q5: Name one detection improvement and one customer-side architectural change that this incident argues for. Detection: alert on the rate of temperature rise and on cooling-controller reachability, not just high absolute temperatures. Architecture: for every zonal dependency, maintain a second copy in another zone or region (Data Guard standby for BMS databases, a second GCVE private cloud with replication, NetApp volume replication or regional pools) and prove it with regular failover drills.

One-Liner Summary

A 3 ms dip in utility power, one failed backup generator, one overloaded row and one chiller controller without protected power cooked a Google Cloud data hall to 44°C and took three zonal products down for almost 15 hours, a reminder that redundancy has to be real on the night, and that zonal services leave the zone failure to you.

Sources