Google Cloud europe-west4-a Power and Cooling Failure (July 2026)
On the evening of 15 July 2026 (Pacific time), which was the early morning of 16 July in the Netherlands, the power feed into a data center serving Google Cloud's europe-west4-a zone dipped for 3 milliseconds. That dip should have been routine. A data center facility is built to ride through it: utility breakers trip, rotary UPS units with diesel engines pick up the load, and the chillers keep running.
That night, one of the two backup power systems failed to take the load. One row of racks lost both of its feeds. The chiller controller dropped offline in the same event, the cooling pumps did not restart, and the redundant cooling source was unavailable because of construction work at the site. Within two hours the data hall reached 44°C and Google began shutting machines down to stop them cooking.
Three products went dark: Google Cloud VMware Engine (GCVE), Google Cloud NetApp Volumes (GCNV) and Bare Metal Solution (BMS). According to Google's full incident report, total disruption lasted 14 hours and 55 minutes. Most other Google Cloud services in the region were not named in the incident.
This case study is not about a dramatic bug. It is about correlated failures in physical infrastructure, redundancy that quietly was not there, and a product-design fact many cloud customers never think about: some cloud services live in one zone, sometimes one building, and the zonal risk is yours to carry.
Sourcing note
Almost everything below comes from Google's own incident report on the Google Cloud Service Health page (preliminary report 17 July, full report 25 July 2026), with context from The Register, W.Media and DCD. Google's report gives times in US/Pacific (PDT, UTC−7). This page shows PDT and UTC side by side. Where sources disagree on numbers, the page says so instead of picking one silently.
TL;DR
- What: A utility-side voltage transient tripped both utility feeds into a data center in europe-west4-a. Feed B's diesel rotary UPS (DRUPS) took over cleanly. Feed A's DRUPS failed because of electrical component failures. Rows 1 and 2 moved to feed B, but Row 3 hit an overload breaker trip (Google attributes it to a "load deployment discrepancy") and lost all power.
- Cooling: The chiller controller went offline during the same transient, so the chilled-water distribution pumps did not restart and chiller A shut down. The redundant cooling source was unavailable because of ongoing construction at the facility. The hall reached 44°C.
- Impact: GCVE (24 private clouds, 20 customers), BMS (9 customers) and NetApp Volumes (all 6 clusters in the zone; Standard, Premium and Extreme tiers) were unavailable for between roughly 8 and 13 hours each. Total disruption: 14 h 55 min.
- Detection: Automated telemetry alerted at 16:30–16:39 PDT for power. The temperature alert came at 17:44 PDT, more than an hour after the event began.
- Response: Pumps switched to hand mode, a portable UPS was put on the chiller controllers, a swing DRUPS restored feed-A redundancy, Row 3 breakers were reset, and then a controlled restart: network fabric, then storage, then hosts.
- Lessons: N+1 redundancy that is under construction is N. Power and cooling share failure modes. Products deployed in a single zone need you to design the second zone. Graceful thermal shutdown protects data but needs automation and staffing that is ready when the hall heats fast.
The Alert
At 16:30 PDT (23:30 UTC) on 15 July, GCVE engineers received a power alert. At 16:32 the NetApp Volumes team got a high-temperature and switch failure alert. By 16:41, GCVE and BMS servers were reporting power feed redundancy failure: a server with two power supplies that has lost one of them.
On its own, "lost one feed" is not an emergency. Every enterprise server in a tier-III-style facility has two power supplies fed from independent A and B paths for exactly this reason. What turned the alert into an incident was what happened next: the temperature kept climbing.
The quiet part
At 17:05 PDT GCVE's network switches were already above 60°C. Google's own preventive action list includes "add earlier alerts for power and cooling events," which is an admission that the cooling signal arrived later than it should have. Engineers were alerted to the temperature rise at 17:44 PDT, about 80 minutes after the grid fault.
Impact
| Product | Customer impact | Impact window (PDT, preliminary report) | Duration (preliminary report) |
|---|---|---|---|
| Google Cloud NetApp Volumes | Standard, Premium and Extreme volumes inaccessible. Control-plane operations (creating storage pools, volumes, backups) failed | 15 Jul 16:39 – 16 Jul 01:10 | 8 h 31 min |
| Google Cloud VMware Engine | Lost connectivity to private clouds and their workloads. 24 private clouds, 20 customers | 15 Jul 17:09 – 16 Jul 02:33 | 9 h 24 min |
| Bare Metal Solution | Database servers and storage appliances unreachable. 9 customers | 15 Jul 18:37 – 16 Jul 07:34 | 12 h 57 min |
The full report frames the overall incident as 15 Jul 16:39 to 16 Jul 07:34 PDT, which is the 14 h 55 min figure (Google incident report).
Where the numbers disagree
Different sources, and even different parts of Google's own status site, give different durations. None of these are invented; they are just measured differently.
| Item | Value A | Value B | Notes |
|---|---|---|---|
| Overall start | 16:24 PDT (grid fault, full report data-center timeline) | 16:39 PDT (first customer impact, full report) | The status page header shows 16:57 PDT as incident start. |
| BMS duration | 12 h 57 min (preliminary report table) | 12 h 28 min (status summary list) | W.Media says "nearly 13 hours". |
| GCVE duration | 9 h 24 min (preliminary report) | 9 h 34 min (status summary list) | |
| NetApp Volumes duration | 8 h 31 min (preliminary report) | 8 h 03 min (status summary list) | |
| BMS end | 07:34 PDT (impact end in the report) | 08:42 PDT ("mitigated" in the BMS timeline) | The report's own BMS timeline marks mitigation after its stated impact end. |
| Total | 14 h 55 min (Google full report, DCD, W.Media) | "about 15 hours" (The Register) | Consistent. |
The likely explanation is that the summary list measures from the status-page posting time rather than from first impact, but Google does not say so. Treat the preliminary and full report figures as authoritative.
What was not affected. The incident lists only GCVE, NetApp Volumes and BMS. Compute Engine, GKE, Cloud SQL and other mainstream services in europe-west4-a were not named. The Register reports that Google runs these three services from a separate data center that lost power. W.Media notes Google's reference to a "third-party facility provider" and that it is unclear exactly which facility in or around the Eemshaven campus lost power (W.Media). Google did not answer The Register's question about whether it discloses which services depend on a single data center.
Timeline
All times from Google's full incident report unless noted. PDT is UTC−7. Local time in the Netherlands (CEST) is UTC+2.
| PDT (15–16 Jul) | UTC | Event | Source |
|---|---|---|---|
| 15 Jul 16:24 | 23:24 | Utility grid fault: 3 ms voltage drop, utility breakers trip on both feeds. Side-A DRUPS fails. Chiller controller goes offline | Google full report |
| 16:30 | 23:30 | GCVE receives power alert | Google full report |
| 16:32 | 23:32 | NetApp Volumes receives high-temperature and switch-failure alert | Google full report |
| 16:37 | 23:37 | Row 3 fails to transfer and loses all power (overload breaker trip) | Google full report |
| 16:39 | 23:39 | First customer impact (NetApp Volumes). Automated telemetry alerting from this point (preliminary report) | Google preliminary and full reports |
| 16:41 | 23:41 | GCVE and BMS servers report power-feed redundancy failure | Google full report |
| 17:05 | 00:05 (16 Jul) | GCVE network switches exceed 60°C | Google full report |
| 17:09 | 00:09 | GCVE customer impact begins | Google full report |
| 17:32 | 00:32 | All 6 NetApp Volumes clusters shut down automatically due to heat | Google full report |
| 17:44 | 00:44 | Engineers alerted to the temperature rise | Google full report |
| 17:52 | 00:52 | BMS detects rising temperatures | Google full report |
| 18:29 | 01:29 | Data hall reaches 44°C | Google full report |
| 18:37 | 01:37 | Four NetApp SAN nodes serving BMS fail. BMS customer impact begins | Google full report |
| 18:59 | 01:59 | BMS reserve and buffer servers begin powering off | Google full report |
| 19:00–19:55 | 02:00–02:55 | GCVE devices shut down. Status update says "full shutdown" began 19:00 PDT (04:00 local) | Google status updates and full report |
| 19:55 | 02:55 | Machine shutdown procedures implemented across the hall | Google full report |
| 20:30 | 03:30 | Earliest public status update shown on the incident page, advising GCVE customers to shut down workload VMs "so that recovery is safer" | Google status page |
| 20:43 | 03:43 | BMS sees temperature drop | Google full report |
| 20:52 | 03:52 | BMS SAN nodes recovered | Google full report |
| 21:05 | 04:05 | Cooling fully recovered | Google full report |
| 21:24 / 21:46 | 04:24 / 04:46 | Facility provider sends cooling, then power, recovery notices | Google full report |
| 21:56 | 04:56 | GCVE network restoration starts | Google full report |
| 22:08 | 05:08 | First NetApp Volumes cluster recovered | Google full report |
| 22:40 | 05:40 | GCVE onsite technician arrives | Google full report |
| 23:10 | 06:10 | GCVE console servers recovered | Google full report |
| 23:23 | 06:23 | First BMS customer server reboot | Google full report |
| 23:55 | 06:55 | All 6 NetApp Volumes clusters and customers recovered | Google full report |
| 16 Jul 00:33 | 07:33 | GCVE network recovery complete for all placement groups | Google full report |
| 02:33–02:36 | 09:33–09:36 | GCVE mitigated | Google preliminary and full reports |
| 05:25 | 12:25 | Final status update: GCVE and NetApp Volumes resolved, BMS "believed to be affecting a very small number of customers" | Google status page |
| 06:32 | 13:32 | Last BMS customer server reboot started | Google full report |
| 07:34 | 14:34 | End of impact window (overall) | Google full report |
| 08:42 | 15:42 | BMS marked mitigated in its product timeline | Google full report |
| 17 Jul 17:22 | 18 Jul 00:22 | Preliminary incident report published | Google status page |
| 25 Jul 06:16 | 13:16 | Full incident report published | Google status page |
sequenceDiagram
participant Grid as Utility grid
participant Power as Facility power (A and B)
participant Cool as Chillers and pumps
participant Hall as Data hall
participant Svc as GCVE, GCNV, BMS
participant Eng as Google engineers
Grid->>Power: 16:24 PDT — 3 ms voltage drop, both utility breakers trip
Power->>Power: Side B DRUPS takes load, side A DRUPS fails
Power->>Hall: 16:37 — Row 3 overload trip, row loses all power
Grid->>Cool: Chiller controller drops offline, pumps do not restart
Svc->>Eng: 16:30 to 16:41 — power and redundancy alerts
Hall->>Svc: 17:05 — switches above 60°C
Svc->>Svc: 17:32 — NetApp clusters thermal auto-shutdown
Eng->>Eng: 17:44 — temperature-rise alert reaches engineers
Hall->>Hall: 18:29 — hall reaches 44°C
Eng->>Svc: 19:00 to 19:55 — controlled shutdown of remaining devices
Eng->>Cool: Pumps to hand mode, portable UPS on chiller controllers
Cool->>Hall: 21:05 — cooling fully recovered
Eng->>Svc: 21:56 onwards — network, then storage, then hosts
Svc->>Eng: 16 Jul 07:34 — impact ends, BMS last to recover
Root Cause
Google's full report names the root cause as "a 3ms voltage drop in the power feed from the utility provider and subsequent utility breaker protective action." That is the trigger. The reason a 3 ms dip became a 15-hour outage is a chain of four independent weaknesses that lined up.
1. The power path: A/B feeds with rotary UPS
A typical hyperscale-adjacent facility feeds each row of racks from two independent electrical paths, A and B. Each server has two power supplies, one on each path. Each path has its own utility connection and its own backup.
The backup here was a DRUPS (Diesel Rotary Uninterruptible Power Supply). A DRUPS couples a motor-generator and a flywheel to a diesel engine. When utility power drops, the flywheel's stored kinetic energy keeps the output alive for the seconds it takes the diesel engine to start and clutch in. No battery room is involved. When it works, the load sees nothing.
flowchart TB
U["Utility grid"] --> BA["Utility breaker A"]
U --> BB["Utility breaker B"]
BA --> DA["DRUPS A (flywheel + diesel)"]
BB --> DB["DRUPS B (flywheel + diesel)"]
DA --> PA["Distribution A"]
DB --> PB["Distribution B"]
PA --> R1["Row 1"]
PA --> R2["Row 2"]
PA --> R3["Row 3"]
PB --> R1
PB --> R2
PB --> R3
U --> CC["Chiller controller"]
CC --> PU["Chilled-water pumps"]
PU --> CA["Chiller A"]
CB["Redundant cooling source"] -.->|"unavailable: construction"| PU
CA --> HALL["Data hall air"]
2. Side A's DRUPS failed
On 15 July, the voltage transient tripped the utility breakers on both feeds. That is expected behaviour: protective breakers open on a disturbance and the DRUPS take over.
- Side B: "The transition of side B to DRUPS system was successful without any power interruption" (Google, quoted by The Register).
- Side A: "The DRUPS back-up power system for side A failed to take over the facility load due to electrical component failures."
Google has not published which components failed. One of the preventive actions due in August 2026 is to investigate why the load did not transfer; as of this page's review date we found no public follow-up.
3. Row 3 could not survive on one feed
With side A gone, every row should have run on side B alone. That is the whole point of dual feeds. Rows 1 and 2 did. Row 3 did not. Its side-B breaker tripped on overload, and the row lost both feeds. Google attributes this to a "load deployment discrepancy" and said it was still investigating.
The report does not explain the term. The usual meaning in data-center operations is that the equipment actually installed on a circuit draws more than the circuit was planned for. In a dual-fed design, each feed must be able to carry the whole row's load alone, which means each feed should normally run at no more than about 50% of its rating. If a row has crept above that (more racks, denser servers, a planning record that does not match reality), everything looks fine every day, because both feeds share the load. The problem is invisible until one feed fails and the survivor has to carry 100%.
The latent-failure pattern
A dual-fed row loaded at 60% per feed shows no symptoms in normal operation. It fails only at the exact moment you need the redundancy. This is the same class of bug as a database replica that cannot actually keep up with primary write volume: it reports healthy until failover, then falls over. Note: Google has not confirmed that this was the mechanism; it is the textbook reading of the phrase.
The Register's summary of the full report quotes the outcome: "This resulted in loss of both redundant power feeds to the single row."
4. Cooling failed in the same event
Losing one row of power is bad but bounded. What made it hall-wide was cooling.
- "The chiller controller dropped offline during the voltage transient event." The controller was apparently not on protected power, or its protection did not ride through the event.
- Because the controller was offline, the chilled-water distribution pumps never received the restart signal, and chiller system A shut down.
- "The redundant source was not available due to known ongoing construction work at the facility."
So the cooling system had redundancy on paper, but on this night it was a single point of failure, and that single point was controlled by a device that shared the triggering event.
flowchart TD
T["Utility voltage transient (3 ms)"] --> B["Utility breakers trip on A and B"]
B --> DA{"Side A DRUPS takes load?"}
DA -->|"no: component failures"| LA["Feed A lost"]
B --> DB{"Side B DRUPS takes load?"}
DB -->|"yes"| OKB["Feed B carries rows 1 to 3"]
LA --> R3{"Row 3 fits on feed B alone?"}
OKB --> R3
R3 -->|"no: overload trip"| R3X["Row 3 loses all power"]
T --> CC["Chiller controller offline"]
CC --> PUMP["Pumps do not restart, chiller A stops"]
PUMP --> RED{"Redundant cooling available?"}
RED -->|"no: construction"| HOT["Hall heats to 44°C"]
HOT --> SHUT["Thermal auto-shutdowns + manual shutdown"]
R3X --> OUT["GCVE, NetApp Volumes, BMS outage"]
SHUT --> OUT
Why heat, not power, set the duration
Row 3 lost power at 16:37. But the NetApp clusters shut themselves down at 17:32 due to heat, and the BMS SAN nodes failed at 18:37, again in a hot hall. Cooling was not fully back until 21:05 PDT, and only then could the controlled restart begin. Then recovery itself took hours: the last BMS customer server reboot did not start until 06:32 PDT the next morning.
This is the general shape of a data-center thermal event:
- Minutes: temperature rises fast once airflow or chilled water stops. Modern dense racks have very little thermal mass.
- Tens of minutes: devices hit thermal limits and either throttle, shut themselves down uncleanly, or must be shut down by operators.
- Hours: cooling has to be restored and the hall brought back to safe temperature before anything restarts.
- Many hours: a careful restart, in dependency order, with validation, because a hard thermal shutdown can leave storage in states that need checking.
Detection
What worked. Power alerts reached engineers within six minutes of the grid fault (16:30 PDT). The NetApp clusters had automatic thermal protection and shut down at 17:32 rather than running until damaged.
What did not.
- The temperature-rise alert reached engineers at 17:44, about 80 minutes after the controller dropped offline and nearly 40 minutes after switches crossed 60°C. Google's preventive action list includes "add earlier alerts for power and cooling events."
- The signals were split across teams. The power alert went to GCVE, the temperature and switch alert went to NetApp, and BMS saw a redundancy failure. Three product teams each saw part of one facility event.
- The public status page lagged. The earliest update visible on the incident page is 20:30 PDT, nearly four hours after first impact. (We cannot rule out that earlier updates were edited or merged, but the page as published starts there.)
flowchart LR
F["Facility BMS/SCADA telemetry"] -->|"power events"| P1["GCVE power alert 16:30"]
F -->|"temperature"| P2["NetApp temp alert 16:32"]
F -->|"PSU redundancy"| P3["GCVE + BMS 16:41"]
F -->|"hall temperature"| P4["Engineer temp alert 17:44"]
P1 --> IC["Incident commander"]
P2 --> IC
P3 --> IC
P4 --> IC
IC --> SP["Public status page 20:30"]
The detection lesson for everyone
A single physical root cause showed up as three different symptoms on three products. If your alerting routes by product, you need correlation by location too: "every alert in building X in the last 15 minutes" is often the fastest way to see a facility event.
Response and Recovery
The full report lists these mitigation steps:
- Pumps switched to hand mode to restore chilled-water circulation without waiting for the controller.
- A portable UPS was connected to the chiller controllers, so a repeat voltage dip could not knock them out again.
- Two remote power panels needed manual verification before power could be restored to the affected row.
- Row 3's tripped breakers were reset once the fault was isolated.
- A swing DRUPS unit restored redundancy on feed A while the failed DRUPS was repaired.
- Controlled restart by layer. For GCVE: network restoration started at 21:56, an onsite technician arrived at 22:40, console servers were recovered at 23:10, and network recovery for all placement groups completed at 00:33. NetApp clusters came back one by one from 22:08 to 23:55. BMS servers were rebooted customer by customer from 23:23 to 06:32.
Google also told GCVE customers, in a status update, to shut down their VMware workload VMs "for precaution so that recovery is safer." That is good advice for any customer during a thermal event: a VM shut down cleanly by its owner is easier to bring back than one that lost its storage mid-write.
flowchart LR
C["Cooling restored (21:05)"] --> N["Network fabric (21:56 to 00:33)"]
C --> S["Storage clusters (22:08 to 23:55)"]
N --> H["Hosts and private clouds"]
S --> H
H --> B["BMS servers, one by one (23:23 to 06:32)"]
B --> V["Validation and mitigation"]
Preventive actions Google committed to
From the full report (Google):
| Action | Owner area | Target |
|---|---|---|
| Investigate why load did not transfer to DRUPS | Facility | Aug 2026 |
| Resolve the cause of the Row 3 overload | Facility | Aug 2026 |
| Review chiller pump control redundancy | Facility | Sep 2026 |
| Add earlier alerts for power and cooling events | Facility / monitoring | Aug 2026 |
| Improve automated shutdown during fast thermal runaway | GCVE | Oct 2026 |
| Update incident classification and SLOs so staff are available during major events | BMS | Sep 2026 |
| Review SLAs and runbooks, run monthly joint drills, look at faster cluster recovery | NetApp Volumes | Aug 2026 |
We did not find a public follow-up confirming completion of these actions as of 9 November 2026.
What Would Have Prevented It
Split this into two halves: what the facility operator controls, and what you, the customer, control. Most readers are in the second group, and that half is where the actionable lessons are.
Facility-side controls
These are standard data-center engineering practices. We list them because each one maps directly to a link in the failure chain.
| Weak link | Control |
|---|---|
| Side-A DRUPS failed to pick up load | Regular integrated systems tests (pull-the-plug tests that open the utility breaker under real load), not just engine-start tests. Component-level maintenance on transfer gear |
| Row 3 overloaded on a single feed | Per-feed load cap (commonly ≤ 50% of breaker rating on each side of a 2N row, with margin), continuous branch-circuit metering, and capacity planning that blocks new deployments that break the cap |
| Chiller controller dropped offline | Put cooling controls on the same protected (UPS-backed) power as IT load. Google's own fix was a portable UPS on the controllers |
| Pumps did not restart | Fail-safe pump logic: pumps default to running on loss of controller signal |
| Redundant cooling under construction | Treat construction or maintenance that removes redundancy as a declared risk state: extra staffing, pre-positioned temporary cooling, and a freeze on risky work elsewhere until redundancy is back |
| Temperature alert at 17:44 | Rate-of-change alerts on hall and inlet temperature, not just absolute thresholds |
A rate-of-change alert is simple and catches thermal runaway long before an absolute threshold does. In Prometheus terms (the same idea works in any monitoring stack):
# Alert on fast temperature rise, not just on a high absolute value.
groups:
- name: facility-thermal
rules:
- alert: DataHallTemperatureRisingFast
# More than 3°C rise in 10 minutes at any inlet sensor in the hall
expr: delta(dc_inlet_temperature_celsius[10m]) > 3
for: 2m
labels:
severity: page
annotations:
summary: "Inlet temperature rising fast in {{ $labels.hall }}"
- alert: CoolingControllerUnreachable
expr: up{job="chiller-controller"} == 0
for: 1m
labels:
severity: page
- alert: RowSingleFeedOverCapacity
# Each feed in a 2N row must be able to carry the whole row alone.
expr: |
(sum by (row) (pdu_feed_power_watts))
/ on(row) group_left min by (row) (pdu_feed_rating_watts) > 0.8
for: 15m
labels:
severity: ticket
annotations:
summary: "Row {{ $labels.row }} could not survive on one feed"
The third rule is the one that would have caught the latent Row 3 problem weeks earlier: it asks whether the total row load fits on the smaller feed, which is the only question that matters when one feed fails.
Customer-side: you own the zonal risk
Google's region documentation is explicit that a zone is a failure domain and that you should spread workloads across zones for high availability (Google Cloud regions and zones). Many Google Cloud services do that for you: regional managed instance groups, regional GKE clusters, Cloud SQL high availability, regional Persistent Disk, multi-region Cloud Storage.
But GCVE, BMS and most NetApp Volumes service levels are zonal products. A GCVE private cloud lives in one zone. A BMS server is a physical machine in one facility. A Standard/Premium/Extreme NetApp Volumes pool sits in one zone. When the zone goes, they go, and no amount of in-zone redundancy (more ESXi hosts, a bigger cluster) helps.
flowchart TB
subgraph REG["Region europe-west4"]
subgraph ZA["Zone a"]
GCVE["GCVE private cloud"]
BMS["BMS servers"]
NV["NetApp Volumes pool"]
MIGA["MIG instances"]
end
subgraph ZB["Zone b"]
MIGB["MIG instances"]
end
subgraph ZC["Zone c"]
MIGC["MIG instances"]
end
end
LB["Regional load balancer"] --> MIGA
LB --> MIGB
LB --> MIGC
APP["Your VMware workload"] --> GCVE
DB["Your Oracle on BMS"] --> BMS
In this diagram, the managed instance group survives the loss of zone a. The GCVE workload, the BMS database and the NetApp volume do not, unless you built their second copy yourself.
Control 1: Regional by default for things Google can spread
For ordinary compute, make the regional option the default in your Terraform modules so nobody accidentally ships a zonal workload.
# Regional managed instance group: instances spread over three zones,
# and the autohealer recreates them in surviving zones if one zone fails.
resource "google_compute_region_instance_group_manager" "web" {
name = "web-rmig"
region = "europe-west4"
base_instance_name = "web"
distribution_policy_zones = [
"europe-west4-a",
"europe-west4-b",
"europe-west4-c",
]
# EVEN keeps capacity balanced, so losing one zone loses about a third.
distribution_policy_target_shape = "EVEN"
version {
instance_template = google_compute_instance_template.web.id
}
# Size for N-1 zones: if you need 6 instances to serve peak,
# run 9 so that 6 remain when a zone disappears.
target_size = 9
auto_healing_policies {
health_check = google_compute_health_check.web.id
initial_delay_sec = 120
}
update_policy {
type = "PROACTIVE"
minimal_action = "REPLACE"
max_surge_fixed = 3
max_unavailable_fixed = 0
}
}
# Regional persistent disk: synchronously replicated across two zones.
resource "google_compute_region_disk" "data" {
name = "data-rpd"
region = "europe-west4"
replica_zones = ["europe-west4-a", "europe-west4-b"]
type = "pd-balanced"
size = 500
}
# Cloud SQL with high availability: standby in another zone, automatic failover.
resource "google_sql_database_instance" "main" {
name = "main-pg"
region = "europe-west4"
database_version = "POSTGRES_16"
settings {
tier = "db-custom-4-16384"
availability_type = "REGIONAL"
backup_configuration {
enabled = true
point_in_time_recovery_enabled = true
}
}
}
Note the N-1 sizing. Spreading over three zones does nothing if the two surviving zones cannot handle the full load. See Redundancy and Replication for the arithmetic.
For Kubernetes, use a regional cluster and force replicas across zones:
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
replicas: 6
selector:
matchLabels: { app: api }
template:
metadata:
labels: { app: api }
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels: { app: api }
containers:
- name: api
image: europe-west4-docker.pkg.dev/acme/api/api:1.42.0
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: api
spec:
minAvailable: 4
selector:
matchLabels: { app: api }
Control 2: Build the second zone yourself for zonal products
For products like the three that failed, the provider gives you building blocks and you assemble the redundancy.
VMware Engine. Options, roughly from cheapest to strongest:
- Second private cloud in another zone or region, with VM-level replication (for example VMware HCX or a third-party DR tool) and a tested runbook to fail over. Accept an RPO of minutes and an RTO of an hour or more.
- Stretched private cloud across two zones where Google offers it, which gives vSAN-level synchronous replication between zones. Check current Google documentation for regional availability and constraints. Note that a separate 14 July 2026 status entry reports GCVE stretched cluster customers experiencing zonal network connectivity problems (status summary), so stretched does not mean immune: test it.
- Move stateless tiers off GCVE to regional compute and keep only what truly needs VMware there.
Bare Metal Solution. A BMS server is one machine in one place. Typical database HA on BMS is application-level:
- Oracle Data Guard (or the equivalent for your database) with a standby in a different zone or region, and Fast-Start Failover or a scripted switchover.
- Keep backups outside the zone (Cloud Storage dual-region or multi-region bucket).
- Measure the real RTO: how long does it take your team to promote the standby and repoint the application at 2 a.m.?
NetApp Volumes. Google's status update said only the Standard, Premium and Extreme service levels were affected. Google documents the Flex service level as supporting regional storage pools that span two zones, and NetApp Volumes supports volume replication to another region. Check the current documentation for which features apply to your service level before relying on them.
Control 3: Prove it with a zone-failure game day
Architecture diagrams lie. The only way to know you survive a zone loss is to remove a zone and watch. A simple, reversible GKE drill:
#!/usr/bin/env bash
# Zone-loss drill for a regional GKE cluster.
# Cordons and drains every node in one zone, then checks the service still answers.
# Run in a staging cluster first. Reverse with the uncordon command at the end.
set -euo pipefail
ZONE="${1:-europe-west4-a}"
URL="${2:-https://staging.example.com/healthz}"
echo "Cordoning all nodes in $ZONE"
kubectl get nodes -l "topology.kubernetes.io/zone=$ZONE" -o name \
| xargs -r kubectl cordon
echo "Draining nodes in $ZONE"
kubectl get nodes -l "topology.kubernetes.io/zone=$ZONE" -o name \
| xargs -r -n1 kubectl drain --ignore-daemonsets --delete-emptydir-data --timeout=10m
echo "Probing $URL for 10 minutes"
fail=0
for i in $(seq 1 120); do
code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 "$URL" || echo 000)
[ "$code" = "200" ] || { fail=$((fail+1)); echo "probe $i -> $code"; }
sleep 5
done
echo "Failed probes: $fail of 120"
echo "To restore: kubectl get nodes -l topology.kubernetes.io/zone=$ZONE -o name | xargs kubectl uncordon"
For zonal products the drill is a DR failover exercise: shut down the primary VMs or database in zone a, promote the standby elsewhere, measure RTO and RPO, and fail back. Do it on a schedule. Chaos Engineering covers how to run this safely.
Control 4: Audit what is still zonal
You cannot fix single-zone dependencies you do not know about. A small script that lists zonal resources is worth more than a slide that says "we are multi-zone."
#!/usr/bin/env python3
"""List Compute Engine instances and disks per zone in one region.
Flags any instance group or label-defined service whose instances
all sit in a single zone. Uses the gcloud CLI's JSON output so there
are no extra dependencies.
"""
import collections
import json
import subprocess
import sys
def gcloud_json(*args):
out = subprocess.run(
["gcloud", *args, "--format=json"],
check=True, capture_output=True, text=True,
).stdout
return json.loads(out)
def zones_by_service(instances, label="service"):
zones = collections.defaultdict(set)
for inst in instances:
svc = inst.get("labels", {}).get(label, "<unlabelled>")
zones[svc].add(inst["zone"].rsplit("/", 1)[-1])
return zones
def single_zone(zones_map):
return sorted(svc for svc, zs in zones_map.items() if len(zs) < 2)
def main(region):
instances = gcloud_json(
"compute", "instances", "list", f"--filter=zone~^{region}-"
)
zones = zones_by_service(instances)
for svc, zs in sorted(zones.items()):
print(f"{svc:30} {sorted(zs)}")
bad = single_zone(zones)
if bad:
print("\nSINGLE-ZONE services:", ", ".join(bad))
return 1
return 0
if __name__ == "__main__":
# Self-check of the pure logic, no gcloud needed: python zonal_audit.py --test
if sys.argv[1:] == ["--test"]:
fake = [
{"zone": "z/europe-west4-a", "labels": {"service": "api"}},
{"zone": "z/europe-west4-b", "labels": {"service": "api"}},
{"zone": "z/europe-west4-a", "labels": {"service": "db"}},
]
assert single_zone(zones_by_service(fake)) == ["db"]
print("ok")
sys.exit(0)
sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else "europe-west4"))
Extend the same idea to the zonal products you use (GCVE private clouds, NetApp storage pools, BMS servers) by keeping them in an inventory with a zone and second_copy field, and failing the review if second_copy is empty for anything tier-1.
Control 5: Graceful shutdown on thermal events
Google's GCVE team committed to "improve automated shutdown during fast thermal runaway." Customers can do their part: when the provider announces a thermal or power event in your zone, shut your own workloads down cleanly (databases first flush and stop, then app VMs) instead of waiting for the hardware to cut power. That turns an unclean crash into a clean restart and shortens your own recovery. Wire Google Cloud Service Health notifications into your paging system so you see the announcement at the same time Google posts it.
Lessons for System Design
1. Redundancy you cannot use is not redundancy
The facility had two power feeds, two DRUPS units and a redundant cooling source. On the night, one DRUPS failed, one row could not run on its surviving feed, and the redundant cooling was taken out by construction. Count the redundancy you actually have right now, not the redundancy in the design document. Planned maintenance, construction and degraded components all reduce N+1 to N, and N is where the next small fault becomes an outage. The AWS S3 2017 outage is a software version of the same idea: subsystems that had not been restarted at scale for years.
2. Redundant paths fail together when they share a trigger
The voltage transient tripped both feeds, killed a DRUPS, and took the chiller controller offline. These were supposedly independent systems with a common upstream event. When you design for failure, ask "what single event touches both copies?" For software, the equivalents are a shared config push, a shared DNS dependency, a shared certificate, or a shared control plane. Facebook October 2021 shows how a single control-plane event removed every redundant path at once.
3. Latent capacity bugs hide behind redundancy
Row 3 ran fine on two feeds and failed on one. Any system where each redundant copy must be able to carry the whole load needs a check that it can: per-feed load caps, database replicas sized for primary write load, a cluster that can absorb a third of its nodes vanishing. Load-test the failover state, not just the steady state. Availability Patterns goes into active-active sizing.
4. A zone is a failure domain, and some products have only one
Cloud documentation tells you to use multiple zones. That advice only works for services that can span zones. Products that pin you to one zone, or even one building, leave the zonal risk on your side of the shared-responsibility line. As Gartner's Adrian Wong told The Register, "It is very hard to figure out how an individual region is architected," and customers "are often surprised by that" (The Register). The practical response: classify every dependency as zonal, regional or multi-regional, and treat zonal ones as needing a second copy you build. See Multi-Region Architecture Patterns and Failover Strategies.
5. Recovery is slower than failure, and ordered
Power was back by 21:46 PDT. The last BMS server reboot started at 06:32 the next morning. Recovery had a dependency order (cooling, then network fabric, then storage, then hosts), needed hands on site (the GCVE technician arrived at 22:40), and was serialised per customer for BMS. Your own recovery plans should model that: write down the order, estimate each step, and multiply the total by the number of things that need restarting. Disaster Recovery covers RTO and RPO planning.
6. Staffing is part of the architecture
Two of the product-level preventive actions are about people, not hardware: BMS will update incident classification "so staff are available during major events," and NetApp Volumes will run monthly joint drills. An incident that starts at 01:24 local time needs an on-call model that can put engineers on site fast. If your RTO assumes a human, it also assumes that human can be reached and can get there.
7. Rate of change beats thresholds for fast failures
Thermal runaway in a dense hall is fast. The difference between a 16:39 alert on "temperature rising fast" and a 17:44 alert on "temperature high" was about an hour of reaction time. The same applies to error rates, queue depth and disk fill: alert on the derivative when the failure accelerates.
Related Pages
- AWS S3 February 2017 — another single-region dependency that took down much of the internet
- Facebook October 2021 — one event removing every redundant path
- Netflix Chaos Engineering — why you deliberately remove zones
- Chaos Engineering — running zone-loss game days safely
- Disaster Recovery — RTO, RPO and DR runbooks
- Multi-Region Architecture Patterns
- Failover Strategies
- Multi-Region with Terraform
- Redundancy and Replication
- Availability Patterns
- Multi-Region Databases
- Database Failover Runbook
- GKE and Cloud SQL
- SLI, SLO and SLA
- Postmortem Framework
What Would You Do?
Scenario 1: You run a customer-facing Oracle database on Bare Metal Solution in europe-west4-a. Your RTO target is 1 hour. What is the minimum architecture that meets it during an event like this one?
A standby database outside zone a (another zone if available for BMS in your region, otherwise another region), kept current with Data Guard or equivalent, plus a tested, scripted switchover and application connection strings that can be repointed quickly (DNS with short TTL or a connection manager). Backups alone do not meet a 1-hour RTO for a large database, and a second server in the same zone fails in the same hot room. Test the switchover quarterly and time it.
Scenario 2: Google posts that your GCVE zone has a cooling failure and recommends shutting down VMs. Your app is still answering. Do you shut down?
Usually yes, in a controlled order: stop writes, flush and stop databases cleanly, then shut down app VMs. The hall is heating, and a hardware-level power cut later will turn a clean stop into a crash with possible storage recovery work. If you have a standby elsewhere, this is also the moment to fail over deliberately instead of waiting to be forced. The exception is if you have strong evidence your rack is unaffected, which you will rarely have.
Scenario 3: Your facility team says maintenance will take the redundant chiller offline for six weeks. What do you ask for?
A risk acceptance with explicit compensating controls: temporary cooling on standby, extra on-site staffing during the window, a freeze on other risky changes (new rack deployments, electrical work), tighter rate-of-change temperature alerting, and verified UPS protection for cooling controls. On the cloud-customer side, if you depend on that facility, it is a good window to pre-stage your failover.
Key Lessons
- A 3 ms voltage drop became a 15-hour outage because four weaknesses lined up: a failed DRUPS, an overloaded row, an unprotected chiller controller, and redundant cooling removed by construction.
- Each feed of a dual-fed system must carry the whole load alone. Monitor for it, or you find out during the failure.
- Zonal products leave zonal risk with you. GCVE, BMS and most NetApp Volumes tiers needed customer-built cross-zone or cross-region copies to stay up.
- Rate-of-change alerts catch thermal runaway much earlier than absolute thresholds.
- Recovery takes far longer than failure and follows a dependency order. Plan and drill it.
Quiz
Quiz
Q1: What was the triggering event, and why should it normally have been harmless? A 3 ms voltage drop on the utility supply caused the utility breakers to trip on both feeds. Normally the diesel rotary UPS units on each side take over the load seamlessly using flywheel energy while the diesel engines start, so the IT load sees no interruption.
Q2: Rows 1 and 2 survived on feed B alone. Why did Row 3 not? Row 3's breaker on feed B tripped on overload. Google attributed this to a "load deployment discrepancy," which in practice suggests the row's load was more than a single feed could carry. Dual feeds only provide redundancy if each feed can carry the whole row by itself.
Q3: Why did the cooling fail, and why was there no backup cooling? The chiller controller dropped offline during the voltage transient, so the chilled-water pumps never got a restart signal and chiller A shut down. The redundant cooling source was unavailable because of known ongoing construction at the facility.
Q4: Why did mainstream services like Compute Engine with regional managed instance groups not appear in this incident, while GCVE, BMS and NetApp Volumes did? The affected products were zonal and, according to The Register, ran from a separate data center that lost power. Regional products such as regional MIGs, regional GKE clusters and Cloud SQL HA spread capacity across zones, so a single facility or zone failure does not take them down. Zonal products need the customer to build cross-zone or cross-region redundancy.
Q5: Name one detection improvement and one customer-side architectural change that this incident argues for. Detection: alert on the rate of temperature rise and on cooling-controller reachability, not just high absolute temperatures. Architecture: for every zonal dependency, maintain a second copy in another zone or region (Data Guard standby for BMS databases, a second GCVE private cloud with replication, NetApp volume replication or regional pools) and prove it with regular failover drills.
One-Liner Summary
A 3 ms dip in utility power, one failed backup generator, one overloaded row and one chiller controller without protected power cooked a Google Cloud data hall to 44°C and took three zonal products down for almost 15 hours, a reminder that redundancy has to be real on the night, and that zonal services leave the zone failure to you.
Sources
- Google Cloud Service Health: incident 3BvH3LVGcupoYqV6F4Nw — status updates (15–16 Jul 2026), preliminary report (17 Jul) and full incident report (25 Jul), including product timelines, root cause, remediation and preventive actions. Primary source.
- Google Cloud Service Health: incident summary list — per-product durations as listed by Google, and the separate 14 July 2026 GCVE stretched-cluster entry
- The Register: Google Cloud outage shows it's still hard to understand hyperscalers' real resilience regimes (21 Jul 2026, updated 27 Jul) — quotes from the full report, Forrester and Gartner analyst comments, unanswered questions to Google
- W.Media: Electrical fault knocked out Google Cloud's Netherlands zone (22 Jul 2026) — Eemshaven location, per-product durations, Google recovery statements
- DCD: Google Cloud experiences outage after power and cooling failure in Netherlands data center — total duration and start time (14 h 55 min from 16:39 PST)
- Google Cloud: Regions and zones — zones as failure domains and the guidance to distribute workloads