Google’s own incident report says the backup generators had power available to start — the sensors just couldn’t see that they needed to

Overview

On October 19–20, 2026, a single zone in Google Cloud’s London region (europe-west2-b) went dark for over six hours after a chain of failures that started with an underground utility cable fault and ended with backup generators that never got the signal to start. Google’s own final incident report, published on its status page, is unusually specific about the root cause: the sensors responsible for detecting a power loss were wired to measure voltage from the wrong point in the circuit — so from their perspective, nothing was ever wrong. The outage took down roughly 50 Google Cloud products region-wide and disrupted UK banking services including Revolut, Lloyds, and HSBC.

The chain, start to finish

• October 14, 2026 — A routine generator load test at the facility leaves one of two utility power feeds (feed B) locked out for repairs. The site is now running in a single-feed (“1N”) state — no redundancy if the remaining feed fails.
• October 19, 22:53 PT (October 20, 06:53 BST) — An external underground cable fault on the utility network cuts the remaining active feed, feed A.
• Immediately after — Backup generators should start automatically. They don’t. The facility’s backup power controllers (PLCs) have undervoltage sensors wired upstream of the isolated feed B breaker — meaning the sensors were still reading live voltage from a part of the circuit that was never actually lost, and never issued a start command for a loss the facility was, in reality, already experiencing on feed A.
• Within about 5 minutes — UPS batteries, sized for a short bridge until generators take over, run out. The zone loses power entirely.
• Remote transfer was inhibited — on-site staff had to manually move load to the standby utility source instead of the system doing it automatically.

Every individual piece of this — the maintenance state, the cable fault, the sensor placement — is the kind of thing that passes review on its own. Lined up in this exact order, they took an entire zone offline.

What actually broke, and for how long

Google’s final report puts the primary outage phase at 5 hours 29 minutes (23:00 to 04:29 PT), with the full incident closing at 6 hours 8 minutes. Zone-scoped services in europe-west2-b — Compute Engine VMs, Persistent Disk and Hyperdisk, zonal GKE control planes and nodes, non-HA Cloud SQL and AlloyDB instances, single-cluster Bigtable — were simply unreachable for the duration. But the damage didn’t stay zone-scoped: cross-zone calls into the dead zone timed out, and the resulting demand surge caused 500 and 503 errors and admission-control queuing for services in the two healthy zones too. Google’s final list of affected products region-wide runs to roughly 50, including AlloyDB, Apigee, BigQuery, Cloud Run, Cloud Spanner, Cloud Storage, GKE, Looker, the Memorystore family, Pub/Sub, Cloud SQL, Speech-to-Text, and VPC. Some failovers that should have moved traffic out of the dead zone automatically failed anyway, blocked by capacity constraints and storage-replication issues on the receiving end.

New VM creation didn’t resume until 04:46. By 05:08, over 99% of Compute Engine VMs were healthy again; the remaining customers were routed to individual support cases.

Why this is more interesting than “the power went out”

Power failures at data centers aren’t rare, and redundant generators exist specifically to make them boring. What makes this one worth reading past the headline is that the redundancy didn’t fail from lack of capacity or a mechanical fault in the generators themselves — it failed because the monitoring system watching for the failure was looking at the wrong wire. The PLC’s undervoltage sensors were positioned upstream of a breaker that had already isolated part of the circuit for the Oct 14 maintenance, so they were structurally incapable of seeing that feed A — the part actually carrying load — had gone down. The system had a blind spot built directly into its failure-detection logic, and nobody found it until the one scenario that specifically exercised it actually happened.

If that sounds familiar, it should: it’s the same failure mode as a health check that pings a load balancer instead of the service behind it, or a monitor that checks “is the process running” instead of “is the process actually serving traffic.” A sensor watching the wrong signal gives you confident, continuous, wrong information — which is worse than no signal at all, because nothing ever alerts.

How Google responded, and what’s still in progress

Completed in the days immediately after:

• Located the cable fault (October 26).
• Reconfigured and validated the PLC wiring (October 20 and 26).
• Ran a full generator load-transfer test with updated procedures (October 20 and 26).
• Updated Emergency Operating Procedures and had on-site staff practice them (October 24 and 26).

Still outstanding, per Google’s own published timeline:

• Utility repair of the damaged feed A cable, targeted for October 30.
• Controlled transfer back onto both primary utility feeds, and a full generator load test on primary feeds, both targeted for October 31.
• A full PLC code review and resiliency testing, also targeted for October 31.
• A freeze on planned maintenance at facilities with a similar topology, with elevated change-approval requirements, starting October 30.
• Global validation of PLC wiring at other sites using similar topology and hardware, targeted for completion by November 15.

Google also committed to building an automated alert-to-communication mechanism for location-impacting incidents, and expanding platform-wide testing specifically for infrastructure-level events like this one — both implicit admissions that the detection and communication side of this incident, not just the hardware side, needs work.

Real-world impact, briefly

This wasn’t an abstract cloud-status-page event. Revolut customers in the UK reported being unable to access accounts or complete card payments during the outage. Lloyds and HSBC both acknowledged customer-facing disruption, and Halifax (part of Lloyds Banking Group) reported app and online banking issues. Google publicly apologized for the disruption. However you feel about any single vendor’s reliability record, this is a concrete reminder that “it’s just a cloud zone” can mean “people couldn’t pay for groceries” depending on what’s actually built on top of it.

The broader lesson

Redundancy only protects you if the thing checking whether redundancy is needed is actually watching the right signal. A generator with fuel, maintenance, and a clean bill of health from last month’s load test is worthless if the sensor deciding whether to start it is wired to a part of the circuit that was never going to lose power in the first place. If you run any kind of automated failover — database replicas, multi-AZ load balancing, backup power, anything — the question worth asking after reading this isn’t “do we have redundancy,” it’s “has anyone actually verified what our failure detection is measuring, under the exact failure condition it exists to catch, recently.” Most teams can answer yes to the first question and have genuinely never checked the second.

Takeaway

Google Cloud didn’t lose a zone because it lacked backup power — it lost a zone because the system meant to detect the need for backup power was wired to watch a wire that could never show a problem. That’s a more uncomfortable failure than a simple equipment fault, because it means the safeguard was present, tested, and still blind. If your incident response has ever stopped at “we have a backup for that,” this is the argument for going one layer deeper and asking what, exactly, your backup’s trigger condition is actually looking at.