CCT 378: Two Feeds, Full Backup, And The Generators Never Started (Domain 3) cover art

CCT 378: Two Feeds, Full Backup, And The Generators Never Started (Domain 3)

CCT 378: Two Feeds, Full Backup, And The Generators Never Started (Domain 3)

Listen for free

View show details

This Season's Biggest Deal | $0.99/mo for 3 months + $20 Audible bonus

Auto-renews at $8.99/mo. after 3 months - terms apply.
Summary
Send us Fan MailCISSP Domain 3.9 — design site and facility security controls, built on a real data center power failure where every control in the building worked and the building still went dark.Start free: https://www.cisspcybertraining.com/free-cissp-self-study (360 practice questions, a Domain 1 diagnostic, weekly questions by email, and the slides from this episode)In October, a cloud provider's London data center lost its utility power. The batteries picked up the load exactly the way they were supposed to, ran for about five minutes, and then ran out. The generators should have taken over in that window. They never got the signal.Nobody attacked anybody here. There are no bad guys in this one. No ransomware, no intrusion, nothing for incident response to chase. It was a cable fault outside the building and a logic error inside it.HOW THE BUILDING WAS BUILTStart with the design, because the design was good. Two completely independent utility feeds coming in from the power company — they call them A and B. Google's own words for that are a 2N utility power architecture.Now listen to where that label stops. Behind both of those feeds sits one central bank of high-voltage generators, run by a dual-redundant controller, a PLC. Two of the thing coming in. One of the thing that was supposed to save them. And between the utility going away and the generators coming up, a battery system with about five minutes of runtime. That isn't a lot and it isn't supposed to be. It's a bridge.So when somebody tells you a site is 2N, the only honest follow-up is: 2N at which layer?Five days before any of this, they ran a quarterly load test on the generators. A real test, under load. And that test found a problem. So they did the right thing — they scheduled the repair. But to do the repair they had to physically open the breaker on feed B and lock it out, which they did properly: keyed interlock, locked out, tagged out.And that is the moment the building stopped being 2N on the utility side. For several days — Google doesn't publish exactly how many — that site ran on one utility feed. Google's own words for it are a temporary single-feed, 1N, bridged maintenance posture.Then the cable broke. An underground cable out on the utility network, outside the building entirely, developed a fault. And it was on feed A. The only feed that was live.WHY THE GENERATORS NEVER STARTEDThe PLC decides when to start the generators, and it decides it by sensing voltage. If the voltage goes away, start the generators. Simple enough.But those voltage sensors were wired upstream of the feed breakers. On the line side. And feed B — the one that was open for repairs — was still energized upstream, all the way up to that open breaker. The power company was still feeding it. It just wasn't connected to the building anymore.So the PLC looks at its sensor, sees voltage on feed B, concludes that a healthy utility source is available, and never issues a start command.Google's own root cause is this: a misconfiguration of the PLC logic that did not verify both an undervoltage condition and physical breaker position for both utility feeders. It checked whether there was voltage. It never checked whether that voltage was actually connected to anything.And here's the part worth a line in your notes. That PLC was dual-redundant. There were two of them. Both of them were wrong in exactly the same way, because they shared a configuration. Duplicating the box does not duplicate the judgment. That's common-mode failure, and it is the reason a redundancy count is not a resilience measurement.Five minutes later the batteries were done and the data halls went dark.TWO NUMBERS, KEPT APARTAbout ninety minutes with the data halls unpowered. Six hours and eight minutes before the services on top of them were whole again. Both figures are in Google's report and they measure different things. According to The Register, the companies caught in that window included Lloyds, HSBC, Royal Mail and Revolut — Google published no customer list and no customer count.To their credit, Google published the whole thing: the timeline, the wiring, the root cause, and a list of fixes with dates on them. One of those action items is the most honest line in the document. They committed to making PLC code review part of facility commissioning — which tells you it wasn't.THE MANAGER'S TAKEAWAYTwo things, and neither one is "test your generators."They DID test. The quarterly load test is what found the fault in the first place. What nobody did was look at the consequence: fixing that fault meant opening a breaker, and opening that breaker meant the building ran on one feed for days. As far as anybody treated it, that was a maintenance ticket. It wasn't a changed risk posture.And the second one, the one to carry into the exam room: an automatic control is only as good as the thing it can sense. The generator was perfect. The battery was perfect. The logic was asking...
No reviews yet
In the spirit of reconciliation, Audible acknowledges the Traditional Custodians of country throughout Australia and their connections to land, sea and community. We pay our respect to their elders past and present and extend that respect to all Aboriginal and Torres Strait Islander peoples today.