Millisecond outage nearly leads to disaster: 60 data centers with 1.5 GW capacity suddenly disconnected from the power grid

As it turned out recently, Virginia (USA) found itself on the brink of disaster last summer after 60 data centers, consuming a total of 1.5 GW, simultaneously lost power from the grid. This forced power grid operators to take emergency measures to avoid large-scale power outages for other consumers in the region, Datacenter Dynamics reports.

The incident, first reported by Reuters, is well-documented: data centers in Fairfax County were forced to switch en masse to backup generators due to equipment failures in the power grid. Grid operator PJM Interconnection and local utility Dominion Energy were forced to urgently reduce the amount of power coming into the grid from power plants or the systems would have shut down themselves from excess power. This could have potentially led to power outages across the state.

According to the North American Electric Reliability Corporation (NERC), the outage began on the evening of July 10, 2024. A 230-kV power line experienced a protection failure, which caused the entire system to malfunction. The result was a series of power outages lasting several milliseconds — enough for local data centers to temporarily switch to backup power systems.

Image source: dhahi alsaeedi/unspalsh.com

The NERC investigation found that when a series of outages occur within a short period of time, data centres do not switch back to the main power grid – they have to do so manually, and in this case the data centres ran on backup power for several hours. Although voltage “did not rise to levels that posed a risk to reliability”, grid operators still had to take steps to reduce the power supply to the grid.

Moreover, such issues are likely to become more common in the future, as demand for electricity for Virginia data centers only grows. In February, Dominion said it now has 40 GW of “contracted” data center capacity through December 2024 — an 88% increase from July.

Other problems are possible. NERC says the incident did not cause significant problems reconnecting large-capacity facilities, but there is potential for future problems. Last year, NERC set up a special team to monitor high-load situations and the impact on data center power infrastructure and other large power consumers. Such consumers already pose a problem and sometimes a danger to ordinary residents, as they increase the risk of fires.

Network operators and utilities should take steps to study the likely impact of large customers suddenly going offline and coming back online. NERC points out that data centers are getting bigger and using more power, but the grid is not designed to handle 1.5 GW of data center outages. At some point, the problem will become intractable if left unaddressed. Northern Virginia, where the outage occurred, is recognized as the largest data center market in the world.