For the past month, we’ve been working with (or against, depending on your viewpoint) AT&T to troubleshoot a failing router. Unpredictably, the fiber link light goes dark, and an office is suddenly without internet. Crippling. Our multiple requests for an onsite AT&T technician were disregarded because the Midwest AT&T service team had been sent down to Houston to rebuild infrastructure.
Perspective. Talk about crippling.
Hurricane Harvey, and the impending Irma undeniably destroy homes, buildings, streets, cars and on and on. But what about data centers, fiber lines, and websites? Any interruption to data centers would create a ripple effect of downtime throughout the nation. I started researching. Remarkably, Houston data centers were able to “withstand Hurricane Harvey.” IBM, Telco and ISP Level 3 are all up, and did not experience major service disruption. Typically flooding causes electrical outages, forcing data centers to rely on generators. But if fuel becomes limited or pumps are unreachable, power goes out. Servers shut down. Websites are unavailable.
In 2012 Hurricane Sandy flooded New York, and took down at least 8 data centers. One of these, Datagram, was down for 4 days before restoring generator power, and another month before ComEd restored power to the building. And without power, sites like Gawker and HuffingtonPost were also down. So how does it happen? In Sandy’s case, ComEd had cut power to Lower Manhattan in preparation of the storm, so Datagram was forced to rely on generator power almost immediately. Hours into the storm, the center went dark. Even though it was located on the 25th Floor, basement fuel pumps were damaged by flooding, and could not be repaired until the entire electrical infrastructure was checked.
Other data centers suffered similar challenges. Equinix attempted to lighten the load on its generators after air found its way into the fuel pump system. This resulted in a loss of cooling capacity, raising temperatures over 100F. Systems started to overheat, and shutdown. Employees panicked, and setup rudimentary methods: floor fans and open windows.
And then there’s the cost in all this. The business cost of downtime is one thing, but there’s a cost of materials, too. The fuel to run generators to keep a data center online is estimated at over $10,000 per day. Datagram was running on generator power for nearly a month. Most client contract SLAs are simply ignored (force majeure). It’s truly a disaster.
Fortunately, Houston’s data centers weathered the storm. Physical center locations are on high ground, they are staffed 24/7, and stocked with food, water and fuel. This type of planning is becoming a necessity for our online expectations. Hard lesson learned.