Extreme hurricanes, tornados, droughts, and heat waves have affected the North American mainland for as far back as records go, and perhaps even from the dawn of time. The National Centers for Environmental Information keeps a log of billion dollar weather and climate disasters.
There were four severe storm events and two winter storm events in the first six months of 2018. Climatologists say the frequency is getting worse. The 2013 to 2017 annual average was 11.6 compared to 6.0 events between 1980 and 2017. What lessons can we learn from this?
The chances are high a data center is going to take a hit several times in its lifetime, particularly if it is on the Gulf of Mexico where Hurricane Harvey struck. The storm dumped over 40 inches of rain in many places as it moved over Texas, making it the wettest and costliest tropical hurricane on record. Yet the data loss in Houston was relatively insignificant. How did city data centers prevail over Harvey? What lessons should we learn?
Storm Proofing Data Centers Against Extreme Weather
A data center’s immediate worst threats are power cuts, power surges, water, wind, and lightning strikes. The first two are dealt with by running grid electricity instantaneously protected via UPS with lead-acid or lithium-ion battery banks to servers, and having generators handy to maintain them on full power as backup.
Waterproofing is a matter of choosing the right site, being above the 500-year flood mark, and investing money in a suitable structure that can never fall down, and keep the water where it belongs, outside. With these measures in place a data center should be able to prevail over a hurricane like Harvey while it roars outside.
One of the lessons demonstrated is the importance of placing the facility and associated substations on higher ground, where natural topography and road systems are likely to deflect inland and coastal flood waters. The data center should be three feet higher off the ground than indicated by the worst-possible scenario, and be fully protected against water intrusion.
One area that is often overlooked with respect to protection from water is the roofing system. Many datacenters have flat roofs with equipment that sits on the roof, and roof drains. Any type of roof penetration or flat roof presents a point that water may pool and seep into the facility providing unwanted water in critical areas. The most discerning operators ensure their facility has multiple layers of leak protection on the roof with zero roof penetrations and no equipment on the roof. Equipment on the roof, while effective in saving yard space, can become a major liability during strong winds – knocking loose sheet metal and components that can skip across the roof and cause multiple punctures in the roofing system thereby exposing the facility to widespread water penetration.
Wind and lightning strikes are also threats that were not significantly present during Hurricane Harvey. Widespread wind can cause trees to fall that can cause long term outages from the grid, which doubles the importance of not only backup generation, but fuel strategy to keep the generators running. Compound the issue of wind with water and accessibility, and a poorly planned datacenter may have trouble obtaining diesel resupply which can be the achilles heel of any operation. Having fuel resupply agreements that include dedicated resources such as fuel resupply tankers, and checking the drivability of fuel resupply routes are key criteria.
Lightning strikes can take down any buildings electrical utility service and cause devastating damage. The use of lightning protection systems is key, which can include a system of lightning rods located around the roof tied to a specialized grounding system. One often overlooked item includes inbound surge suppression on all exterior equipment to prevent lightning strikes from hitting outdoor equipment and the surge finding its way into the facility. UL Listed buildings require the use of both of these technologies in concert to achieve full lightning protection. Finally, the use of double online-conversion UPS systems may provide enhanced protection versus using standby online technologies. Being wary when selecting equipment and making sure you have everything properly protected will pay in the long term when catastrophe comes knocking.
Keeping the Data Flowing While the Hurricane Roars Outside
A data center is part of a larger network where information flows through cellphone towers and fiber-optic cables beyond its walls. If these fail it can keep running internally, but be unable to service its customers. Therefore, the installation should have 100% redundancy through a mirror clone elsewhere. Only then can it guarantee seamless service with a reasonable prospect of success, after applying lessons learned from previous storms.
Route diversity of interconnection and multiple carriers can be key, so that you are not relying on a single aggregation point or backhaul to another location. A common trait datacenter colocation providers is to house many carriers, often times in excess of 10 which give customers many options to choose from.
How Houston Data Centers Stood Up to Hurricane Harvey
DCD News reported on the hurricane as it happened, and tore through southeastern Texas leaving catastrophic flooding and tens of thousands of displaced people in its wake. Most Houston data centers kept going through a combination of preparation, redundancies and being away from flood areas, although this was down to good planning and not luck.
- Many facilities continued running smoothly without needing to fall back on alternate power sources, fiber links, generators and battery backup. Some reported a few disruptions but were able to continue streaming data unabated.
- One major data center was online but inaccessible to its oil, gas, energy, and production customers. This was owing to surrounding streets being closed due to flooding according to Houston Chronicle. The company confirmed its employees were safe, and that it was ensuring its customers had alternative strategies should the situation worsen.
- Other data centers remained online throughout the storm, thanks to lessons learned from historic hurricanes. These continued to provide retail and wholesale data colocation services to businesses along the Gulf Coast. One large Houston facility sheltered in a 185-mph wind-rated structure with chilled water cooling and access to more than a dozen network providers.
The DCD News report opens a window into how hurricanes impact consumers in a wider area, and how important it was Houston’s facilities were able to withstand the incredible force of 1,000-year storm Hurricane Harvey. There are more lessons to learn from what weather watchers warned would be the big one. Houston datacenters, in a sense, got lucky dealing only with the water and not a combination of water and wind.
An Insider’s View in the Eye of the Storm
At one facility employees sheltered in a multistory building. They knew they had left it too late to evacuate, but they were determined to sit the storm out with backup generators, bunks and showers. As it relented, their families joined them in one of the few warm, dry spots in Houston.
They welcomed U.S. Marshalls who used the facility as a control center until the storm passed over and normality gradually returned to Houston. Throughout this period the facility continued to perform.
When it was safe to do so, the data center people ventured outside. They saw routine returning as people emerged from shelters, turned on their cellphones, and reached out to friends and family. They may have never spared a thought for why points of sale were working when they restocked essential supplies.
America has come to rely on these services. Perhaps we should wonder what would happen if they were to fail. Much the same happened with Hurricane Irma. The system held, enabling life to return thanks to cellular and internet connectivity. We should spare a thought for the stalwarts who kept systems running.
Back on the Ground in the World of Business
The Seattle-based Uptime Institute rates dater centers against four tiers of resiliency. These are data center design, construction, management, and operations against extreme events such as storms, earthquakes and run-of-the-mill power failures. The institute’s rigorous tests begin with ‘pulling the plug and seeing what happens’. It can be said that grid reliability contributed greatly to Houston’s data centers prevailing over hurricane Harvey.
How Do You Know Whether Your IT Will Survive a Hurricane?
This question applies whether you use IT service or provide them. Network World is concerned some data centers are ill-prepared for extreme weather. It says this is an increasing concern for small and medium-size businesses, which now face the prospect of multiple-day power outages.
Network World wonders whether their own power and cooling infrastructures will be able to support short or long-term periods off grid. Batteries cannot last forever, and they need regular replacement before they fail. Some power management and UPS systems may be decades old, and only remembered when extreme weather threatens. The possibility of utilities and data centers failing simultaneously is no longer that remote.
We as an industry have passed the point where we hope our backup strategies work without really testing them. The future is wetter and windier than when we installed or signed up to them. It is time for datacenters and cloud providers to ensure reliability through properly designed redundancy and actual testing. This often falls out of the scope of traditional IT personnel as these tests are mostly within the disciplines of electrical and mechanical. Having expert staff in these areas is the hallmark of a good colocation provider, who has trained personnel focused entirely on reliability and their reputation is staked upon providing continuity day in and day out.
It has become extremely important for cities, cloud providers, and businesses to review their ability to survive an extreme weather event.
We Should Not Allow Houston’s Harvey Survival to Drop Our Guard
Houston’s data centers did a superb job of keeping city business information running. However, they relied on the reticulation infrastructure to deliver their services to their customers. These customers should ask themselves ‘what if’ the reticulation infrastructure fails. If we are above the flood level how will we keep operating?
A business without data is a shell that cannot function. We should never assume the impossible will not happen. Weather events are becoming more extreme and we must remain vigilant about our data. We should not forget the lessons of Hurricane Harvey. Because one day an even more extreme one may rise.
