Identifying The Risk
Through its journey into the Yucatan peninsula and into the gulf, the TRG team tracked Hurricane Beryl hourly. Due to the angle of the Texas coast, we knew that a small “wobble” on the hurricanes track could cause a large movement up the coast.
As we progressed into the weekend, we saw an increasingly high likelihood of, at minimum, impact, and a growing threat of a direct hit. Utilizing the NOAA inland wind model and guidance from Space City Weather, we determined strong tropical storm force winds at TRG.
Enacting The Plan
We called in fuel resupply contracts on a discretionary basis on Saturday, to be scheduled to come Monday evening just in case power was lost by the grid to start the process, even though we have five days of fuel resupply on site. We also enacted our emergency plan and initiated a ride through team stocked with provisions and sleeping arrangements at the facility. We are fortunate to have our own well and septic system, complete with on site showers so we had absolutely zero dependency on any municipal resources.
As the hurricane approached the coast, we were lucky that the pervasive dry air delayed intensification. This could have easily turned into a much stronger storm. Even though we were not originally anticipating a stronger event, we did take all precautions mentioned above due to the possibility of rapid intensification.
As the storm came, its focus shifted on what was a direct path to our facility in Spring, TX. While we were fortunate to have not lost utility power for a meaningful period of time the entire event, largely in our belief due to the critical load designation of the facility as well as the location in newly developed area with higher quality electrical infrastructure, we did elect to go to generators as a precautionary measure from 6AM -8PM on Monday as conditions deteriorated, until such a time that they improved.
During this time, we were extremely thankful to have the generators built inside the building, which afforded us the opportunity to check them every 30 minutes during the event to ensure all things were running smoothly. The systems were well protected being built inside and everything performed exactly as was our goal.
Post-Storm Mobilization
After the storm passed we dared a breath for the evening, thankful that TRG was entirely unaffected by the storm. By morning however, we began to hear reports of individual carriers failing city-wide due to their less protected infrastructure. Many of whom had failed the day prior but had failed to communicate to customers and ourselves their issues.
By our count as many as 6 carriers had intermittent city-wide issues, with as many as 3-4 simultaneously (the timelines are difficult). The TRG team reacted quickly and began sending out notifications to clients about what we knew about various carriers, and we enacted a rapid cross connect strategy partnering with local TX hosting providers, MSP’s, and carriers Ezee Fiber, TXnet, and Netdepot and were able to assist in getting 17+ businesses online that had carriers that were affected.
This rapid mobilization by the team, and our stock of pre-terminated and armored cable allowed for us to react quickly and serve our market. We were thanked by many clients for TRG taking leadership in communication about the carriers when they weren’t getting the information they needed from the carriers themselves. We heard some reports of locations that rely on just 1 or 2 carriers being offline for extended periods of time, underlying some of the real value in being in a fiber rich data center such as TRG with 13 carriers.
For the rest of the week, we were able to be a haven for clients who were without power, utilizing our backup office spaces with power/hvac/cooling/internet. At some point in the process an espresso machine showed up at the facility, and our CEO Chris Hinkle was the only show in town for coffee, lifting spirits around the office with a good latte.
Ending the week, we realized our work had just begun. With businesses offline for indeterminate periods of time all over the city, TRG sprung into action with emergency moves and moved multiple clients over the weekend. We now triple our efforts, continuing to provide emergency services to get people into a reliable data center so that they don’t have to worry about storms affecting their IT anymore.
The Results & Lessons Learned
Hurricane Beryl was a big wake up call for many, and a terrible event. TRG remains extremely focused on servicing our market and supporting Houston getting back, and staying, online.
The biggest shortcoming that we saw was the reliability of various carrier infrastructure around the city, as well as broadly speaking the reliability of the grid (although we were lucky enough to not be affected). TRG is producing an open letter to Texas legislators and Centerpoint regarding grid improvements as well as critical designation for all multi-tenant data centers and telecom providers, along with standards for carriers to follow.
We heard stories of life critical hospitals without network services, its time we enforce standards on not only the power companies but also telecom providers and also recognize the true value of data centers as infrastructure in our community.
