What We Owe The Future of Houston

An open letter to the city of Houston from our CEO, Chris Hinkle

A Personal Note

Hurricane Beryl has tremendously impacted all our lives.

I waited a week to write this to allow my baser and reactionary instincts of anger to subside. It is important to allow for the delicate balance of enough time to pass so to approach these things with a clear presence of mind, but also a short enough time such that I retain a strong and visceral memory of the events that transpired.

I do not consider my thoughts entirely original; I am an aggregator of many ideas and books, and a survivor of the repeated disasters that have befallen our City so the aggregation is logic originated by many others, and my own small contributions of perspective and experience, towards the applied topic of Houston.

I hope to do good here, and if I fail in any of my arguments or do not properly articulate something, my wish is that this will at least be a contribution towards a larger discourse of people that can help enact positive change.

Why Now

Paraphrasing the following passages, and the structure of much of my argument from William Macaskill’s wonderful book, aptly named “What We Owe The Future”:

Periods of plasticity are times where ideas or events or institutions can take one of many forms, followed by a period of rigidity or ossification. The dynamic is like glassblowing: in one period the glass is still molten and malleable; it can be blown into one of many shapes. After it cools, it becomes rigid, and further change is impossible without remelting.

Plasticity frequently comes after a crisis.

For example, the US Constitution was written over just four months – a moment of great plasticity – and amended eleven times in its first six years of operation. After that, it became more rigid. Between 1804 and 1913, only three amendments were passed. Today, the constitution is again very rigid, its only been amended once in the last fifty years.

Another example, following World War II, the international community debated a variety of ways nuclear weapons could be governed. One proposal, put forward by the United States, was the Baruch Plan, according to which the United States would disband its nuclear weapons program and transfer its bombs to the UN to be destroyed. The USSR countered with the Gromyko Plan, which also proposed universal disarmament. Both of these plans failed, and its not clear that either ever had much of a chance, but it was clearly a time of much greater plasticity in nuclear governance than we see now. Today, the idea that the UN could control the mining of uranium seems entirely off the table.

I am glad I read this book for many reasons, but the parallel of nuclear weapons and opportunities for change stood out in my mind. Hurricanes produce a tremendous amount of energy and release them on the people in our community. The heat release of hurricanes has been said to be the equivalent of a 10-megaton nuclear bomb exploding every 20 minutes. The sheer force of the wind and power inside these storms is simply unlike anything else on the planet and must be taken seriously.

We have been hit by these storms which carry more energy than nuclear weapons. We will be hit again. We stand, after this hurricane, in a moment of self-awareness and great plasticity. What we do next defines us and will determine our communities future trajectory. The opportunity is tremendous.

Why it matters.

We don’t know what the future looks like. With regards to flooding, as we learned from Hurricane Harvey, the old 500-year events are now revised as the 100 year flooding. This is our second wind event this year. Uncertainty will have great impact on the decision making for future businesses and people to come here, as well as stay here. We must build confidence in reliability if we want to thrive as a community, and to ensure reliability we must have standards that meet the needs of the area now and in the future.

Insurance providers are charging more, leaving the state, and making it more difficult for Texans to live here and thrive here.

The net real economic impact is still being measured but has been estimated to exceed $2Bn net loss in Texas output. I would argue this accounting-based measure far under-estimates the real long-term impacts. To enact change, we need to think, as a city, beyond the next quarter or year or election cycle. We need to think long term.

I live here, and I came from somewhere else. We are a city of immigrants. My wife, who I met here, came here from another country. People voted with their feet to come to Texas. Horst Schulze says the moment we take our customers for granted; they leave for something better. We live in a free country. In a recent survey, three in ten Houstonians are considering moving to other Texas cities after enduring the severe weather this year. To under-react or run in the wrong direction is to set the stage for us to take our people for granted that have benefited the growth of Houston.

People’s wellbeing, both economically as well as mentally are affected. Truly, it is those most vulnerable among us who are affected the most: Hourly workers are likely to bear the brunt of Beryl’s economic punch – Houston Public Media, due to loss of wages, affecting small businesses with about 50% of applicants being denied FEMA assistance. Not to mention the wellbeing of people after living without power for a week. 1 in 3 people experience negative mental health effects after disasters in Houston, and one year after Harvey just 8% of those affected able to access mental health services: Vulnerability & Impacts | Understanding Houston. This is not isolated, 27% of respondents to the Tropical Storm Imelda survey experienced negative emotional changes, while 20% also experienced negative mental health outcomes as a result of the May 2019 storms.

The Value of The Future

Imagine hundreds of generations after us in our young city. What does an increasing wealth gap due to repeating disasters to the most vulnerable of us do long term? What is the indirect long-term damage of population reduction? What is the impact of repeated exposure to crisis and mental health events on our children, and their children, and their children’s children? Think of all the great things those people leaving and affected by mental health events in a compound sense, could have done for our community and our city.

Future people count, and the future is big, and it can be very good or very bad*

What we do next defines us.

Linemen and electrical workers are the smartest, hardest working, and best tradesmen I know. I have worked with CenterPoint extensively and can tell you that their team members and engineers care. Many of them are our neighbors. They are working with the standards, and hand that they have. Please treat them with kindness and understanding.

While the execution of the recovery was not perfect, distributed disaster response is incredibly difficult. A traditional concept of disaster response is that the availability of resources is not the issue, rather the marshalling and logistics of getting the resources to the right place over many different disasters always proves to be the largest challenge.

This is an important thing to discuss, and should be improved upon, however, rather than solely focus on how fast we get things back up (time to recovery), we most also focus on survivability of our infrastructure, and this is accomplished through engineering standards that meet the need of our market, as determined by us.

Historical code standards have not been dictated by true extremes. I disagree with this approach.
Nicolas Taleb, a world leading expert on risk theory, makes an interesting argument: Never cross a river that is on average 4 feet deep.

When hearing average of 4, most humans assume most datapoints are going to be close to 4, but it’s possible that most datapoints are closer to zero with a few outliers of 100 or more.
This kind of “intuition” can lead to disaster.

The same is true with Options in financial trading. Certain options have huge downside in case of a rare event, while having small upside in the case of nothing happening. Tales compares those with “picking up pennies in front of a steamroller”.

We as the citizens need to set the standards for any type of organization, for profit, governmental, or quasi-governmental, to adhere to. To not build appropriate standards is to essentially provide profit options with the risk largely borne of the city, its people, and its future, introducing a massive risk symmetry issue.

It’s often not the daily averages, but black swan or catastrophic events and how we respond the shape outcomes. Ignore this warning at your own risk.

This isn’t my first rodeo on infrastructure hardening in Houston. Here is what we, and others did, and why.

Over the course of building TRG in Houston, I went far beyond code minimum in many areas. This sometimes involves “fighting the accountants”. I fought and advocated for the right standards for our data center until they occurred. They go far beyond code minimum and other standards. Our detention pond is 70% beyond code minimum, which takes up way more of our property than it needs to. We also spent a lot of time, money, and effort on engineering. I am thankful for the contributions of the engineering department of Harris County being collaborative towards this goal. We put our generators indoors. We were the only colocation data center to do it. Why? To build trust and convey understanding of the needs of the market. It is my experience personally, and as the CEO of TRG, that has, and will continue to pay off.

Several Texas businesses have done an outstanding job at demonstrating understanding of these needs. Enchanted Rock, Ezee Fiber, TRG Datacenters, and HEB to name a few. Why are the grocery stores online when everything else is offline? Because a grocery store had the foresight to build stocking warehouses and build backup generators for every one of their stores in partnership with Enchanted Rock. Ezee Fiber is burying their lines. TRG is putting their generators indoors in hardened facilities. I work in varying capacities with all of these companies (directly or indirectly) and can only say that their levels of focus and preparedness are outstanding. Organizations like this and I am sure many others are great examples and advocates of what proper design can accomplish in Houston. I would say by and large these Texas based businesses have outperformed non-Texas based businesses in serving the Houston markets in spades and continue to do so.

How do we architect for the future?

On The Topic of First Principals

Disaster is an opportunity to build trust and confidence, so lets archetype the following design philosophical principals:

We build/design on the first principal of establishing trust.
We solve for the black swans, because our city is too dear to tolerate anything less.
We watch out for fallacious thinking with regards to risk, paying attention to risk symmetry of our infrastructure, our shareholders, and the markets we serve.

On the Topic of Governance

The current arrangement of CenterPoint as a for-profit quasi-governmental agency is tenuous if not managed appropriately. I would argue it’s a “worst of both worlds” scenario at the moment, but it can work. They work within the funding constraints that they have, which requires public approval, and optimize shareholder returns. It will require legislative support to create standards that they must follow that we feel like are appropriate to our market.

On The Topic of Infrastructure Requirements for Power

NOAA recommends that utility poles in coastal areas are able to withstand wind speeds of at least 145 MPH.

Taking a more extreme view, The Inland Wind Model shows sustained wind speeds from a category 5 hurricane pushing well into coastal areas with >145MPH winds beyond landfall, and 107-127 MPH winds past College Station.

Understanding the difference between sustained wind and 3 second gust is also important, and confusing for many. The 3 second gust calculation may be estimated as 1.28 times faster than the 1 minute sustained, yielding a need for 185 MPH for the near coast contour, and 156 MPH for the inland contour.

The Inland Wind Model and the Maximum Envelope Of Winds (noaa.gov)

The 30 year 3-second gust wind speed risk factor for Houston is shown as 166 MPH for southwest Houston and 152 MPH for the heights, with the risk going down only to 145 MPH by Spring:

And this is based on historical data. Hurricane Harvey taught us, and Houston has become a case study that, with new flood maps you have to look at where the puck is going, not where it is, when doing planning for climate.

So we are building for 500 year and 100 year flood plain planning. Why are we not doing at least 30 year planning for power outages? Speaking frankly, we are overdue for something much worse.

On The Topic of Service and Maintenance for Power Systems

Another important factor of distribution poles is their strength towards end of life, and what their lifetime is. Poles rot and pre-fail. It has been estimated that rotting can reduce the strength of the pole by >30%. So a de-rate should be applied, such that the wind speed requirements may be met at the end of the life of the pole (not just the beginning), and a lifetime plus reasonable testing protocol should be in place to ensure the poles are within the tolerable range.

We should look at best practices for tree and foliage management. I love our trees as well, but we need to be eyes open about what they mean with regards to proximity to pole lines. Should they be there at all? Can they coexist? Should the trees be cut short? Do the lines need to be underground if we want trees next to it? These are practical and unavoidable questions that must be asked.

On Benchmarks for Success

I am not going to attempt to dictate where a C1 pole should be, and where a fiberglass pole should be, nor what transmission lines should be buried or how to make a sub-station survivable within the wind speeds mentioned above. It is specific by area, and context.

What I will say is that we need to set a target and standard of outcome, as proposed above, that meets the needs of our market, and then solve towards that. The solutions should be comprehensive. We, collectively as a public, need to ask ourselves and then answer, should we be solving for 30-year wind events, and following NOAA recommendations? Should we be watching and solving for where the puck is going?

On The Topic of Standards, and Critical Load Designation for Telecom Providers and Data Centers

I did an entire article on “Why the internet went out in Houston” – you can learn more about that here, but the takeaway is this:

Standards for telecom providers such as distributed low end redundancy works in a lot of cities, but it’s not architected to be successful in Houston. If you have business critical needs, ask your carrier if they are popped in a data center for their gateways. Self-managed carrier grade facilities proved to not be sufficient for multiple reasons. Cell towers may benefit from natural gas solutions at the edge. Battery backup may work in some cities but is not appropriate for Houston.

The initial event is just the beginning of issues for the internet with a hurricane, with the worst coming 12 hours after the initial loss of power. Prepare accordingly for this.

Lingering and new impacts can occur for weeks afterwards due to recovery efforts and fiber cuts.
We heard reports that hospitals with power were entirely offline for their important EHR and ambulatory systems due to no internet. We at TRG think it’s time to re-frame what critical load means from a statutory designation perspective, and standards should be built that apply to carriers to build to meet the needs of Houston. Carriers and multi-tenant data centers play an incredibly important role in infrastructure, and to our knowledge the multi-tenant data centers in Houston, for the most part, performed without failure. The carriers need to meet the same standards and serve the market appropriately.

As the world continues to digitize, the internet is an increasingly important part of every aspect of our lives.

Why does my opinion matter?

Most importantly, I live here. I met my wife here. My children were born here. My extended family has moved here. I imagine my story is much like yours. Which is why your opinion also matters.
Professionally, my background in reliability through my time operating data centers, certifying with Uptime Institute power systems has given me a ground level view of the hardships of infrastructure in Houston relative to other markets. I have been operating here since 2018 and was under construction during Harvey and saw that / lived through that.

My studies are in electrical engineering, and to this day I deal with power systems in our data center environment, working closely with CenterPoint.

Prior to data centers, I spent a short amount of time at MIT working for Lincoln Laboratory at a group focused much on distributed disaster response. I also grew up around mission critical infrastructure, with both my father and grandfather working in the space, so I have had the benefit of many years of wisdom and to see how telecommunications, critical infrastructure, and reliability risks evolve over long periods of time.

With our business, we serve over 120 businesses including medical and critical infrastructure. While we are thankful to be entirely unaffected, it was frustrating for me to have stayed online for our clients yet see other parts of the city and telecom infrastructure that I would consider critical infrastructure go offline.

I am not the expert to solve all the problems presented by the grid. However, relating as a resident here, my hope is that I speak and articulate to the benefit of our community and on behalf of those who feel strongly but perhaps do not have the words to say, to help ask the right questions and hand a giant microphone to those who can answer.

It requires people like us to stand up and take an interest in our community and contribute whatever influence or expertise we have to put and keep ourselves, our family, our community, and our Country on the right track.

What should you do about it?

Share this with your utility company, local, city, and state representatives. Could individual cities and municipalities take the step forward and ask for something better for themselves and become a blueprint for others?

Summary and Specific Steps:

Standards need to be established for municipalities near the coast that address extreme events. These standards need to be legislated and then enforced and funded for TDSP (transmission and distribution service providers) to follow. Suggestions made above.
Standards for ISP providers need to be established for municipalities near the coast that address extreme events and integrated as code. Cell towers should be considered as well.
ERCOT needs to amend critical load designation to include multi-tenant data centers as well as ISP owned facilities. The world is interconnected and digital and while power is on, we cant have 911 centers not operating due to the internet being out.

We are at an extremely dynamic point in time of humanity as well as our city, and rather than doom and gloom it represents an extreme opportunity for affecting change for good. I would strongly urge you to read “What We Owe the Future”, and think about what you can do for the future.

Find something that interests you, that you can do good in, and pursue that. If that includes pursuing electrical engineering and being a voice to advocate for the next 30 years to make sure that things get done properly, and remind them of the past, then do it. Or running organizations, using political influence to change the priorities of an organization. In general, learn, build options, and then do good. Its okay to iterate on this and change, as long as you continue to pursue good.

Consider buying shares in CenterPoint and have a voice. Perhaps we should start a fund for the voice of Houston. We can get a board seat and assist to provide governance. If everyone in the Greater Houston Metro bought $100 of shares per month for 2 years, we would nearly own the entire thing.
Write your representatives, CenterPoint reps, and government officials, and send them this letter.

Let’s make 7/24 a reliability holiday – to celebrate lineman and infrastructure across power, telecom, etc. to also remind people our commitment and the importance of 24×7 uptime operations as well as provide additional awareness of upcoming reliability events such as hurricanes so that we don’t forget. How many lives can we save with this continuing commitment to awareness?

Do not lose the moment for change, for the community abroad, but most importantly yourself.