This Q&A with CoreSite leaders Ryan Murphy, Director of Data Center Operations for the Southeast region, and Dave Brennan, Vice President of Data Center Operations Programs, describes CoreSite's unique approach to data center technician training, with a special focus on preparation for hurricane seasons.
Ryan oversees CoreSite’s data centers in Atlanta, Miami and Orlando. Dave’s group focuses on operational excellence and runs CoreSite’s qualification program for technicians, technician training, the maintenance management system and reporting metrics.
How do you prepare for risk to infrastructure?
Dave: Our clients trust us to reduce risk to their operations. That starts with us understanding and preparing for risks to our data centers. To be as ready as possible, we use a matrix that plots potential risks against impacts, built from risk model information we’ve tracked over the past seven or eight years. The model allows us to customize technician training for recurring or avoidable risks in the data centers which, in the Southeast, includes hurricanes. We check inputs to the model to be sure each one is correct and necessary.
We have developed Technical Program Manager (TPM) teams as well as a program focused on data center technician training and development. We run “table-top” scenarios as well as drills based on likelihood and impact to our critical infrastructure, and we make sure people are following procedures step-by-step. This is done in all CoreSite data centers, with modifications to procedures based on the types of threats from Mother Nature you see in different parts of the country.
We believe that operational preparedness equals operational excellence, and operational excellence equals high uptime and availability across the network and data centers. I’ll brag a bit by saying that CoreSite’s uptime percentage in 2024 was seven 9s, which means 99.99999% availability. The industry uptime standard is five 9s, or 99.999%.
Tell us a little about hurricane preparations.
Ryan: With storms, there’s no such thing as being over-prepared. June 1st is the start of the hurricane season, but we train our staff, run drills and do all the systems maintenance and checks on backup equipment throughout the year. Then, when a storm is headed our way, we run pre-start assessments to ensure all equipment is functioning normally, in the event it’s needed.
And it’s not just about the equipment. Many customers send employees onsite to ride out the storms, so we support them, too. We will bring in extra water, organize food, test batteries and emergency equipment, prepare emergency procedures, evaluate and eliminate any external risks including debris at our property as well as surrounding properties.
An important part of emergency planning is resource allocation, scheduling the right resources to be onsite is very important, but less obvious is the importance of scheduling resources for post storm coverage. The storm itself is what most people focus on, but the hours and days after a storm carry its own risk as utilities are making adjustments and completing restoration efforts. Ensuring you have a plan for post-storm support is important to avoid risks associated with fatigue.
Some CoreSite data centers have an on-site well to supply emergency water to the evaporative cooling systems. In other locations, we bring in tanker trucks. Also, we have contracts with both primary fuel suppliers and emergency fuel suppliers that we activate during any natural disaster.
A “fun fact” about the Orlando data center (OR1) is that the backup power system is run on diesel generators, but OR1 also has gas turbine generators like you see on a commercial aircraft. Gas turbines can operate as a primary power source and can run for months on end before needing maintenance. Another unique feature of gas turbines is that they are designed to be flexible on fuel. We operate our turbines on Jet A (refined kerosene), but they can also operate on natural gas or diesel.
How do you get teams ready to respond?
Ryan: We have defined a couple types of training regimens. In a "tabletop" exercise, we map out an incident scenario on paper with the ops team in a conference room. As we talk through the situation, everyone speaks up as to what they would do or where they would go, but no one leaves the room. Tabletops help everyone think through roles and responsibilities, identify gaps and make adjustments.
In a drill, members of a TPM team go into a conference room, along with an observer or two. It’s kind of like a treasure hunt. We previously designated a high-risk event somewhere in the building and marked the issue with a paper “fault” label. Then we send emails simulating real-world inputs, such as alarms, to the technicians. They spread out to investigate the data center, collaborating with one another using walkie-talkies and deductive reasoning to find the problem. The outcomes of each drill are used as guides for the next drill.
Tabletops and drills also help us assess whether changes to standard response procedures are warranted. It’s no secret that operational errors are the most common mistake, and these can be virtually eliminated with training. We have a great chart showing a strong correlation between operator error reduction as drills and tabletop exercises increase in frequency. Last year, we conducted a total of 214 drills across CoreSite’s data centers.
Dave: Some customers contractually require a certain number of drills every year. They can log in through our portal to observe and to train their people at the same time. They also track the team’s communications. That way, customers can train their people to understand the language technicians use and also transfer what they learn to their on-premises environments.
We run drills for the CoreSite staff, but we can involve a customer’s data center resources upon request. One customer thought so highly of a drill that they engaged us to help them with their deployment. We set up the drill so that they could observe and train their people at the same time.
We also look at seasonality issues. In March or April, for example, we tailor drills for hot weather environments, such as a utility power outage due to an overstressed grid. In the fall, we run snow storm and cold-weather scenarios.
Let's discuss the importance of customer communications.
Ryan: As a company, we strive to be as communicative and transparent as possible. It gives customers peace of mind around maintaining their critical services. While data center operations are essential to our customers’ businesses, we know that we’re just one part of their business continuity plans. Proactive communication gives customers the best chance to successfully execute their plans, whether that is active failover, managing services through the Open Cloud Exchange®, or another option.
I’ll clarify for readers. The Open Cloud Exchange is a platform that you can use to order and control network services. OCX makes requesting and orchestrating connections and managing data between cloud providers, enterprises, service providers and others on the platform a matter of a few clicks. This ability to quickly establish interconnection or switch between data centers for redundancy comes in handy any time, although it’s especially valuable for redundancy and continuity planning.
As previously mentioned, communication is the key to success in emergency response. We notify customers that we’re kicking off our annual hurricane preparation process and the actions we’ll be taking. After a storm is named, our operational support center communicates with the customer regarding timing and expected impacts. The operations team and operational support center keep in touch, providing emergency response details related to power, cooling, fuel and so on. Our leadership Team will handle executive briefings, if the situation warrants. Those briefings can include the level of impact and restoration effort based on the incident, standard operating procedures and situational status.
We also get to know the leaders of task forces that control city and county emergency response. Many people don’t realize how vital data centers are to public safety. Most public communications, in one way or another, depend on the interconnection capabilities in data centers. And, we directly support hospitals, first responders and government agencies. When local task forces know this, they can ensure that CoreSite data centers are a priority for restoration if needed.
What are some of the lessons you’ve learned over the years?
Ryan. One lesson is about property maintenance. When a storm is coming in, we make sure that our property is clear of anything that could become airborne and damage infrastructure. In addition, we have learned that it is critical that we also inspect neighboring properties and help remove things like pallets or overfull dumpsters that can be hazards.
Another lesson is staffing for storm coverage. We identify staff who will ride out a storm on-site. We’re prepared to accommodate them for days. But staffing after the storm is just as important, so we have reinforcements ready to step in for people who’ve spent days at the data center. I talked a little about that before as well, but bring it up again to point out that although systems are important, the people running them are absolutely critical to operations.
Final thoughts?
Ryan: Hurricanes happen, and they’re big news. People often ask why providers put their data centers in hurricane-prone areas. The answer is simple; there is a trend in the data center industry where data centers are strategically being placed based on population density, unofficially this is known as the “eyeball network” data center trend. Simply said, data centers are located where they are most needed. The hyper-growth in population and business in Florida has created increased demand for data center services.
The truth is, here in Florida we’ve lived with hurricanes our entire lives. Homes and businesses have withstood the test of time. Operating data centers here includes some risk, but it’s a calculated risk that we are well-prepared to mitigate.
Dave: Preparing for hurricanes is intense. It involves risk assessment, hundreds of details, training, clear communications and much more. We can’t say we know everything, but we know a lot. Our training program is robust and adaptable, and we learn things every year that add to our understanding and knowledge of efficient, effective disaster response.
Know More
To learn more about CoreSite data centers, our services and commitment to customers, contact us.
We welcome the opportunity to learn about your business and data center requirements.
Ready to learn more about CoreSite's Orlando data center, OR1? Click here or on the image to get details!