As part of emergency readiness, HCA has vastly improved our information storage and recovery capabilities.
HCA has long maintained the ability to recover IT services at remote facilities. In addition to protecting its Nashville Data Center, the company has facilities in Fort Worth and Orlando. It also maintains a small data center for Alaska Regional Hospital. Recovery capabilities for each of the data centers are located at a site in Philadelphia.

Early recovery systems relied on tape backups, which could take days or even weeks if an entire data center had to be recovered due to a catastrophic event. New technologies have made for more effective and faster recovery operations, and the company wanted to take advantage of that, said Brian Blackburn, AVP of Performance Management within Information Technologies & Services.

Philly site is recovery hub

“We historically had contracted with SunGard Recovery Services in Philadelphia to provide us with the equipment we need to recover our services in the event of a disaster,” Blackburn said. “We had been taking tape backups every night and sending them to off-site storage locations. If something happened in Nashville or the other data centers and we needed to restore all of our systems, then we would have to send those tapes and people to Philadelphia and begin the process of recovering our servers.”

There were a few problems with this setup. For one, the restoration could only be done up to the point of the last backup, so it would be possible to lose as much as a few hours to potentially a couple of days’ worth of information from key clinical and financial systems. Travel to Philadelphia for personnel and tape shipments would add to that time. So, a full recovery of a data center was typically measured in days, which doesn’t meet business requirements given the critical nature of many of HCA’s IT services.

But now, with the investment in new technologies, data is synchronized between the centers and recovery site effectively in real-time, and full recoveries are measured in hours, instead of days, for critical services.
“We have built out a recovery site, basically another data center, within SunGard’s facilities,” Blackburn explains. “We bought all the equipment we needed, so that the servers and storage we would need for recovery of our critical infrastructure would be in one place.”

The new recovery method keeps data updated between sites, and will soon be enhanced even more by a move to an Atlanta recovery site from its current home in Philadelphia. That move will be completed this summer, and will double the overall floor space to about 5,500 square feet. It also will have 550 kW of power, a significant power boost, and will operate at about 65 percent of current costs due to savings provided by the new location.

Efforts could expand

“We’re refreshing some key pieces of equipment, such as the storage environment,” Blackburn said. “We’ll be building our storage networks, which already are designed to carry our full load of data, to be able to do even more to keep up with anticipated demand.”

The recovery infrastructure primarily handles systems such as Meditech. However, with the increase in floor space and power capacity, other services could be considered such as those running within the divisions.

“We are very interested in leveraging the capacity of the new recovery location to enhance the disaster recovery capabilities throughout the enterprise, where appropriate,” he said. “Additionally, I see us working over the next five years to create active-active data centers, where we aren’t just replicating data but also balancing workloads across multiple physical locations and avoiding downtime due to disasters, where possible.”