“Our incremental backups take less than eight minutes, it is incredible!”
Company size: 300-bed non-profit hospital
Industry: Healthcare
The Dasher engineering team was called in to help one of our clients re-architect their backup and restore environment. This blog post is a case study of that project. In order to protect the security of our client, we will call this company “Regional Hospital.”
About Regional Hospital
Regional Hospital is a medium-sized hospital with a level II trauma center and neonatal intensive care unit. A wide array of medical services are provided including full surgery, heart care, orthopedics, cancer care, and maternity care. Like most hospitals, this regional hospital leverages hundreds of IT systems to enable its caregivers to provide excellent care. Of course, the availability and reliability of these IT systems are critical to the daily operations of the organization.
The business challenge: “Our backups are not reliable!”
Regional Hospital’s legacy data protection system was not capable of protecting the data of 200+ applications in a reliable or timely manner. The solution was designed in the early 2000’s before the hospital went through a 10 year cycle of rapid growth. Regional could not keep up with the growing business requirements and changing IT security landscape of a modern hospital.
The hospital was experiencing the following issues with their outdated backup solution:
- Backups ran on the same network as the primary applications and were impacting the performance of production IT solutions.
- Backup jobs hung or crashed causing database transaction logs to fill up in the middle of the night. The client complained of required late night IT intervention to “clean up the logs so the backups could be completed.”
- File restores could take one-to-two days to complete and impacted business effectiveness.
- As application data grew, the system was not easily scalable.
- Two full-time staff members were needed to manage the backup environment, rather than the goal of one part-time person. This added unnecessary cost to the overall solution.
- The hospital depended on the failing backup environment to protect it from emerging cybersecurity issues such as cryptolocker attacks. The backup environment alone was not able to offer the protection needed.
Requirements gathering: “What do you want in a 21st Century backup solution?”
Together, the hospital staff and Dasher Technologies, architected, prototyped, and implemented a new data protection system that addressed all of the business problems noted above — and then some! Dasher provided a review of industry solutions to see which solution would be the best fit for the hospital. Detailed requirements were gathered including the following items:
- What is the current network design? A new backup environment will impact the network!
- What is your desired backup window?
- What is your desired recovery window?
- What is your recovery time objectives (RTO)?
- What is your recovery point objectives (RPO)?
- What are your application and data growth projections for the next five to 10 years?
- What are the operating systems types and version numbers?
- Are we backing up desktops or just servers and applications?
- Do you know how much new data is created daily? If not, we have tools to analyze this.
- Do you know what the data change rate is? If not, we have tools to analyze this.
- Do you know the nature of the majority of the data being backed up? For example: images or video do not compress well — so we need to know this for proper sizing. If not, we have tools to analyze this.
- What storage array systems are in place?
- What percentage of your environment is virtualized?
- What are your IT operational goals for the backup environment?
- Is tape still required for local or off-site long term retention?
- Is backup to a public cloud an option?
- What level of staff expertise do you currently have and what training will be required to ensure your team is ready to operationalize the solution?
- Is data at-rest encryption required?
- Is WORM (write once read many) technology required?
- Do you have a standard in place for IT runbooks that you want Dasher to update?
- Who are the key stakeholders for the design and implementation and project management during the implementation phase?
- Do you require application level backups such as databases or email?
- What are the key applications being protected?
- What retention would you like for local disk backups enabling quick restore of data?
- Are there any solutions you have already investigated that you like or do not like?
- Are there compliance requirements (such as HIPAA)?
The solution: “Let’s design a 21st century backup solution”
Once we interviewed this client, with several meetings and calls to clarify the requirements, Dasher installed tools to enable us to confirm the data provided by the hospital. These tools classify the data, rate of change, and the impact to the network that enabled Dasher to architect a solution that addresses current and future business requirements of the hospital.
Dasher designed the following solution based on HPE hardware, Veeam software, and VMware software. The diagram below shows the architecture and data flow through the environment. Some of the items were already in place and others were added as part of the new architecture. Our solution met all of the client requirements and was deployed with Dasher solution architects in a matter of a few short weeks. The implementation included final testing and side-by-side backups as the existing backups were phased out of production. The heart of this solution leveraged the Veeam Scale Out Backup Repository deployment model.
The solution provides the following outcomes:
- Frees up IT staff to work on proactive IT projects, not management of backups
- Local access to files stored on disk for fast recovery
- Off-site tape storage for long term retention
- The potential for backup-to-cloud if the hospital would like that in the future
- Full and granular VMware recovery
- Full file and disk recovery
- Microsoft active directory protection and recovery
- Microsoft Exchange & SQL protection and recovery
- Isolated FC SAN for backups only – removing backup traffic from the primary network
- 99% of backup data now moves over SAN
- The hospital is proactively testing the restore of backup data to prove backup validity
- The hospital is now testing self-restore for some end users
Dasher designed, implemented, and designed a custom IT solution based on HPE hardware, Veeam software, and VMware software. This diagram illustrates the architecture and data flow through the hospital’s environment.
The Results: “Backups, and more importantly, restores just work!”
Look at all of these great quotes from the hospital IT staff:
- “We took control of our data protection environment!”
- “We get to sleep through the night now.”
- “No more daily meetings about how to fix our backups.”
- “We have a new sense of confidence that our data integrity has been regained.”
- “We now have 100% backup success rate of over 130TB of data amongst 600 physical and virtual servers.”
- “We are actually completing a full backup in less than one hour, this used to take days.”
- “Our incremental backups take less than eight minutes, it is incredible!”
Financials matter
In addition to all of the positive client feedback, the hospital’s executive team is ecstatic with the new backup solution. It cost significantly less to architect and deploy the new solution than it would have cost to pay the support renewal fees for all of the hardware and software that was currently in place prior to the backup modernization project. This fact, coupled with the greatly improved operational efficiencies gained (i.e. freeing up staff resources), made the IT executive team heroes to the CEO and board of directors.
In summary, the hospital IT staff is now excited to be able to restore files quickly for end users and bring value to the business. What used to take a day or more, involved many steps, and was always done with fingers crossed, is now completed in a matter of minutes with a few mouse clicks. The IT staff are now backup/restore heroes, not villains!