Disaster Diaries 2: Submerged Server Fiasco
Disaster Diaries are true stories taken from our support logs. The diaries show how we work with IT support companies to improve their profitability. We do this by working together to rescue their clients when overwhelmed by disasters.
Two days’ non-stop rain and the Thames is the highest it’s been in 100 years.
Apex Solar, a high tech start-up has a Monday morning meeting pivotal to securing a new round of venture capital funding. The management team had called a Saturday meeting to discuss Monday’s strategy.
After non-stop rain Thursday and Friday, the team arrived at the office Saturday morning only to be stopped by a barricade at the entrance to the business estate. Beyond the barricade cars floated amidst upended trees and other debris that had spilled over the banks of the swollen river. No light could be seen in any of the buildings. It was clearly too dangerous to enter the property.
Having the meeting take place at another location was no problem. But all the supporting information needed for the meeting was located on the file server. Where was the file server? The company’s server room was located in the basement and the server was now under, no one knew how much water.
Any show of weakness and the deal is off.
The company faced 2 challenges. Demonstrate to the investment company there was no problem meeting their obligations regardless of the flood. The second, equally important challenge was to keep office productivity on track even if their premises were under water. Apex’s management called their IT Support company to find out, in the first instance, how to get the data that was needed for Monday’s meeting. Second, how to keep business running as usual.>
Two alternative scenarios:
Which would you choose?
The Diverse Backup Strategy
The IT Support company had setup a backup process based on a series of actions mostly carried out by Apex staff. A strategy designed with multiple processes was meant to make sure that at least one captured a backup that could be viable if a recovery was needed. So nightly backups were taken to USB disk and critical files and data bases were backed up to the cloud using a file backup service.
Power to the building failed at 19:26 Friday evening when water breached the supply cabinet outside the Apex office...
The cloud backup had only partially completed and the disk to disk backup was underwater with the servers. So the only available copy was the previous week’s full backup. The task fell to the IT support company to download data needed for Monday’s meeting onto a laptop (if possible). The fact that 30% of the support company’s customers were located in similar Thames side business parks was going to make for a hellish weekend for everyone.
Aggravation index: Off the charts!
This is what really happened
DATAFORT Critical Care protects this dynamic, high transaction business. So, even though the backup appliance was water logged along with all the other servers, everything including all of Friday’s information was available from the virtual server images running in the cloud.
Apex management contacted their IT provider and the DATAFORT support desk logged a call from that team Saturday morning at 09:26. DATAFORT invoked the Apex network from the data centre at 09:52 on Saturday, which meant that by the time the Apex team sat down in the lounge of a local hotel to carry out their planning meeting, everyone could access anything they needed. It was as good as being in the office, with better coffee.
By Monday morning Apex was running from a nearby serviced office with laptops provided by their IT Support company. DATAFORT worked with them to make sure that each customer had a secure link to the virtual network and email traffic both in and out of the company was completely unaffected. A month later, DATAFORT and the support company managed the transfer back into the production environment using techniques that allowed work to continue without a huge break in productivity.
Aggravation index: No aggravation here!
What do you want for your business and customers?
Do you have a particular customer whose recovery strategy is giving you concern? Call or email Business Development Director Marcie Terman for more information on how we will work with you to protect your customers while enhancing your business efficiency and reputation.