January 30, 2023

Disaster Diaries 3: Dawn of the Dead Disk

Disaster Diaries are true stories taken from our support logs. The diaries show how we work with IT support companies to improve their profitability and service to their customers. We do this by working together to rescue their clients when overwhelmed by disasters.


A financial trading company was set to complete a significant deal with a US client at 5.00pm GMT on a Tuesday afternoon. The team was working non-stop to make sure that everything was ready to deliver to their US counterpart by 4:00pm. Two hours before deadline and 3 hours before transmission, the database server suffered a 2 disk failure causing the RAID array to go offline. All of a sudden, the database froze, rendering all information inaccessible.   


Two things had to happen in under two hours.

  1. Get the RAID array running again with new, undamaged disks
  2. Restore all the data, including the morning’s transactions in time to get the completed terms and contracts into the hands of the US company.

The company’s IT service provider had implemented a cloud data backup strategy. Backups took place automatically overnight and management was convinced this was a sufficient measure to protect the company.

Rebuilding the RAID

The company CEO was told it would not take the RAID long to rebuild, but the estimated time to download the data onto the repaired array was at least two hours and that would be with the previous night’s data. That meant all the work that had been created during the day would be lost. Six hours of work simply evaporated and there was no way to remedy this in the 2 hours remaining before deadline.

While the server, security settings and data were restored by 3.45PM. There was insufficient time to rework the bid and the company was forced to ask for an extension. A competing company got the contract and the US counterpart never accepted another call from the financial service company.

The IT Support provider was held responsible for recommending a backup and recovery solution that wasn’t fit for purpose and the client exercised their option to exit their service agreement early and were not shy in sharing this information with the rest of the financial service community. The support company’s reputation was severely damaged in this lucrative vertical market.

Revenue growth index: Rock bottom!

In actual fact this IT Support company was our partner and had implemented DATAFORT’s Critical Care service 

at the financial service company. Even though the support company had the competency to deliver a good solution, partnering with DATAFORT allowed them to be certain their customers were cared for no matter what other concerns were had by their salaried engineers.

We provided their client with the ability to run a virtual instance of the damaged server as it existed 10 minutes before the failure. Therefore, there was practically no loss of data and the project team could continue to work to meet their deadline while the support company repaired the damaged production machine.

The US deal completed on time and the IT support company won the confidence of the customer and was able provide more services for the business on contract renewal. They were also able to use them as a reference site when their sales team embarked on a campaign to acquire new business.

Revenue growth index: 30% year on year increase!

Even if you have the technical capacity within your business to provide disaster recovery services, working with DATAFORT is like hiring expert engineers without having to pay an additional salary. 

What do you want for your business and customers?

Contact Business Development Director Marcie Terman to develop a relationship with DATAFORT. Learn more about how we will work with you to protect your customer’s while enhancing your business efficiency and reputation.