May 29, 2017

Disaster Diaries 4: When Employees Attack

Disaster Diaries are true stories taken from our channel partner support logs. They tell a terrible story that might have destroyed one of our partner's customers and then we tell what actually happened because of high quality backup and recovery services provided by DATAFORT. Later we’ll discuss how we can work with you to provide the same level of care for your precious customers.

Situation:

Food Nation, an up and coming natural food company was in the process of transitioning from selling their products in health food shops to selling through supermarkets. Making the jump from specialist retail to mainstream had taken years to engineer and the company's management knew that having a rock steady supply chain and process were critical. As a matter of fact during the supermarket's submission process they were asked to detail their business continuity plan to make sure it was up to the supermarket’s standards. Food Nation management was particularly vigilant against missing deadlines and orders were triple checked prior to shipping. After all there were dozens of small suppliers waiting for their chance at supermarket shelf space.

Office manager Helen Ridgeway winced as the door to her office rattled in its frame. She had just terminated Artie the junior who had been hired 5 months earlier to work in the marketing department. In actual fact, Artie had been hired to create a social networking strategy for the company but did little more than Tweet his mates and avoid doing any task he considered boring. Before the meeting to sack Artie, Helen had called their IT support company Alpha/Omega IT and asked that they suspend Artie’s computer password. She was really glad she did because Artie did not take the news of his dismissal well.

Artie slouched back to his desk, threw his phone into his ruck sack along with a handful of company pens and his favourite mug. But before he slammed out of the office for the last time he decided to leave his calling card at the company that had just “screwed” him.

Artie sat back in his swivel chair and hunched over his keyboard. He typed in his user name and password, but could not get access to his computer. He tried again and got the same reaction. “They locked me out!” he muttered to himself in surprise, then paused momentarily to peer around him to see if anyone in the office was looking his way. The open plan office seated only 5 other people and most were getting ready to go home at the end of the day. Artie hooked his foot around the leg of the desk just across from his and pulled his swivel chair across the aisle. The computer at this desk was logged in so he opened up Windows Explorer and carefully searched until he found what he was looking for. He crossed his fingers and clicked the delete key. 

"Damn right!” he thought.

And then, because no one was looking he opened a browser to update his Facebook page.

Fifteen minutes later, Artie pushed past Helen as she was balancing her purse and laptop trying to open the metal door at the foot of the stairs that let out onto a side street. Helen had to jump out of the way to keep Artie from barrelling into her as he spat on the ground and headed down the street away from her and the offices of Food Nation for the last time.

Helen had a weird, bad feeling that confused her as she opened the door to her Prius. As she slid behind the steering wheel she started thinking about dinner and…

Challenge

Artie’s parting gift to Food Nation was discovered the next day at 08:00 when the shipping department turned on the monitor to start processing the day’s orders. They could open the ERP application but when they tried to access any of the order or stock screens the system crashed.

"It’s never done that before” said Fran a box cutter in one hand, packing tape in the other. “What do you think is wrong?” She asked Dorothy the MD who was standing beside her.

“Reboot the server,“ Dorothy calmly responded. And when that didn’t work she called Alpha/Omega to find out what was wrong?

An engineer from Alpha/Omega was dispatched immediately to the office to help Food Nation figure out what was wrong? The technician didn’t know much about the ERP system they used but he could see from the logs that the system was operating as expected yesterday and that the problem developed first thing in the morning.

“I could have told you that” Dorothy said. Weekly full backups would allow recovery of the software including any DLLs that might have gotten corrupted. Then the previous night’s offsite data backup could be layered on top which should bring everything back into alignment. The technician started the recovery immediately and brought the system back online by 3PM. He was packing up his kit at about 3:30 when the MD walked into the room scowling.

“Look, we’re sure there is data still missing,” said Dorothy, now not so calm.

“What data would you like me to recover?” responded the tech, “Recover everything,“ responded Dorothy. “Whatever you’ve got.”

Thinking about their glitchy ADSL connection the technician replied, “that will take hours.”  “Or maybe days, I’m not sure.”

Dorothy felt sick as a realisation hit her. They had no idea what was missing. She was now fairly sure, after the sheepish conversation with Helen that Artie had purposely caused this disaster. Now her company was at risk and she could not think of how to figure out what had been deleted from the servers? She also realised she wasn’t confident what was actually being backed up? Some folders were targeted for offsite backup. Some disks were meant to be backed up to other disks and taken out of the office but there really was no central plan and Dorothy had always assumed it was going to be ‘good enough.’ She knew it wasn’t really Alpha/Omega’s fault. They had provided her with what she asked for and when the monthly invoices for offsite backup went up last year, they had removed lots of folders from the backup list.  

No orders had gone out that day.

Everyone was focused on the disaster and Dorothy had a tour scheduled with the supermarket chain relationship management team.

Emotion index: Despair!

An engineer from Alpha/Omega was dispatched immediately to the office to help Food Nation figure out what was wrong? The technician didn’t know much about the ERP system they used but he could see from the logs that the system was operating as expected yesterday and that the problem developed first thing in the morning.

“I could have told you that” Dorothy said. Weekly full backups would allow recovery of the software including any DLLs that might have gotten corrupted. Then the previous night’s offsite data backup could be layered on top which should bring everything back into alignment. The technician started the recovery immediately and brought the system back online by 3PM. He was packing up his kit at about 3:30 when the MD walked into the room scowling.

“Look, we’re sure there is data still missing,” said Dorothy, now not so calm.

“What data would you like me to recover?” responded the tech, “Recover everything,“ responded Dorothy. “Whatever you’ve got.”

Thinking about their glitchy ADSL connection the technician replied, “that will take hours.”  “Or maybe days, I’m not sure.”

Dorothy felt sick as a realisation hit her. They had no idea what was missing. She was now fairly sure, after the sheepish conversation with Helen that Artie had purposely caused this disaster. Now her company was at risk and she could not think of how to figure out what had been deleted from the servers? She also realised she wasn’t confident what was actually being backed up? Some folders were targeted for offsite backup. Some disks were meant to be backed up to other disks and taken out of the office but there really was no central plan and Dorothy had always assumed it was going to be ‘good enough.’ She knew it wasn’t really Alpha/Omega’s fault. They had provided her with what she asked for and when the monthly invoices for offsite backup went up last year, they had removed lots of folders from the backup list.  

No orders had gone out that day.

Everyone was focused on the disaster and Dorothy had a tour scheduled with the supermarket chain relationship management team.

Emotion index: Despair!


What do you want for your business and customers?

Contact Business Development Director Marcie Terman to develop a relationship with DATAFORT. Learn more about how we will work with you to protect your customer’s while enhancing your business efficiency and reputation.