Surviving a server crash: ‘Lose your data, lose your firm’

One day in 2014, a small firm in Littleton, Colorado, faced a disaster. Geris Zebarth’s servers, as well as its backup servers, completely failed, and for several hours, it looked like every scrap of data on every client had been irrevocably lost.

Enter Mike Stefanski.

At the time, Stefanski was an IT contractor for Geris Zebarth. A CPA himself, he said that he’s never been an IT professional, but has “dabbled in the geek arts for many years.” So he was called upon to see if he could help fix the mess.

At the time of failure, the server, which resided in its own room in the office, began making an “alarming, high-pitched tone,” Stefanski said. Because no one in the office had any real technical expertise, the solution was to shut the door. Unfortunately, that caused overheating, and the server failed.

It’s not that Geris Zebarth hadn’t done its due diligence in data security — the firm’s data was, in fact, stored in two places, with everything on the main server mirrored on another hard drive. But when they went to retrieve data from the backup, the firm discovered that the secondary hard drive had failed as well, without anyone knowing about it. (This is less rare than might you might think, especially if the drives aren’t being maintained properly and tested regularly). The firm had no third backup, in the cloud or otherwise.

AT120717 cybersecurity concerns chart

As Stefanski described it, the first failure was an alarm — but like many home smoke alarms, it wasn’t taken seriously. When the backup failed, the reality set in: With no data, there is no firm.

“The server was not brand-new, so parts and drivers and the like were not in ready supply, even from Dell,” Stefanski recalled. “I began what became a 48-working-hour recovery process to get the hardware and drives back up to spec and get everything back on. I got it back up and running out of sheer luck and some skill.”

As luck would have it, the owner of the firm, Mark Geris, was out of the country at the time. Stefanski felt like the entire firm’s fate, and the livelihood of everyone in it, was in his hands.

The solution Stefanski knew the firm needed was cloud hosting. He had heard about CrashPlan, which provides data backup services for small companies, through some industry contacts, so in this emergency situation, with little time to consider a range of options, he called the company up and said he needed help ASAP.

“Before I took myself off the clock, part of my billable time was to get CrashPlan on without even asking the question,” Stefanski said. “There was no other option. They weren’t going to work another day without a capable backup. If you lose your data, you lose your firm.”

“Since then, CrashPlan has run quietly and without problems in the background,” Stefanski said.

CrashPlan for Small Business, like other cloud-hosting products, backs up a firm’s data on physical servers located off-site. Such providers usually have multiple locations and backups, located in different geographical regions to safeguard against natural disasters as well. Now, Geris Zebarth has a new onsite server, as well as some hard drive backups again, but if they ever fail, CrashPlan can populate their data back onto a new server immediately.

“I’m a firm believer that if you care about data, it should live in three places,” Stefanski said.

Four years later …

Today, Stefanski is a full-time accountant with Geris Zebarth. Every two weeks, he double-checks the configurations of the firm’s backup with CrashPlan, making sure that staff are allocating data to the correct folders and every piece of information the accountants are generating is being captured. Once a week, he gets an email from CrashPlan giving him the status of every device being backed up. It gives him the time-frame each device last communicated with the server, and the status of the backup. If those two time-frames are within a couple hours, that means everything is running as planned, so he just deletes the email.

On CrashPlan’s side, the company uses a proprietary file structure, which means that it’s not susceptible to widespread viruses or malware that are typically written to target known structures. Naturally, all data is encrypted, so if a malicious hacker gained access to the files, they wouldn’t be able to read the data.

Today, Geris Zebarth breathes a lot easier than it did four years ago. “If a client suffered an attack, we’d just wipe and restore data — no second thoughts, no debate,” Stefanski said unequivocally. “There’s no reason to even think about it, because we know it’s all [on CrashPlan], sitting there if we need it.”

For reprint and licensing requests for this article, click here.
Data and information management Cyber security Network security Device security Data security
MORE FROM ACCOUNTING TODAY