Everyone has a story of the email they accidentally sent to the wrong recipient, or the computer that crashed and lost an entire day’s worth of work.The disaster recovery industry has proven to be a lucrative one in the age of information, and recent snafus at the Alaska Department of Revenue and the Canada Revenue Agency only serve to underscore that problems can happen -- and illustrate the different ways in which an organization can be prepared for the worst-case scenario.
In Alaska, the state controls the Alaska Permanent Fund, an appropriation unit that invests proceeds from the sale of minerals (oil reserves) and other state investments, and then redistributes some of the wealth among state residents. Most years, Alaskans who have lived in the state for a calendar year can expect an annual check in the neighborhood of $1,000 -- not a bad perk.
But this summer, in the course of reformatting a disk drive during a routine maintenance check, a computer technician mistakenly reformatted the fund’s entire back-up drive, in the process losing about 800,000 electronic images that had been scanned into the system. Those images contained the 2006 paper applications that Alaskans had submitted via the mail, or in person, as well as supporting documentation such as birth certificates and proof of residence.
The worst-case scenario didn’t present itself until the fund went to its back-up tapes, updated nightly, only to discover they tapes were unreadable. And what of the back up, to the back up? The original paperwork, which was stored in more than 300 boxes.
Just last week, the Canada Revenue Agency announced that Canadians would be able to resume e-filing their tax returns, after a computer glitch discovered last week put the processing of more than 1 million tax returns on hold.
The agency hasn’t provided too many details about the problem, other than to announce that the malfunctioning software patch was not the work of hackers, or a computer virus, and had not threatened the security or privacy of taxpayer data. The agency had to shut down all of its 75 databases either late in the night March 11, or early in the morning March 12, after the problem was discovered, which caused the processing of all returns, including paper-based returns submitted by mail, to be delayed.
Temporary workers were brought in to help the agency play catch-up in processing the backlog of returns, but it was obvious some sort of system was in place that the end-line customer wouldn’t bear much of an inconvenience, or cost.
In Alaska, poor back-up procedures ended up costing the department more than $200,000.
Seasonal workers came in to assist the regular division staff and about 70 people working overtime inputted all of the lost data back into the system by the end of August. More than 600,000 checks went out in last October and November to the public -- including about 30,000 residents whose applications were still under review when the data was wiped out.
Alaska officials have seemed proud of their response in the press, as well as the fact that there was no witch hunt to assign blame for the data loss.
But that’s probably more due to the fact that blame could have been spread far and wide throughout the department in a never-ending cycle of buck-passing. Part of the reason the incident is in the news is that the department is now asking lawmakers to approve a supplemental budget request for about $220,000 to cover the six-week recovery effort’s expenses -- including about $130,00 in overtime and $70,000 for computer consultants.
Alaska’s lucky that the money could, in theory, be pulled from the permanent fund’s dividend earnings, meaning that the cost can be spread among the state’s residents, to the tune of less than 40 cents each. The state’s lucky that it has the resources to cope with the unexpected expense, for a smaller enterprise, the burden could have been the death knell for its operations, as well as its livelihood
Not surprisingly, the Alaska officials say they now have a regularly tested back up and restore procedure. But $200,000 is still a tough way to learn its lesson of preparing for the worst.
