Once is not enough — firms’ disaster recovery plans must be tested, tested and tested again
by Stuart Kahan
A disaster recovery plan test is not unlike the familiar grade school fire or disaster drill.
When students hear those repeating peals of the fire alarm, they automatically know how and where to exit the building in an orderly fashion. In the business world, the importance of testing and the frequency of the testing process are critical to ensure an organization’s survival should a disaster strike. Because of typical and customary changes within an organization, such as key staff turnover, changes in venue and changes in business focus, the plan needs to be tested and refined on a regular basis.
A plan that was appropriate for a company in 2002 may no longer be suitable for the company in 2004. It needs to adjust as the organization alters internally, as external and environmental factors vary, and as the world’s current events change, as well.
According to Robert Murray, principal in the accounting firm of Grassi & Co., in Lake Success, N.Y., after a plan is designed, it must be tested to ensure that it is feasible and appropriate for the organization at that point in time. “The testing process should highlight any deficiencies in the plan and, more important, in some cases, it should re-familiarize the plan participants with the recovery process and their specific roles during the disaster recovery period.”
Initial planning
Technology disaster recovery plans lay out the steps and strategies to be followed in the event of a technology disaster, whether that disaster involves a systems failure, network outage or loss of critical data. But these plans lose meaning if they are not tested. Until the disaster happens, disaster plans are largely theoretical.
Any assessment of a plan, therefore, should provide format and content guidelines for preparing effective disaster recovery test plans, so that one can validate and verify recovery strategies in the event that disasters occur and technology operations are disrupted.
The Disaster Recovery Journal reported that 94 percent of companies that experience a catastrophic data loss go out of business within two years, and 97 percent of respondents reported that their specific plans needed to be altered in light of the new realities of the attacks of Sept. 11, 2001.
Kenneth Cerini, partner in the firm of Cerini & Associates, in Islandia, N.Y., said that there are really two pieces to creating a disaster recovery plan. “The first part is the part that most people are already doing somewhat, i.e. backing up data and carrying it off site. Even so, we still see problems in this area where people don’t rotate in enough tapes — daily, weekly, monthly, annually — and they don’t test the data on a periodic basis to see if it is backing up correctly.”
He pointed out that the place where most people fall short is that they often don’t back up the programs and store the programs off site. “This is not a big deal for off-the-shelf software, but when you have customized software, losing your applications could cost you a tremendous amount of time and related funds. If you can’t access your data, you’re lost.”
Disaster recovery, he noted, is also important for non-electronic data. “If you maintain much of your data in paper format and there’s a fire, flood or other catastrophe, you probably won’t be able to recreate the data — work files, medical records, etc. With the expansion of the Internet and the low cost of memory, companies should be looking to convert much of their paper to electronic means, where it can be backed up automatically or accessed from anywhere — although you do have to make sure it is held in a secure site.”
As part of its array of client services, Grassi routinely designs and implements disaster recovery plans. In fact, according to founder and managing partner Lou Grassi, the firm’s comprehensive plans are designed to ensure that when a disaster strikes, its clients have maximum protection, that their recovery period is minimal, and that they are back up and running in the shortest time frame.
“As one would expect, the goal of any disaster recovery plan development process is and should always be a positive one,” Grassi said. “The reality of the actual activation of an existing disaster recovery plan, however, could be very different than these expected positive results.”
Test implementation
Once the plans are developed, then initial tests are conducted and any necessary modifications are made based on an analysis of the test results. It should be noted that the approach taken to testing the plans would depend, in large part, on the strategies decided to meet the requirements of the company. As these strategies are defined, specific testing procedures would then be developed to ensure that the resulting plans are correct.
Testing also helps to evaluate the staff’s ability to implement the plan quickly and effectively. The planning coordinator should develop a plan that is designed to test all elements against certain objectives. The test plan should also set forth scenarios and logistics. The scenario chosen may be a worst-case incident, or one that will likely occur. In other words, it should be realistic.
Most experts agree that testing should take place immediately after the plan is developed. Key personnel should be involved in such testing to enhance effectiveness and to provide experience. There should also be provisions to update and test the plan on an annual basis, or more frequently if major changes happen.
According to Murray, the testing process also varies depending on the complexity of the organization. “We work with our clients to test their systems by conducting a mock disaster drill. Our mock disaster drill assumes a disaster — usually a fire — has destroyed the client’s main home office in the middle of the night. The entire drill process begins with the initial phone call to the owner that the fire has engulfed the office and all is lost. The communication chain begins and all disaster team members are activated. Each and every aspect of the plan is tested.”
During the mock disaster drill, Murray said that each team member is required to take notes as they pass through the recovery process. “After the test has been completed, we meet with the disaster team to review all notes taken during the drill. It is based on this review and discussion that the disaster recovery plan will be refined and/or updated.”
Testing means exactly that: testing, testing and more testing. Most experts agree that there is no substitute for adequate testing, because it is the culmination of all the preparations. And it never ends, because the requirements never remain the same.
To many in the field, testing is like a vicious cycle. The more you test, the more you update your plan. And every time you update your plan, the more you need to test it. Remember, when something changes, you need to review not only your plan, but also your disaster recovery contract to make sure that the new requirements are included in the contract.
A disaster recovery plan quickly loses value if it is not regularly updated to account for changes in personnel and equipment, pointed out Stanley Weiner, partner-in-charge of management advisory services for New York-based accounting firm Cornick, Garber & Sandler. He maintained that a plan created by a CPA or an outside consulting company should be updated every six months during a meeting of the company’s crisis committee. It is also necessary to continually evaluate and update the items stored off-site in the company safe deposit boxes.
Murray emphasized that he works with the firm’s clients not only to develop their disaster recovery plans, but also to test, re-test and refine their plans on a regular basis. “In fact, we recommend a re-test every six months. Should there be a significant change within the company or with the world’s current events, we strongly recommend that the plan be re-tested more frequently. Since the 9/11 terrorist attacks here in New York City, many more of our regional clients have been testing and enhancing their disaster recovery plan systems more frequently and in greater detail.”
The bottom line: It’s not only important that every organization and individual have a plan for recovery, but that the plan itself be considered a working document that is updated on a regular basis.
An answer for continuity
“The most effective way to minimize the adverse effects of a disaster is to implement a proactive business continuity plan,” said Joe Harpaz, chief technology officer and co-founder of Immediatech, a provider of document management solutions. He added that companies now have an affordable way to manage and protect both paper-based documents and electronic files in order to ensure business continuity.
One way is with GoFileRoom, a Web-hosted document management service that provides companies state-of-the-art security and disaster recovery that is not available with either a paper-based or an in-house system. All documents then are hosted at IBM’s off-site data center to avert the loss of critical records.
If disaster strikes, the new facility will continue to provide GoFileRoom clients with round-the-clock uptime and availability, enabling them to access their files anytime, from anywhere. The system is constantly in a testing mode to ensure full functionality.
Never enough
Matson, Driscoll, and Damico, in Atlanta, is an international accounting and consulting firm specializing in loss accounting. “When an organization must evaluate losses or damages, leading adjusters, lawyers and business professionals want competent, experienced and professional accounting services to meet their complex needs,” said John Damico, founder, managing partner and chairman of the executive committee. “These professionals demand comprehensive insurance claims and damage evaluations that will provide a thorough review of the accounting issues, as well as the amounts at issue.”
Damico explained that, although MDD is basically a forensic accounting firm, it has considerable experience in testing arrangements for disaster recovery plans. In fact, it is presently one of the lead firms in the 9/11 work.
“Insurance carriers and corporations have us come in generally before any disaster to recognize what would happen after the fact,” he said. “We evaluate the adequacy of everything, including insurance coverage, to see if there is any weakness. In testing, we always look for hidden exposures that would occur. For example, we can expose problems if a supplier to a company had an accident. What happens then if a business or a supplier sustains a disaster? How is it handled? In that context, we have a catastrophic team that has been put together and that functions all over the world, so that we can go anywhere at any time.”
According to Damico — and to most of the people involved in creating and testing a comprehensive disaster recovery program — the plan itself should never be considered the “final version.” Rather, it must be viewed as the “most current version.”
In answer to the age-old question of how often a disaster recovery plan should be tested, Damico sees it in this light. “Three years? Five years? The answer, really, is neither. It is never enough!”
At your fingertips The American Institute of CPAs and the National Endowment for Financial Education have jointly developed a new, broad-based guide to help people who are affected by natural and man-made disasters recover from financial loss. “Disaster Recovery: A Guide to Financial Issues” will be offered free of charge by local American Red Cross chapters across the country as a public service to aid people who are affected by disasters. It is being offered as a public service of the AICPA, the AICPA Foundation, the American Red Cross and the NEFE. “Prior to this guide being produced, there were no comprehensive, easy-to-understand resources to guide people affected by disasters through the three key stages of recovery — the days immediately after a disaster, the months following that period and the future,” said Terry Sicilia, executive vice president of American Red Cross Disaster Services. “CPAs can play a central role in the process by providing much-needed financial planning advice so that families can recover as soon as possible.” The 32-page guide addresses several topics, such as defining the steps to take immediately following a disaster (e.g., restoring stability, managing an injury or disability, and financial decisions after death); identifying steps to take in the weeks and months after the disaster has occurred to help people affected by disasters try to settle into a more normal routine by establishing a steady flow of income, handling expenses and debt, and working through potential lawsuits or other settlements; and illustrating the steps to take in planning for the future and moving on in areas such as assessing financial needs, getting retrained, and estate planning. “For people who’ve suffered because of a disaster, the right kind of advice and guidance is even more critical,” noted Anat Kendal, AICPA director of financial planning. “This guide outlines the steps people affected by disasters should take to recover. And, more important, it includes a checklist of proactive steps to plan for in the event of a possible future disaster in order to minimize the risk of loss.” To request a copy of the guide, visit www.redcross.org/services/disaster/beprepared/financeprep.html. |
