Disaster Recovery: The Time Is Now

• By Dian Schaffhauser
• 10/21/05

On the heels of Katrina, it's time to get a top-flight disaster recovery plan into place. Here's how to catch up.

AS THE WATERS RECEDE, Tulane technologists bring systems back on line and look for new ways to protect, secure, and prepare them for crises.

Within 48 hours of Hurricane Katrina battering the Southeast, only two of the many four-year universities in New Orleans had Web sites up. The others showed “failure to connect” messages to students, parents, family members, friends, faculty, staff, or others trying to learn about the state of the schools in which they had a vested interest.

Two weeks later, the silent schools showed signs of life online, albeit with rough edges: links that went nowhere and highly visible content that was dated. On its home page, one Catholic university mourned the passing of the Pope and shared the schedule for April 2005 memorial masses. Another university, whose buildings sustained flood damage up to the second floor, advertised on its home page for data entry operators to aid in hurricane recovery efforts from its temporary headquarters in Baton Rouge.

And what of the two schools that remained viable in some form almost throughout the duration of the storm? One, Tulane University (LA), ran a blog updated daily by President Scott Cowen, keeping visitors abreast of the safety of faculty, staff, and students and of conditions at its campuses. It was hosted on an emergency Web site, which the university’s normal URL redirected site visitors to. After the disaster, Tulane moved temporary operations to Houston and held online chats with Cowen so that he could respond to fears and questions from the public (no, emergency power hasn’t been maintained at the medical school, which means research has been disrupted; and, yes, formal rush for sororities will take place—in the springtime). The university has also provided a continuous and unbroken chain of updates via its Web site.

The New Orleans campus of Louisiana State University (with its main campus in Baton Rouge) also remained live online through most of the storm onslaught and disastrous aftermath. The institution posted an emergency notice to its Health Sciences Center Web site, lsuhsc.edu, stating that the academic campus for LSUHSC in downtown New Orleans had suspended operations and that “hospital personnel should follow Code Gray procedures and instructions from hospital leadership.” It also advised Blackberry ( www.blackberry.com) subscribers that they could use their devices as a means of e-mail communication, “in the event we lose our Exchange servers during Hurricane Katrina.”

When Is a Disaster Too Catastrophic?

Although many have argued that the proportions of the Katrina disaster (a Category 4/5 hurricane accompanied by no end of additional and devastating complications) went far beyond a scale that can be planned for, others insist that the tools of disaster recovery planning (DRP) aren’t necessarily prohibitive in terms of dollar or time investment, nor do they need to be highly complex. Keeping up a blog on an emergency Web site d'esn’t require complicated technology, for instance. And while populating a blog is a far cry from rebuilding research data that has been destroyed, on the heels of any disaster communication becomes vital: It is the link that holds survivors and supporters together while they take emotional and physical stock. In such times, reliable information becomes gold.

Days after Katrina’s passing, new LSU CIO Brian Voss wrote to Campus Technology, “Yesterday, I ‘suspended’ normal IT support activities (with the exception of our administrative systems and computing functions, which ‘run’ the university).” An absence of “normal users” on campus allowed him to redeploy his staff to help state and federal recovery agencies set up temporary phone and network operations on campus, and helped his staff to create “quick and dirty” applications to aid in information gathering and sharing. The LSU crew also helped colleagues at another school rebuild their Web site and start to recover e-mail services.

As Voss now observes, “All this means that, as a new CIO with lots of efforts underway to build the future of IT at LSU, I suddenly found that we were pulled into a more pressing, vital role to support the relief and response efforts on campus.” What lessons for CIOs (even for new CIOs) might be evident here?

“I suppose there are two,” says Voss. “First, if disaster strikes, be prepared to see your mission make a sudden shift, and hope that you’ve a capable staff willing to be flexible and adapt to the change—I am fortunate to have such a staff.” (CIOs might also hire with an eye to such flexibility.) “Second,” adds Voss, “disaster-recovery planning efforts in higher ed IT shops need to take on greater urgency in terms of resources devoted to their planning and preparation. The aftermath of a crisis is no time to try to recover ad hoc, especially when one considers that ramifications of a disaster can be far broader than the traditional ‘data center hit by a tornado.’”

In fact, besides hurricanes, in the last 20 years universities and colleges have faced deadly ice storms, earthquakes that leveled whole campus buildings, terrorist attacks, train derailments that required campus evacuation, animal activists taking over a research building, the potential presence of anthrax and SARS, and even the more seemingly mundane computer worms and viruses that haunt any network.

The fact of the matter is: Disasters encompass people and services. According to Ross Armstrong, senior research analyst for the Info-Tech Research Group (www.infotech.com), in the face of disaster, the role of those in charge of IT is to figure out how to get regular users— including the IT department itself—back up and running again. That challenge becomes an exercise in risk analysis, he points out, and the first question a CIO should ask himself is: What are the potential hazards we face? The second question is: What is to be brought up first, second, third, and so forth, and in what order?

Who’s ready, who’s not. Yet what surprises Armstrong is the number of organizations in general—regardless of industry—with either no plan in place, or merely the intention to implement such a plan over the next two to three years. Higher education is not immune to this ill: According to Info-Tech’s DRP in the Education Sector 2005 Benchmarking Report, a surprising 47 percent of universities and colleges currently have no disaster recovery plan in place. These institutions, do, however, acknowledge the importance of having such a plan: 68 percent of them say they are currently in the process of planning. Unfortunately, it may be some time before many college and university DRPs see implementation: 32 percent of schools with no current plan concede it may be up to three years before they have one in place, according to the report, and that may be because security and end user support are higher IT priorities than disaster recovery—just a notch above the categories of network/LAN/WAN, Web site and IT governance. But the good news is that, according to Info-Tech, among the 53 percent of schools currently with a plan in force, a whopping 86 percent are improving that plan. (For more DRP data, see “Lessons in Disaster,” page 14.)

The money factor. Another challenge unique to the university setting is the fact that faculties and departments receive different levels of funding, says Info-Tech senior research analyst Curtis Gittens, adding that this makes a comprehensive campuswide disaster recovery plan “difficult to implement.”

“Whereas large corporations can focus their resources on implementing a comprehensive and cohesive strategy,” he says, “universities just don’t have that luxury. This means that to create an effective disaster recovery planning strategy, universities should implement a flexible, heterogeneous [plan] that matches the resources available to the individual units. Well-funded faculties should implement the complete [plan]; less well-funded faculties should implement the lower-cost version.”

Hard Choices

Ben Whorton is a network administrator at Birmingham-Southern College in Alabama. He and a cohort staff comprise the Network Services Group, which is in charge of e-mail and server administration, and network security for the school. (Another IT team handles administrative computing; a third team takes care of client services.)

While the school as a whole has a disaster plan for the physical campus, and administrative computing has a disaster recovery plan in place for its pieces of IT, until recently, the network services group didn’t have its own plan—that is, aside from a disk backup spanning a couple of weeks—and that backup resided on a single server. “If something were to happen to that box, then we had no backup,” says Whorton.

Bad, worse, worst. When he was appointed, he relates, an early goal was to put a plan in place that included redundant backups: to replace all single-drive servers with redundant servers equipped with RAID controllers. Still, by 2004 when Hurricane Ivan was rolling in, the plan hadn’t yet been implemented. At that time, knowing the storm would soon hit, Whorton saw only three options. Best case, he remembers, “was leaving our systems up in the hope that we didn’t lose power.” The next was to “leave our systems on and if power went out, pray that there was no dirty power that could fry our systems.” Neither approach was palatable to him. “We stood to lose a half-million dollars’ worth of equipment— plus all the data, which you can’t really put a price on.”

A third option—the worst-case scenario— was to go into the server rooms and proactively power-down all systems. Then, once power was restored, the IT people would bring them back online. Yet, administrators didn’t want to shut down systems, since that was how they expected to be able to contact parents and let them know what was happening in Birmingham with the students. “We had a quandary,” Whorton admits. In the end, the group shut down operations for about 12 hours, until the worst of the storm had passed. Ironically, the college ended up losing power for only about 30 minutes.

Clarifying priorities; weighing resources. That experience led the team to come up with its disaster scenario. “Since the college lost power for only a short amount of time, we realized we needed a backup solution to keep our systems up so parents could visit our Web site to find out what the college was doing to protect their students, provide a means to communicate with their students here at college via e-mail, and keep our phone systems up and running,” says Whorton. “We wanted to make sure there were as many opportunities as possible for parents and students to communicate.”

As a result, the team has made two primary changes: First, it has decentralized many of the servers so that, for example, if one building suffers water damage and floods a computer room, most of that data will be replicated elsewhere on campus. Whorton admits that even that plan is not failsafe. “If we have the unfortunate luck of having two buildings that house the equipment suffer the same flood damage, we’re in trouble. But that’s the chance we take, given our limited resources.”

Second, the team researched and proposed three backup power solutions. The least expensive (about $12,000) required purchasing 26 uninterruptible power supplies (UPSs) for key servers; this solution would buy 30 minutes of uptime in a power outage, and would last two to three years. The second solution, (around $25,000) was a rack-mounted UPS. A third solution (most expensive, at about $35,000) involved a gas-powered generator plus UPSs. Challenges included the need for the space to add a concrete pad to mount the external generator, and in the long-term, increased maintenance capability. As it turned out, the administration went for the mid-priced solution, and Whorton’s group purchased APC equipment that would keep operations running in their current state for five to six hours.

Reality test. As it happened, the school had a chance to try out the new solution when Hurricane Dennis swept through in July 2005. Recalls Whorton, “The storm was supposed to hit on a Sunday and we started preparations on that Thursday. By end of day Friday, we were ready to go. By Saturday morning, Dennis was hitting the coast, and I said, ‘OK, here’s our plan: Everyone has contact numbers. Let’s just stay in contact.’ If we needed to make an adjustment, our security team on campus volunteered to pick us up and take us to different buildings.” In the end, Whorton says, “We didn’t have to. I stayed up until midnight on Sunday, and once the eye [of the hurricane] passed, I knew the worst was over. We had suffered no damage or outages.”

Rather than trying to keep everything up, Whorton’s group focused on keeping communication links up: the phone, Web, and e-mail. Since no one on campus was working through the storm, the school simply shut down its ERP system and storage facilities.

Looking at Unique Concerns

Lori Franz, a management professor in the College of Business at the University of Missouri-Columbia, has explored the topic of disaster recovery planning for a presentation she delivered at Educause2003 (www.educause.edu). According to Franz, one of the DRP challenges unique to a university setting is the fact that “everyone likes to have control over his or her environment.”

Centralize, centralize. At MU-Columbia, she says, “We’ve done a lot of work to try to get all of our decentralized file servers into the facility where there is backup. We try to get everyone to realize that it isn’t safe just to have these departmentally- owned file servers where everything is contained in a single place, and they’re not taking care to back up, and tapes aren’t being stored in a different location.” Changing such a mindset, however, means making participants aware of the value of a centralized computer effort—and “creating trust in the computing organization.”

Public or private? Importantly, her school’s disaster recovery plan must take into account a unique situation, says Franz: “We have the largest university run nuclear reactor in the country on this campus. When we’re thinking about things, we think very broadly about what could happen.” And, understandably, the institution has struggled with how public its disaster recovery plans—or any related campus details—should be. Franz recounts how at one time, the university had placed online maps of every building on campus, complete with drill-down layers. “You could see the plans and schematics for every building. You could access them. Now, we don’t have that,” she says.

Yet, a key element of the plan that has been made public is the existence of an emergency Web site. “We’re sitting here with 24,000 students,” she says. “How do you communicate to the outside if something happens on campus? The Web site is available to everyone—it’s where people will go for the news if something happens.”

More Mundane Concerns

At the other end of the spectrum, opposite problems of biblical proportion, are the daily worries of finding the time to keep servers running optimally and clean of potentially devastating viruses and worms. Michael Kearns, systems administrator at Syracuse University (NY), knew a better plan for disaster recovery was necessary after the campus was hit by the MyDoom virus in 2003, sending IT personnel scurrying to patch and rebuild systems. Although, fortunately, vital data wasn’t lost, Kearns worried about damage to the functions performed by the school’s energy servers which automate climate control and electrical operations campuswide: He knew that if the wrong energy server had gone down during a winter blast, vital research work could have been lost due to extreme temperature changes.

Real-time recovery. The solution he and his team settled on focused on maintaining real-time, disk-based backups using Symantec LiveState Recovery software (www.symantec.com). The advantage of going to disk is that recovery can be quick. Kearns says this has made the IT team more willing to apply security patches; the solution allows a “snapshot image” of the server to be recalled if a patch job d'esn’t go as planned. (Tape is used for archival purposes.) He reports that the school had the chance to test out the system in real time when a domain controller upgrade went awry and the server became inoperable. A quick recovery through LiveState had the system back up and running in less than 30 minutes.

Mining a Pan-Departmental Team

Of course, disaster recovery g'es beyond protecting data on servers. At the Rhode Island School of Design, Judy Tanzi, one of the school’s two telecommunications experts, participates in her university’s DRP committee which was formed two years ago (in the wake of 9/11), and also includes representatives from Facilities Management, Housing, Health Services, Residential Life, Environmental Health and Safety, and Fire and Public Safety. All 12 members of the group have gone through Community Emergency Response Team (CERT) training, which educates individuals about disaster preparedness for hazards, and teaches basic disaster response skills such as fire safety, light search and rescue, team organization, and disaster medical operations.

Drilling down to essentials. As the committee members developed more of a “world view” about disaster preparedness and recovery, they had to sort through what RISD’s plan should encompass.

“That was very difficult,” Tanzi recalls, “because there were so many thoughts that each individual had from his or her side of the house. So, we just put all ideas out there, and kept breaking down the list as to what were the 10 most important things we needed to do.”

Student safety came first; there was no dispute about it. “We started from that point,” Tanzi says, pointing to the questions the committee members faced. “What happens in a disaster? How do we get to the students? What do we do?” After many meetings, she says, a Top 10 list was formed. Then the group had to consider what types of disasters were likely to happen. One challenge for the school is that the campus is spread throughout the city of Providence, which meant that committee members also had to come to agreement on where the main point of contact would be, in the event of a disaster. Tanzi reports that the committee has met monthly and often includes outside representatives (such as those from the police and fire departments), in order to sort through what role the school would play in disaster scenarios.

Informing parents. After student safety, says Tanzi, RISD’s next step is to put together a plan that can be distributed to parents, so they will know what steps would take place if an emergency were to occur on campus, as well as which means to use in order to communicate with the school. From there, the committee will start pulling in other campus participants to act as emergency leaders.

Re-examining telecom opportunities. The RISD taskforce has also recognized that IT and telecom could work together in multiple ways. Now, to enable nonobstructed communications, says Tanzi, 10 campus employees have cell phones with priority status. That means that in a disaster, when everybody hits their cell phones, the calls made by those 10 participants will go out on the same frequency as police and fire calls. Tanzi has also set up certain phones on campus (running “plain old” copper wire) so that in the event of an emergency that disrupts regular phone service, the phones can host the redirected 800 number phone service given out to parents in the school’s disaster plan.

Tanzi isn’t new to considering the vagaries of disasters. In 2000, she pushed her school to implement a 911 system that would ensure that calls made anywhere on campus could be identified by location. The problem with many campus phone systems is that they use private branch exchange, which feeds all calls through the same billing number. When an emergency is reported to 911, there’s no way to identify location from that phone number. Says Tanzi: “[The 911 people] would call back and say, ‘We had a 911 call.’ But the operator would have no clue where the call came from. So I said, ‘We have students. It’s too important to know where that call comes from; sometimes seconds are so important in a life.”

Tanzi chose the Telident 911 enterprise solution from Teltronics (www.teltronics.com), a turnkey solution that places a PC in the school’s switch room, running a database application that identifies a specific phone with its location. When a 911 call occurs on the PBX, the database feeds location information to the emergency dispatch centers of local police and fire departments, as well as to emergency personnel on campus. The system can also perform e-mail notification, and send text to cell phones. Tanzi says she feeds the initial records into the database herself, which is then transmitted to Verizon (www.verizon.com), which populates the data in the Providence 911 system. (Many states have already passed regulations stipulating that all new PBX installations implement a comparable location identification system, but Rhode Island isn’t one of them. The Federal Communications Commission has also considered national regulation in this area off and on for many years, but nothing has been finalized.)

Finally—data. RISD’s committee is also working with other colleges to locate their data off-site. It would be an informal arrangement, says Tanzi, and the plan is only in the “talking stages” at this point. “We’re on one side of the state, and we would probably work with a college on the other side of the river,” she explains. “If the city experienced a disaster, but the other side of the state was still up and running, we’d be able to work from that facility.”

Data Decisions

For some, data is concern number one in any DRP. Keenan Baker, a storage specialist with technology products and services provider CDW-G (www.cdwg.com), considers the focus on data protection the starting point of any disaster recovery plan. At the most basic level, he says, clients decide to look for a good backup scheme (typically, tape) where they can easily back up data on a daily basis. More sophisticated planning calls for failover operations, where an institution can “flip” or “switch over” operations from one campus to another, in the event that the systems in the first one are hindered.

Recovery time/recovery point. Developing a plan at an institution of higher education requires mapping out two objectives, he says: recovery time (RTO) and recovery point (RPO). “The recovery time objective is the time needed to recover from a disaster: how long you can afford to be without your systems,” says Baker. The recovery point objective “describes the age of the data you want the ability to restore, in the event of a disaster. For example, if your RPO is six hours, you want to be able to restore systems back to the state they were in as of no longer than six hours ago. To achieve this, you need to be making backups or other data copies at least every six hours. Any data created or modified inside your recovery point objective will be either lost or must be recreated during a recovery. If your RPO is that no data is lost, synchronous remote copy solutions are your only choice.” If a school, for example, is backing up only to tape, he says, administrators might expect to have their data up and running within a day. If the facility is running replication, however, recovery can happen in minutes.

Plans for the ‘plans’—and practice. Still, never underestimate the ability of a disaster to thwart the best-laid DRP plans. Baker recounts that following a talk he gave at a conference, one audience member confided that his school’s disaster recovery plan was kept online. When disaster struck, wiping out all power sources, nobody could gain access to the plan. Now, physical plans are kept in the homes of all vital personnel.

Another mistake common among higher ed institutions: not rehearsing sufficiently. The plan has to be second nature for the participants, says Baker. In addition, run-throughs and simulations give the IT people a chance to check out how well recovery will work. If the tape media has flaws, for instance, they learn that up front—instead of when it’s too late.

There Are Backups, and There Are Backups

Ellen Rome, a VP for universal hardware solutions provider STORServer (www.storserver.com), says that when the disaster recovery d'esn’t work (preferably during those practice runs) it’s always nice to have “one throat to choke.”

The device her company sells is an integrated solution incorporating server, disks, tape, and software (along with integration services). It’s typically rackmounted, though the company can ship the goods as a single appliance, too. Installation takes two to three days. Once it’s up and running, says Rome, the appliance backs up everything on the network. From there, it performs incremental backups. Sound like your own standard operating routine?

Put the plan on the tape. What stands out is the software. It creates, says Rome, a “daily disaster recovery plan.” The operator can include notes regarding availability of individuals and the shifting order of priority in bringing downed servers back into operation. (That prioritization, she pointed out, requires meetings between the IT people and the decision-makers.) This information gets written to the tape. Should a disaster occur, the off-site tapes supply vital responder instructions: who should do what, in what order, and where specific pieces of equipment are located (in order to gain access to them). That same documentation can also be delivered daily from the system, in other forms such as e-mail going out to all participants and their various e-mail addresses.

STORServer founder Bill Smoldt likes to proclaim, “What I sell is boring restores. Usually a restore is exciting, and that’s not good. A restore should be boring.”

Where’s the tape? The question he frequently faces in helping clients set up their backup operations is what to do with the backup tapes themselves—and how to retrieve them in the event of an outage. He says sometimes the solutions get creative. One client (not in education) ships them a couple of states away. Should the tapes be needed, a plane will be sent to retrieve them. Another client, with a number of remote offices sans broadband access, has chosen a plan in which servers will be rebuilt and delivered by car to the remote sites. In most cases, he says, organizations choose the “disaster recovery zone” that’s appropriate to the types of disasters they expect to experience. That may mean the backup g'es a building away, across town, or across the state.

Have a Plan and Work It

LSU’s Voss points out that many regions face situations far more dire than the loss of a single data center. “In Louisiana, we’ve lost a major US city. While that d'esn’t diminish the impact of the crisis on a given university, it d'es mean it’s competing for resources with a much broader range of needs. We all need to think through this more carefully. I, for one, am going to move smartly, making disaster recovery a key component of my new organizational structure and mission. Because next time, it could be LSU.”

: