Disaster Recovery from the Inside

• By Dian Schaffhauser
• 02/27/09

No matter how solid your disaster recovery plan, the consequences of true disaster languish for years. For the individuals and institutions involved, a catastrophe changes assumptions, priorities, behaviors, and expectations. That was the case for New Orleans-based Delgado Community College, which was put out of business temporarily during its first week of classes in August 2005 when Hurricane Katrina flattened New Orleans and levee breaches flooded the city. In the process of becoming operational again, Delgado improved its IT processes and infrastructure in meaningful ways.

Pre-hurricane, the school had 17,400 students, 1,300 faculty and staff, and three main campuses in New Orleans, as well as several satellite learning centers. Immediately afterward, when the city was under feet of water, downed power lines blocked roads, and the telecommunications infrastructure, which was mostly underground, was either destroyed or dramatically damaged, Delgado cancelled that semester's classes. The main campus at City Park, which sat under as much as eight feet of water, suffered major storm damage and the school's Slidell Learning Center on the north shore of Lake Pontchartrain was declared a total loss,

Assessing the Damage
As CIO Thomas Lovince explained: Immediately after the flooding, senior management in the office of information technology (OIT) communicated via their Nextel devices and created "phone trees" to get in touch with employees. IT reported for duty in Baton Rouge on the Friday after the storm. There, on the Board of Regents Web site, the team examined aerial photos showing one to six feet of water on the Delgado campus.

The computer room of the data center, which was on the second floor, was intact. The challenge for IT was that the first floor of that same building had three feet of water. Likewise, the campus' aged phone system, which it rented from BellSouth, wouldn't work as long as the provider's feeder cables were non-functioning.

About a week later, once the water had receded to a couple of inches, a skeleton IT crew drove into the City Park campus under state police escort to assess the state of the computer center and retrieve tapes and equipment that would allow the college to restore critical services such as Blackboard, e-mail, and administrative systems. Among the rescued equipment were 11 200-pound servers that were dismantled and carried down two dark flights of stairs illuminated by a string of lights hooked up to a generator. "The building had no air conditioning. We were working in 100 degree temperatures," Lovince recalled.

The equipment was loaded into a van and hauled to Baton Rouge, LA, about 70 miles away, where the college had a hastily signed contract with a newly opened co-location facility. "We ended up being probably their largest customer right after the storm," he says. "We used a thousand square feet of space to co-locate equipment. We have that now." Once the IT environment had been set up in Baton Rouge, cables needed to be laid, systems booted up, and diagnostics run to check all systems.

Like many other institutions of higher education, the college also had a prior agreement with a partner, a hospital in Shreveport, LA, in the northern part of the state, that would allow it to use the hospital's equipment to restore critical functions in the event of an emergency. So in addition to moving hardware, the IT team took tapes removed from that equipment to the hospital in order to restore data. But that partnership could go only so far, said Lovince. "We were very thankful for what they did. But they don't sit there--like we don't sit here--and plan for someone else to come and use the equipment. They were getting requests from other agencies and entities to use their facilities and capabilities as well."

Once the equipment was set up and networking rerouted to point to the new hotsite, said Lovince, "there was really nothing we couldn't do, with the exception of things that happened in the city." The Web site for the school was restored quickly, in less than a week after the move, says Lovince. Because the school was using Blackboard, it was able to open a handful of classes online, but it wasn't enough to sustain faculty or students.

Impact on People
At the same time that the IT organization had to respond to the demands of re-establishing operations for the college, individual staff members were trying to grapple with the impact of the disaster on their personal lives. "Some families were separated," said Lovince. "They had all kinds of issues--travel, lodging, logistics. Some of the staff weren't able to be part of the restoration effort; some staff who relocated couldn't come back at all."

Then, about a month or two after the hurricane struck, layoffs began. "Because we had no fall session, our revenue was significantly down," he explained. "We had to have a significant staff reduction across the board. Even while we were doing the restoration, we knew that we weren't able to bring some of those individuals back as we were going forward." Before the storm, the IT organization had about 40 people; afterward, it was 26.

Return to Normalcy
Almost immediately after the storm moved through, students began the effort of getting enrolled in other schools wherever they had landed. That generated numerous calls for transcripts, which the college couldn't produce, since the information was either still in transit or the physical files were under water.

A decision to go paperless in 2001 and 2002 in its administration work proved to be a boon in the post-Katrina era. The college digitized paper-based workflow in its admissions, registrar, health services, and financial aid departments. Budget constraints limited the effort to records collected in 2002 and going forward.

In 2003, it added Sungard Higher Education's Banner ePrint to the mix in order to give users a means to view documents through a browser. ePrint allows Delgado to pull reports from its mainframe, export them to an off-site ePrint server, and give users the ability to review reports via PDF over the Internet. As Lovince described the process, "They'd automate anything that came into the office. If you went to a screen for a student, you could pull up every document on the imaging platform, be it an application, health record, recommendation, and view that with the student information at the same time."

Unfortunately, that didn't encompass all critical student records. But it did mean that with the restoration, all files recorded with the imaging platform could be viewed via the Web. That allowed the college staff, which was working from temporary quarters all over the region, to do standard reporting and to handle at least some requests. It didn't take care of anything that required special forms, such as checks, purchase orders, or labels, but it did represent another step toward regaining normalcy.

And the veneer of normalcy was important. "We determined as a college, as a group, and as individuals to come back to city, and to bring our college back by whatever means necessary," said Lovince. That meant sharing office space, working from home, using laptops with wireless cards, so people could work wherever they were. "There was a lot of adjustment and out of box thinking that had to happen as a result of that."

One aspect that required flexibility was work schedules. "Our computer operators were located in Mississippi, New Orleans areas, elsewhere in regional area," said Lovince. "They had to work at the hotsite. We had no equipment in the New Orleans facility. With them we had to get creative in terms of their scheduling. We had one individual who worked 12 hours--three days straight , then four days off. And vice versa. A person who may have worked during the day before, now because of personal circumstances may have had to work at night."

Also, staff had to shift responsibilities. When the person who handled imaging, for example, realized that she couldn't return from Atlanta, where she'd relocated to, a person from the server group took a "crash course" from vendors on how to operate the imaging software.

That points to another aspect of dealing with changing conditions and addressing challenges on the fly: Lovince's team had to make quick decisions regarding the continued use or purchase of services and products. Throughout, he said, they maintained two core principles: "One, we wanted to deal with vendors with whom we had very strong relationships, who we thought could deliver based on circumstances. Two, we wanted to make sure whatever we did would fit into some long range plan or capability that we had. We still went through the normal evaluation of what we thought were best of breed products, best of breed vendors. We also used whatever resources and counterparts we had within the industry and relied upon them as well."

Moving Forward
Finally, by December 2005, power was back on at the City Park campus, and, by January 2006, the college had re-opened. But since phone service still didn't operate there, the IT team ordered and distributed 200 cell phones to strategic offices and personnel. "When you dialed a number from our Web site, you were dialing a cell phone for some six months into the next year," said Lovince.

Some buildings on campus would take many more months to be habitable, in order to remove mold. The on-campus data center finally reopened in May 2006.

The IT organization took the opportunity of disaster recovery to improve many aspects of its operation. An obvious need was to improve the processes and systems it used to recover from disaster. To this end, it is continuing to maintain the hotsite off campus in Baton Rouge and has begun taking its tape storage offsite and outside of New Orleans.

Plus, it has procured faster tape technology for backup and restoration and begun replicating server data. It has also reduced the number of tapes required for restoration from 140 to 14.In the campus data center, an aging UPS has been replaced to extend battery backup from 30 minutes to a couple of hours, a generator is now in place, and air conditioners have been raised off the floor. A new self-operated PBX telephone system now runs over fiber, a media that proved its worth in the city during the flooding by staying operational. Also, it's maintaining a duplicate phone system in Baton Rouge for redundancy.

Delgado has also replaced its legacy network, integrating wireless into the mix. And the college has begun to image all remaining student records--some 2.6 million images--on an upgraded imaging infrastructure, along with accounts payable and HR paperwork.

Finally, the college realized that its critical paper-based student records were vulnerable to disaster. The institution began a large back scanning process to convert student records and documents to digital formats. That effort uses Perceptive Software's ImageNow. An ImageNow server is maintained at Delgado's off-site disaster recovery center where staff can relocate in the event of another disaster.

Lessons Learned
Four years after the floodwaters rose to shutter his college and the student body has grown to 13,000--still down from its 2005 count--Lovince has had a chance to derive order from the chaos and draw lessons from the experience. First, he said, "You should never take anything for granted, particularly when you're talking about planning for disaster recovery. You have to assume that the worst case scenario can happen." In the case of Katrina, he said, experts had been predicting a hurricane for so many years, and none had ever actually hit the city, "We were lulled into false sense of security. We'd dodged so many storms for so many years."

Second, "We've learned that we must be a much more flexible and agile institution. We're not looking to offer classes the traditional way anymore." The progress of that effort is still on-going and primarily consist of using third party affiliates to offer training and support. Eventually, said Lovince, the school itself will provide more totally online offerings or hybrid classes, using the Blackboard course management system as the main facility for faculty-student communication.

Third, he said, "When you're in a situation like that, you learn that you have to fight for your existence. We had to fight through bureaucracy, fight through personal circumstances, we had to fight as an institution to say that the way we did things before, we can't do it that way in certain circumstances. Now is the time to look at what makes better sense.

As Lovince said, "I would never under-estimate a person's--and in our case, a college's--will and determination to exist."