From the Edge of Katrina: An Update and Thoughts on the Future of IT Planning for Disaster Recovery

By Brian D. Voss,
Chief Information Officer,
Louisiana State University

LSU handles Katrina

About two months ago, I wrote a piece, Hitting the Ground Walking for C2 about my experiences as a new CIO. Today, in the aftermath of Katrina, I have to say that I no longer feel like a newbie. And I say that, having had the good fortune of not having to confront the horrors and challenges faced today and in the days ahead, by my colleagues at Tulane, University of New Orleans, Xavier, Loyola, and other New Orleans-based institutions. As I write this, I'm mindful that we have not yet heard all their stories, as they have been too busy dealing with their burdens and losses to even sit down and write something.

They are dealing with the prototypical disaster recovery scenario. I am not. My data center was untouched after the storm passed us that Monday afternoon. The center was running on a newly repaired back-up generator (I'm glad we found and fixed a problem there a week before) and many of our systems were powered down to conserve fuel for the first 36 hours after Katrina passed. But everything was fine. We were even running our new HPC resource at the limit, as researchers here were analyzing storm data.

Then a new reality set in, as New Orleans flooded the following morning. Everything I thought I was going to be doing as CIO has changed. I am not recovering from a disaster to my facility. But I have been dealing with the aftermath of the worst disaster in our nation's history. Let me tell you a little bit about how things played out, and continue to play out for us, here on the outskirts of this disaster. Then I'd like to share some early thoughts about the future of disaster recovery planning in higher education.

In the days immediately after Katrina's passing, the LSU A&M campus in Baton Rouge became a primary location for medical evacuations. Our Pete Maravich Assembly Center (PMAC) and its neighboring Field House became a M*A*S*H-like triage center and special needs facility respectively. I'm told that it is the largest such facility ever assembled in time of disaster in the U.S. and it handled more than 5,000 patients in the early days after the disaster. Emergency response units sprang up on campus from the alphabet soup of agencies, both state and national, as did volunteer coordination centers. LSU became the primary operations point for these immediate response needs, and with that the university's role shifted from a higher education focus to one of higher calling. Helicopters buzzing overhead became a common occurrence; we didn't even look up at them anymore. Sirens no longer alarmed us; we heard them constantly. And campus streets weren't teaming with students; streets that weren't disturbingly silent were disturbingly filled with uniformed men, medical teams, and volunteers streaming to the aid of the sick, wounded, and displaced.

LSU handles Katrina

Having lived through the past several days (it seems much, much longer), I am simply amazed by my colleagues on campus, and how they performed. There are many exhausted her'es who the week before were ordinary women and men serving higher education at Louisiana's flagship institution. They responded-they improvised, adapted, and overcame. They didn't have a book or manual to follow. One of them said it best: "We're not following the book here, we're writing it."

IT played a part in this. Every one of these centers and agencies needed immediate access to telephone and data service. Local cellular service was overwhelmed by the influx of 250,000+ new residents, evacuees, and emergency response personnel. In the first 36 hours, we installed more than 100 telephones and about that many computers with network connections, including wireless access points for use with laptops. This number grew steadily, as did the moves/adds/changes as the situation on campus flexed to accommodate needs in an unwritten script of activity.

LSU handles Katrina

We called upon our long-standing vendor partners for help, and they delivered. Many gave us the equipment we needed; others accompanied emergency orders that we placed with additional units for 'indefinite loan' or as outright donations. Companies such as IBM, Microsoft, Avaya, Panasas, and Cisco have gone to great lengths in these early days to provide assistance, and we're grateful for this support from our vendor partners. Microsoft and IBM actually put 'boots on the ground' on campus to work with us, and others in Louisiana to help get the job done. And I've gotten e-mail and phone calls from several others, who we will work with as the recovery efforts continue for likely months to come.

It was a 7x24 operation for us in the week that followed Katrina. Classes, postponed for a full week, finally reconvened the Tuesday after Labor Day; we now have our 'regular' jobs to do as emergency operations wind down on the LSU campus. It looks like things will be getting back to normal here-pretty much, any way-soon. Well, if not 'normal' then the new definition of normal that is going to follow what happens in the aftermath of Katrina throughout this state, and this country. I can't tell you what a relief it was, as classes resumed, seeing the students on campus; returning students and our 3,000+ new students looking for classrooms on their freshly printed schedules. It was, at times and in certain parts of campus, as if nothing had happened. Those of us who lived through the week after Katrina needed to see something familiar and comforting-and seeing all of our students with their books, backpacks, and laptops almost made us cry with joy and relief.

I want to laud my staff. When I wrote that last article for C2, I was still getting to know them. In adverse situations, you really get to know people. And my staff are the best anywhere (okay, I'm not very objective at the moment-but that d'esn't change the facts). They pulled together, worked demanding hours, saw things no one should have to see, and worked without a net. They were flexible. They answered the call to write quick-and-dirty information systems that were hot items at 8 a.m., and then no longer needed by 5 p.m. They had the foresight in their normal operations to have a 'cushion' of equipment and supplies, which helped keep us afloat and responsive during the first critical hours. Some really never went home for days during that long week. I can't say enough about them. All of them. I am so very proud.

New Thinking on Disaster Recovery
In the midst of this though, I'm trying to take stock of what is happening and develop some thoughts about the future. Of course, there's the obvious: The need for disaster recovery planning at universities needs to get a big push onto a front burner. Three weeks ago, my line was "Disaster Recovery certainly is important, but we need to have things in place of high value which to recover!" Which was my way of saying I'd spend money on DR once I'd spent money on building infrastructure and services in support of teaching and learning, research, student service, and information systems at the institution. My tune is changing. I was going to have to hunt for money for all those things (as I've said recently, "No one handed me the vault keys when I showed up here in Baton Rouge."); now I'm going to need to just hunt harder and find money for DR as well.

But look at the flip side of this issue. Think about: "What if you're the last ones standing?" Traditional DR thinking has to do with what to do if your data center is gone. How do you recover? How do you restore services for your institution? And that's very valid, as my colleagues at Tulane, UNO, and other New Orleans universities are experiencing most directly right now. But what if your campus suddenly becomes the refuge for not only your neighboring institutions, but also local, state, and federal agencies? What if you are the site of last resort? How will you deal with a sudden influx of users placing unimaginable demands upon your environment and resources? Do you have a storeroom of network gear, laptop and desktop computers, cabling and supplies, and a flexible and capable staff that you can deploy ad hoc, in a highly volatile environment? And what if, in the midst of all this, you also have to continue your 'normal' operations as your campus resumes activities-and do so with a swelling population of students?

I know that NASA has addressed two of its disasters with the phrase: We had a failure of imagination. Meaning the disaster was one we did not imagine could happen, and thus we didn't plan for it. Katrina was that kind of disaster. But now we no longer need to imagine what it's like when a major American city is destroyed-it happened. Most of us watched it on CNN, and those of us in and near the ravaged areas are living it first hand. We now need to find answers to the questions I pose above, and others we have not yet had time to ask or imagine.

I don't have the answers to these questions. I'm still living with this; we're still writing chapters in a book and there is no ending in sight. And while we've turned a corner here in Louisiana, what's around that corner is terra incognito. There is no book-maybe there will be after this is over. But we, in higher education, need to get about the task of writing one as it relates to our roles in such disasters.

Those who do not learn from history are doomed to repeat it. And one thing we need to learn is that we must start imagining that which we have not before imagined. Before it becomes reality.

Share your ideas and opinions on the CT Disaster Planning and Recovery forum.


(Photos courtesy Louisiana State University)

comments powered by Disqus

Campus Technology News

Sign up for our newsletter.

I agree to this site's Privacy Policy.