From the Edge of Katrina: An Update and Thoughts on the Future of IT Planning for Disaster Recovery
By Brian D. Voss,
Chief Information Officer,
Louisiana State University
About two months ago, I wrote a piece, Hitting
the Ground Walking for C2 about my experiences as a new CIO. Today, in the
aftermath of Katrina, I have to say that I no longer feel like a newbie. And
I say that, having had the good fortune of not having to confront the horrors
and challenges faced today and in the days ahead, by my colleagues at Tulane,
University of New Orleans, Xavier, Loyola, and other New Orleans-based institutions.
As I write this, I'm mindful that we have not yet heard all their stories, as
they have been too busy dealing with their burdens and losses to even sit down
and write something.
They are dealing with the prototypical disaster recovery scenario. I am not.
My data center was untouched after the storm passed us that Monday afternoon.
The center was running on a newly repaired back-up generator (I'm glad we found
and fixed a problem there a week before) and many of our systems were powered
down to conserve fuel for the first 36 hours after Katrina passed. But everything
was fine. We were even running our new HPC resource at the limit, as researchers
here were analyzing storm data.
Then a new reality set in, as New Orleans flooded the following morning. Everything
I thought I was going to be doing as CIO has changed. I am not recovering from
a disaster to my facility. But I have been dealing with the aftermath of the
worst disaster in our nation's history. Let me tell you a little bit about how
things played out, and continue to play out for us, here on the outskirts of
this disaster. Then I'd like to share some early thoughts about the future of
disaster recovery planning in higher education.
In the days immediately after Katrina's passing, the LSU A&M campus in
Baton Rouge became a primary location for medical evacuations. Our Pete Maravich
Assembly Center (PMAC) and its neighboring Field House became a M*A*S*H-like
triage center and special needs facility respectively. I'm told that it is the
largest such facility ever assembled in time of disaster in the U.S. and it
handled more than 5,000 patients in the early days after the disaster. Emergency
response units sprang up on campus from the alphabet soup of agencies, both
state and national, as did volunteer coordination centers. LSU became the primary
operations point for these immediate response needs, and with that the university's
role shifted from a higher education focus to one of higher calling. Helicopters
buzzing overhead became a common occurrence; we didn't even look up at them
anymore. Sirens no longer alarmed us; we heard them constantly. And campus streets
weren't teaming with students; streets that weren't disturbingly silent were
disturbingly filled with uniformed men, medical teams, and volunteers streaming
to the aid of the sick, wounded, and displaced.
Having lived through the past several days (it seems much, much longer), I
am simply amazed by my colleagues on campus, and how they performed. There are
many exhausted her'es who the week before were ordinary women and men serving
higher education at Louisiana's flagship institution. They responded-they improvised,
adapted, and overcame. They didn't have a book or manual to follow. One of them
said it best: "We're not following the book here, we're writing it."
IT played a part in this. Every one of these centers and agencies needed immediate
access to telephone and data service. Local cellular service was overwhelmed
by the influx of 250,000+ new residents, evacuees, and emergency response personnel.
In the first 36 hours, we installed more than 100 telephones and about that
many computers with network connections, including wireless access points for
use with laptops. This number grew steadily, as did the moves/adds/changes as
the situation on campus flexed to accommodate needs in an unwritten script of
activity.
We called upon our long-standing vendor partners for help, and they delivered.
Many gave us the equipment we needed; others accompanied emergency orders that
we placed with additional units for 'indefinite loan' or as outright donations.
Companies such as IBM, Microsoft, Avaya, Panasas, and Cisco have gone to great
lengths in these early days to provide assistance, and we're grateful for this
support from our vendor partners. Microsoft and IBM actually put 'boots on the
ground' on campus to work with us, and others in Louisiana to help get the job
done. And I've gotten e-mail and phone calls from several others, who we will
work with as the recovery efforts continue for likely months to come.
It was a 7x24 operation for us in the week that followed Katrina. Classes,
postponed for a full week, finally reconvened the Tuesday after Labor Day; we
now have our 'regular' jobs to do as emergency operations wind down on the LSU
campus. It looks like things will be getting back to normal here-pretty much,
any way-soon. Well, if not 'normal' then the new definition of normal that is
going to follow what happens in the aftermath of Katrina throughout this state,
and this country. I can't tell you what a relief it was, as classes resumed,
seeing the students on campus; returning students and our 3,000+ new students
looking for classrooms on their freshly printed schedules. It was, at times
and in certain parts of campus, as if nothing had happened. Those of us who
lived through the week after Katrina needed to see something familiar and comforting-and
seeing all of our students with their books, backpacks, and laptops almost made
us cry with joy and relief.
I want to laud my staff. When I wrote that last article for C2, I was still
getting to know them.
In adverse situations, you really get to know people.
And my staff are the best anywhere (okay, I'm not very objective at the moment-but
that d'esn't change the facts). They pulled together, worked demanding hours,
saw things no one should have to see, and worked without a net. They were flexible.
They answered the call to write quick-and-dirty information systems that were
hot items at 8 a.m., and then no longer needed by 5 p.m. They had the foresight
in their normal operations to have a 'cushion' of equipment and supplies, which
helped keep us afloat and responsive during the first critical hours. Some really
never went home for days during that long week. I can't say enough about them.
All of them. I am so very proud.
New Thinking on Disaster Recovery
In the midst of this though, I'm trying to take stock of what is happening
and develop some thoughts about the future. Of course, there's the obvious:
The need for disaster recovery planning at universities needs to get a big push
onto a front burner. Three weeks ago, my line was "Disaster Recovery certainly
is important, but we need to have things in place of high value which to recover!"
Which was my way of saying I'd spend money on DR once I'd spent money on building
infrastructure and services in support of teaching and learning, research, student
service, and information systems at the institution. My tune is changing. I
was going to have to hunt for money for all those things (as I've said recently,
"No one handed me the vault keys when I showed up here in Baton Rouge.");
now I'm going to need to just hunt harder and find money for DR as well.
But look at the flip side of this issue. Think about: "What if you're
the last ones standing?" Traditional DR thinking has to do with what to
do if your data center is gone. How do you recover? How do you restore services
for your institution? And that's very valid, as my colleagues at Tulane, UNO,
and other New Orleans universities are experiencing most directly right now.
But what if your campus suddenly becomes the refuge for not only your neighboring
institutions, but also local, state, and federal agencies? What if you are the
site of last resort? How will you deal with a sudden influx of users placing
unimaginable demands upon your environment and resources? Do you have a storeroom
of network gear, laptop and desktop computers, cabling and supplies, and a flexible
and capable staff that you can deploy ad hoc, in a highly volatile environment?
And what if, in the midst of all this, you also have to continue your 'normal'
operations as your campus resumes activities-and do so with a swelling population
of students?
I know that NASA has addressed two of its disasters with the phrase: We had
a failure of imagination. Meaning the disaster was one we did not imagine could
happen, and thus we didn't plan for it. Katrina was that kind of disaster. But
now we no longer need to imagine what it's like when a major American city is
destroyed-it happened. Most of us watched it on CNN, and those of us in and
near the ravaged areas are living it first hand. We now need to find answers
to the questions I pose above, and others we have not yet had time to ask or
imagine.
I don't have the answers to these questions. I'm still living with this; we're
still writing chapters in a book and there is no ending in sight. And while
we've turned a corner here in Louisiana, what's around that corner is terra
incognito. There is no book-maybe there will be after this is over. But we,
in higher education, need to get about the task of writing one as it relates
to our roles in such disasters.
Those who do not learn from history are doomed to repeat it. And one thing
we need to learn is that we must start imagining that which we have not before
imagined. Before it becomes reality.
Share your ideas and opinions on the CT Disaster Planning and Recovery forum.
(Photos courtesy Louisiana State University)
