How Ready Is Ready?

• By Mikael Blaisdell
• 09/26/06

Katrina was a wake-up call for more than the Gulf Coast schools: Our inquiring columnist looks at three disaster-aware institutions, and checks out their websites to see how far they’ve come.

HARD TO BELIEVE it is little more than a year ago that the US higher education community faced a rude awakening in the form of Hurricane Katrina. As levees ruptured, winds raged, and flood levels rose, college and university CIOs and administrators discovered how quickly a campus can lose all access to telephone and cell phone communication, computers, and data. In such a disaster, students and faculty may be scattered locally or regionally with no way of contacting one another, communicating their status, or knowing if the campus is safe or imperiled. E-mail and websites may be down, and phones may be inoperable. Communication among administration, faculty, students, and their families can be lost in a heartbeat, just when the need for a source of reliable information is greatest. And administrative computing resources can come to an abrupt halt, meaning no expediting of services, no payrolls, bills paid, or accounts received. Katrina proved it could happen. Now, a year later, how are schools preparing for the possibility of other catastrophic events?

The first step in disaster planning is, of course, to acknowledge the possibility of a disaster. New Orleans will always be vulnerable to hurricanes, so Xavier University (LA) must prepare to be hit again. Yet, on the other side of the country in California, where earthquakes are a likely occurrence (and recent press coverage points to acknowledgement that California may be unprepared for an anticipated, sizable event), what have the University of Southern California and the University of California-Berkeley done to prepare for potential disaster?

Strategic Business Continuity and Disaster Recovery Planning

When IT is under water, how well can anything else operate? In the thr'es of Katrina, Xavier University administrators and technologists were faced with the painful realization that the university’s main computing facility was vulnerable, and that it could be put out of action for an extended period of time. While other schools volunteered space on their servers for email and other daily functions, taking advantage of such assistance for anything more than that would mean that Xavier would temporarily hand over control of sensitive (donor list, etc.) and financial data to a host institution— functions that simply could not be hosted on loaner systems. Xavier now will have a mirrored site in another region of the country that can immediately be used to provide access to all of the necessary systems. The provisioning of that alternate site brought up another critical point for Xavier CIO Cathy Lewis.

“A lot of people will start off thinking that they have to duplicate everything,” she notes, “but that’s not what they should be doing. You have to ask yourself: What’s critical? What’s really necessary in order to keep on operating?” That’s the kind of analysis that is necessary to proceed with business continuity planning in order to a) keep focus clear, b) keep emergency systems nimble and, importantly, c) actually be able to secure the funds for the project without dangerous delays and without relegating other IT needs (such as learning technologies) to the back burner.

Where should you begin searching for assistance with offsite redundant systems capability? On the data side, companies such as SunGard Higher Education and IBM can offer solutions for offsite mirrored servers and functionality. Sprint has an emergency response team that can bring in mobile broadcast towers to restore mobile phone networks, and Verizon offers Business Resilience consulting and services to help mitigate both critical voice and data outages across the campus enterprise. But look to academic partnering, as well: Other schools that have not been hit by a disaster may step forward to offer both short and longer-term assistance. Yet why leave this to chance? Plan ahead! The availability of such potential resources d'es not mean that no planning is necessary in order to use them effectively.

Of course, in the event of an emergency, the entire campus community—and anyone nationwide or worldwide who is interested in the fate of your institution and any of its community members—will be looking to your website for guidance. Let’s take a look at the websites of the three disaster-concerned institutions we cover here—Xavier, USC, and UC-Berkeley—and see how they fare. What we find just may be instructive for campus administrators everywhere.

Start with the Website

When Katrina took out Xavier’s website, it took with it one of the best means of disseminating vital information to the entire campus community. Clearly, having a remote location to automatically take over the website (if at any time the main site g'es down) is a substantial asset in the event of a major emergency. All three schools—Xavier, USC, and UC-Berkeley—currently have just such a resource.

A visit to the main page of USC’s website reveals an Emergency Info tab at the top right of the screen. That link takes site visitors to a separate site, which informs visitors that it automatically comes up as the substitute main page for www.usc.edu if for any reason the primary site becomes unavailable. The emergency site provides contact phone numbers for parents trying to reach students, and in a real emergency, could be used for general announcements and updates.

Xavier’s emergency site, established in June of 2005, is hosted by a California-based commercial service with four separate data centers. A link to the emergency site, Emergency Preparedness, is prominently displayed on the bottom of Xavier’s main page, although the location on the page means that some site visitors may have to scroll down to see it. The university’s emergency website can also be located via a Google search.

UC-Berkeley’s primary site d'es not have a main page link to its Office of Emergency Preparedness, but the page can be found by doing a web search, leading you to 'ep.berkeley.edu. The school also has an off-site resource, emergency.berkeley.edu (through EarthLink), and is in the process of completing a reciprocal arrangement with the University of California-Los Angeles for mirroring some critical functions at UCLA’s Southern California IT center.

According to Tom Klatt, manager of UC-Berkeley’s Office of Emergency Preparedness, “The equipment is in the racks; we’re just working out the firewall and certificate issues before going live.”

Publishing Plans Online

All three emergency preparedness sites provide downloadable materials for students, faculty, and parents to use in making their individual contingency plans. Since Katrina, Xavier requires its students to make and file such plans within three weeks of arriving on campus. (During post-Katrina evaluations, it was realized that in order to properly plan for evacuation of those students who did not have their own transportation, the university would have to know in advance which, and how many, students would be affected.)

Preparation and suggested procedures for dealing with specific types of disaster events (e.g., hurricanes, earthquakes, etc.) is one thing, but what about a university’s policies and processes for crisis events in general? Both Xavier and UC-Berkeley publish their general emergency procedures on their websites, detailing such vital policies as who is to be in charge and what events will trigger certain procedures. A detailed set of disaster plans and procedures available for public reference on the site is impressive, but how do students, parents, faculty—even staff—know that those plans will work, if the plans haven’t been tested?

UC-Berkeley g'es a step further than USC and Xavier, for it schedules regular testing events and publishes the results of the exercises on its emergency preparedness page. [The latest such test was conducted in April 2006, and the results, After Action (AA)/ Corrective Action (CA) Report, available for online review.] Testing emergency plans can reveal vulnerabilities (e.g., commercial radios without sufficient batteries) and unwarranted assumptions (e.g., that all staff members know how communications are to flow) hidden inside the procedures; such testing can allow administrators and technologists to see precisely where additional options need to be developed.

The tests can also serve another important function, for the willingness of the senior administration team to go along with the testing—especially if there are substantial costs involved—will tell you a lot about the importance the institution assigns to the initiative. If your school d'esn’t have its own testing procedures in place (or even a well-developed set of disaster response plans), your first step might be to visit the websites of institutions that must deal with the potential for disaster (say, the three school websites examined here) and “borrow” the emergency preparedness materials found there, as a starting point or as templates. Truth is, in a world where disaster now seems to lurk around every corner yet budgets are stretched to the max, the sooner we can get up to speed without reinventing the wheel, the better we’ll sleep at night.