Disaster Recovery Planning

It's All About Power

• By Dian Schaffhauser
• 05/01/08

In the strategic scheme of DRP, straightforward issues of power redundancy and backup often are afterthoughts. But here’s your chance to assess and plan now, before that next power interruption costs your campus dearly.

DELIVERY TRUCKS TAKE DOWN POWER POLES. Squirrels sneak into transformers. Summer heat spawns brownouts and blackouts. Power grids seize. Blizzards, earthquakes, tornadoes, and hurricanes proliferate yearly. And then, of course, there’s always construction, scheduled upgrades, and the unscheduled human error "events" that lurk everywhere. No matter the cause, when the power goes out, your data and operations are at risk.

In fact, data loss from life’s little power calamities may be the most common form of IT disaster any campus can face. According to a 2007 industry association survey, 82 percent of higher education institutions reporting a disruptive occurrence in the five years prior to the study revealed the most common event was simple electrical failure. Even a momentary outage can knock out phone service, emergency notification devices, websites, e-mail, specialty equipment, security systems, research projects, legacy hardware, and critical networks.

UPS Plus Communication Hub Generator

Theresa Rowe, CIO at 18,000-student Oakland University near Detroit, was enjoying Christmas Eve 2006 at home when she got the call. As she describes in an industry association post, that evening a systems administrator had noticed that e-mail was no longer available. When he went to the data center to figure out what the problem was, he found the temperature over 115 degrees in the room; both air conditioning units had failed. Rowe and the facilities manager arrived to discover that some of the servers had gone into automated shutdown due to the heat. The systems administrator proceeded to shut down everything except the most essential systems. Then the small team threw open the doors to the Michigan winter, turned on the fans, and called in the contracted HVAC service provider. When the technician finally arrived, he found that the main electrical feed to the roof chillers wasn’t working. So the next call went to the university electricians, who found the circuit breaker had tripped. The breaker was reset, cooling was restored, and everybody went home around 2 am.

Still, Rowe recounts, she and the facilities manager "didn’t have a good feeling." When they returned to the data center on Christmas Day, they found that the situation had repeated itself. Eventually, the electricians figured out that the power feed to the distribution panel from the 480-volt substation had developed a fault to ground and caused the breakers to trip. A temporary fix was created by rewiring the air conditioning units to another panel.

The good news? Through it all, says Rowe, "We had our drill down." Based on the school’s experience in the great Northeast blackout of 2003, "We knew what an electrical outage would do to our space and how to manage through it." There were problems however, admits Rowe: "Until a few years ago, whenever power went out, our phone systems went out 15 minutes later." Finally, during the 2003 outage, she says, "Our campus community agreed that kind of standard was no longer tolerable."

Rowe was given budget dollars to restructure the communications hub, including phones and the internet, to include an Eaton Powerware UPS and power generator (Generac Power Systems); both models are now out of production. Now, she’s hoping the data center itself will get a generator, but the school is trying to blend that into a broader campuswide generator plan. "Funding is always an issue, particularly in Michigan," sighs Rowe.

Until then, when the power goes out, she says, "We have about 15 minutes before we have to make shutdown decisions. Then we start with non-critical services and turn those off to give people access to critical systems as long as possible."

Monitoring for Single Points of Failure

Darrin Zeller is operations manager for The University of Tennessee-Knoxville (enrollment: 26,400), where he manages central IT operations in the areas of environment and monitoring. Although most of the outages on his campus are brief (usually caused by electrical storms), more lengthy outages can occur when animals get into the substations, causing shorts, or when crews need to bring power down on campus to do electrical work during construction projects.

Of course, outages can occur for all sorts of reasons. Recently, a breaker that feeds power to the Eaton Powerware Plus UPS from an emergency panel in an electrical room tripped during a switchover from the power generator back to the utility company. Zeller had no way of monitoring that specific power feed, because it was part of what the facilities department monitors. So even though service should have been restored to normal, the IT operations were running off of batteries. Within 15 minutes, the batteries went dead and everything crashed. The culprit, discovered a month or two after the outage: aged wiring that had gone bad.

From that incident and others like it, Zeller has learned an important lesson: "Any place there’s the potential for a single point of failure, eventually there will probably be a failure."

UT-Knoxville’s data center, which contains hundreds of servers, is housed in a seven-story building. Only Zeller’s floor is hooked up to the generator, a Caterpillar diesel generator set comprising a 3412 diesel engine and an SR4B generator, which provides 750kVA/600kW of coverage. The only service the generator maintains on the other floors is emergency lighting, a small power load compared to the requirements of the data center. If the power stays off for longer than about 10 seconds, says Zeller, the generator starts up and makes the switch. Staff inside the data center notice the switchover only because the lights (which aren’t on a UPS) will go off for a few seconds, then come back on. That’s it. "Everything stays up and running," he says.

Zeller estimates that the generator can sustain an outage of between 24 to 36 hours at full load, without refueling. He’s never had the chance to time it because the campus has never had an outage of that magnitude. But the generator is sized for the capacity to cover services over a holiday, he says- enough time to allow a truck to get to the campus to refuel it.

That doesn’t mean the data center power management setup is foolproof. "There are always those single points of failure," says Zeller, adding practically that "it’s a space-and-money issue to go fully redundant on a lot of these things [UPS, generator, electrical box]." But, he says, "We’re doing much better than we did the prior 10 years when we didn’t have a single UPS."

Still, when the generator finally came in (after several years of delay, due to campus budgetary limits), Zeller made a decision to cut back on the number of batteries in the UPS, from three cabinets’ worth to two. (The data center could run on only half of that, but he prefers to have a level of redundancy.) That saved the school roughly $6,000. (The generator itself cost about $100,000; related hardware and installation costs pushed the entire project to about $500,000.)

Zeller considers cooling and humidity control in the data center a vital aspect of keeping services running. The optimal temperature is 70 degrees, he says; the optimal humidity level 45 percent, plus or minus 5 percent. "If the humidity gets too low, we have a static electricity problem, and if it gets too high, we have a condensation problem," he explains. The data center uses Liebert Deluxe System/3 chilled water environmental control systems, which can humidify and dehumidify, based on sensors in the units.

Right now, the UPS in the data center runs at about 80 percent capacity. Zeller maintains a graph of growth and power usage in the center, and based on the campus’s current rate of growth, he says, "We don’t have too much longer before we’re going to run out of power." By then he hopes to have moved some of the data center equipment that’s part of a research computer cluster, as well as test and development systems, into a building that the university recently purchased, which is already outfitted with a generator. To that end, he’s worked with the facilities people to design what the site will need in terms of wiring and a new UPS.

Other campus locations Zeller is monitoring have varying degrees of backup. The site where the campus telephone switch is located, for instance, has UPS and generator coverage. But the site where the tape robot system resides lacks a generator. "We’d like to have one, but it’s not as critical," says the operations manager. If the power goes out, "we may lose a day on our backup. That’s a level that people are willing to accept, based upon the cost to install."

New Backup Concern: Power Over Ethernet

North Carolina’s Central Piedmont Community College, with six campuses and enrollment of about 35,000, is outfitted with 88 redundant UPS units and 16 diesel (including some biodiesel) generators, all in place to keep services running during the occasional outage.

CIO Malik Rahman believes the good working relationship his IT organization has with the campus facilities organization is key to its power management success. "For any new [building] design, IT people work with end users to make sure technology needs are met and, at the same time, we work with Facilities [to define] the standards that need to be followed for wiring closets for any technologies we have on campus."

The wiring closets are a particular concern for Piedmont technologists, because the school is deep into a data convergence project with Nortel to upgrade the network within and among its six campus sites. That project includes the upgrade of the infrastructure from 1 to 10 gigabit Ethernet, as well as the implementation of voice over IP (VoIP). Currently, about a third of handsets are VoIP, with the remainder migrating as the school installs new switches with power over Ethernet (PoE).

"The key selling point of traditional telephony was the dial tone available every time the handset was picked up," says Rahman. "When we move to VoIP, we have to provide the same level of availability. Otherwise, it’s like a step backward." That requirement necessitated a level of reliability in every closet housing switches for PoE, so the campus brought in an outside engineering firm to evaluate all of the IT power requirements and study the campus emergency power generation capability. Rahman and his team quickly concluded that there were certain areas where emergency generators did need to be bumped up in size, although 90 percent of all "new" construction (completed in the previous decade) had sufficient emergency power available.

Happily, the growing pervasiveness of Ethernet-powered communications has simplified Rahman’s job of justifying funding requests for emergency power. "The network carries network data and VoIP but also carries our surveillance system video and building control systems," he explains. "Because of that, [the power backup upgrades] become a lifesafety issue. For that reason, the college has invested in emergency power availability in these closets."

Although the campus hasn’t experienced an unplanned outage of its data equipment in a "long, long time," Rahman’s team still has a shutdown process in place as part of its disaster recovery plan, which gets tested annually. The development of that process, he says, has required IT to prepare a catalog of services for tech staff, in the rudimentary stages at this time. That catalog records details of service: "Which server is this running on? Which server is the backup server? How critical is it? What is the population that will be impacted if the server is down? Which other services are either dependent on this service, or is this service dependent on?"

Still, even redundant systems can expose weak areas. In the case of Piedmont, says Rahman, its points of weakness are mainly human. A couple of years ago, he recalls, an electrician from the facilities operation walked into one of the data centers and pushed a red button, "which brought down all of our servers simultaneously." Since then, he says, IT has imposed stringent controls on entry into the data centers.

As the Data Center Goes, So Goes the School

Dick Bednar, senior IT director at California State University, Fullerton, says that even though his campus has a generator that will keep the data center and PBX in operation ("as long as we keep putting diesel in"), nobody will be able to work in any of the other buildings, since they won’t have power.

But the fact that all the services delivered from that data center will remain up is the reason why backup emergency power is now so vital to a college or university. Geographic or physical location of workers accessing that data has become of lesser importance. After all, even if a school’s offices can’t stay open due to a power outage on the property, many people still can work from home via the internet. As long as the electricity keeps flowing to those other locations, a university’s work can keep humming along.

::WEBEXTRAS ::
Disaster Recovery: Personal and Up Close. Many disaster recovery plans ignore the human aspect.

It's All About Risk. The shootings at Virginia Tech are a textbook example of the need for business continuity management.