Learning To Share

With a centrally managed data center, researchers can have their cake—supercomputing power at lower cost—and eat it too. Yet convincing them to pool resources can be a tricky political process

• By David Raths
• 03/01/10

With a centrally managed data center, researchers can have their cake—supercomputing power at lower cost—and eat it too. Yet convincing them to pool resources can be a tricky political process

As 2010 began, the University of Washington was preparing to launch its first shared high-performance computing cluster, a 1,500-node system called Hyak, dedicated to research activities.

Like other research universities, UW had learned the hard way that having faculty members purchase and run their own high-performance computing nodes in multiple small data centers scattered across campus is inefficient and expensive. In many cases in those settings, the CPUs are active only 10 to 15 percent of the time, and, in the absence of coordination, the effectiveness of data centers drops off over time. These kinds of inefficiencies can impel research faculty to leave the university, resulting in data center capacity going unused.

“This was a problem that was costing the university many tens of millions of dollars,” says Chance Reschke, technical director at UW’s eScience Institute, which serves as a matchmaker between research faculty and resources, including computing platforms. Clustered nodes, he notes, enable researchers to take advantage of much greater computing power. Clusters also allow central IT organizations to offer better support, standardization, and coordination than individual colleges.

“Deans of research-heavy colleges, such as engineering, struggled to provide the facilities that faculty expected, such as power and cooling,” explains Reschke. “It became a faculty recruitment and retention issue for them. New faculty come to the university expecting to have access to this type of infrastructure. Colleges have no equipment available to satisfy that demand.”

But IT leaders at UW and other research universities say that leading the transition from scattered small research nodes to larger, centrally managed, shared resources is a tricky political process. It often requires CIOs to invest their own discretionary funds, to understand the needs and working styles of researchers, and to sell the concept to others on campus. It involves studying different business models and governance structures to determine which one would work best on a particular campus. Finally, it may require calling for some shared sacrifice.

“We had no fairy godmother sprinkling funding on this,” Reschke states. “Instead, upper-level management had to be willing to make an investment in infrastructure and management, and individual faculty members had to invest their grant funds in order to get their own personal supercomputer in a university-operated cloud—at half the expense of what it would cost them if they rolled their own.”

Building Trust

As planning for the Purdue University (IN) Steele community cluster began in the fall of 2007, the biggest challenge was to establish trust between IT and researchers, according to Gerard McCartney, Purdue’s CIO and vice president for information technology.

McCartney says that researchers often are inherently skeptical that a central IT organization can give them the service that they want, adding that “those misgivings aren’t unfounded. They are based on real experiences where they have been messed around with—[for example] machines sitting on loading docks while they wait for weeks for something to happen.” Some central IT organizations just don’t have strong project-management skills in support of research, he notes.

McCartney, who invested $1 million from his discretionary funds to support the Steele effort, cites a few decisions as key to the successful project management of the 893-node cluster, comprised of technology from Dell, Foundry Networks (now Brocade), and Tyco.

First, he involved researchers in the decision-making process every step of the way, including vendor decisions. After getting researchers to agree to buy from only one server vendor, McCartney’s team got bids from five vendors. He recalls holding a lunch meeting with pizza for the researchers and letting them choose which bid they liked. There were no vendor names attached. Researchers chose based on benchmarking data, prices, and warranty details. “I think the secret sauce,” he says, the pizza notwithstanding, “was letting them pick the vendor.”

Another key factor in building trust, according to McCartney, was to make cluster participation optional. Researchers could choose to buy off the cluster price menu and take charge of their own nodes, but then they would be responsible for service and repair. Only a few researchers did that the first year with the Steele project, and fewer did it the following year when another cluster was built.

“That’s really what we are looking for—two researchers at a cocktail party and one says to the other, ‘Are you really still running your own cluster?’” McCartney emphasizes. “That’s when it kicks in, when one colleague says something like that to another, not when an administrator tells him he should do something.”

Faculty Behind the Wheel

The worst-case scenario for a computing cluster is to have a build-it-andthey- will-come mentality—and then nobody comes. To ensure that wouldn’t happen at Emory University (GA), three years ago CIO Rich Mendola set up a joint task force between IT and the research community to develop a business case to spend $2 million in startup funds on a high-performance computing cluster.

After pushing through a difficult IT governance process, the project team won approval for the 256-node, 1,024- CPU Sun Microsystems computing cluster. As the project moved forward, Mendola saw that involving faculty in a steering committee that developed a memorandum of understanding (MOU) about costs and service levels paid off.

“This has to be faculty-driven,” he advises. “I would never have someone on my team lead the steering committee. It wouldn’t work. You have to make sure researchers’ voices are heard.”

The single most active discussion at Emory was about making sure the charge-back model worked. The steering committee studied a range of options from giving nodes away to fully recovering all the cluster’s costs, and decided on something in between. “To get buy-in, we needed cost to be below market, so no one could procure this for themselves as cheaply,” Mendola explains, “but we didn’t want to give it away. So we settled on tiered-usage subscription charges, and we have plugged some of the funding from that back into upgrading the cluster.”

The steering committee continues to study usage, billing models, and ideas for future clusters.

A Condo-Commons Model

Many universities have adopted a “condo” model to structure their cluster project, whereby the central IT organization houses and maintains computing clusters for researchers on campus. In return, spare cycles on these computer nodes are available for use by the research community (in condo parlance, a common area). But each university applies a slightly different business model to its condo structure.

For example, at Purdue the Steele cluster is a condo that is “owned” by the community of researchers. Each research group or department is allocated research time commensurate with its upfront funding contribution. This is analogous to condominium assessments calculated on a square-footage basis.

Rice University (TX) uses a different condo structure for its cluster, SUG@R (Shared University Grid at Rice), a 9- teraflop cluster of dual-quad core Sun Fire 4150s from Sun Microsystems. Unlike clusters purchased by aggregating funding from multiple researchers, SUG@R started with a commons, a shared pool providing high-performance computing to the entire campus, and then added nodes as dedicated condos for particular research needs. The price to faculty for having central IT manage their cluster’s dedicated condo is a 20 percent “tax” of its CPU cycles, the revenues from which provide additional computing cycles in the commons. The researchers can also take advantage of the commons’ increased computing power at scheduled times.

“In 2007, we started looking at a better way to approach research computing, but we had to come up with multiple incentives for IT, for the researchers, and for the university in order for it to be sustainable,” says Kamran Khan, vice provost for IT.

A steering committee of 19 people, both researchers and IT staff, hammered out a business model and a memorandum of understanding for each researcher to sign. The deal offers Rice researchers more computing power at lower cost. If the total cost of four nodes, including shipping, cabling, switches, maintenance, and so forth, is $18,000, the Rice researcher might pay $13,500, with central IT picking up the rest of the cost, Khan says. Because the first SUG@R condo project was successful, Rice has launched a second, and it is studying a third. It has almost 400 condo accounts.

“We definitely see efficiencies in keeping systems up and running, in costs, and in offering more CPU cycles,” Khan says.

An ongoing steering committee oversees such issues as queuing policies, prescriptions of minimum hardware requirements, day-to-day operational review, and recommendations for changes to the MOU. “The steering committee gives researchers a place to work with us and look at requirements as technology changes,” Khan explains. “It has created a collaborative environment between IT and the research community.”

Among other benefits, Khan notes that many researchers have shut down their own dedicated clusters in favor of SUG@R’s condos, thereby decreasing cooling and power expense and achieving economies of scale in system administration.

In addition, researchers who felt parallel computing was beyond their means enjoy 10-factor performance gains in the commons.

There’s no doubt that a high-performance cluster model saves money, drives efficiencies, and forges stronger bonds between IT and research communities. From a long-term investment point of view, shared clusters also help retain and attract new faculty. Emory University’s Mendola, for instance, says he has spoken to new faculty members who have come from universities where they’ve had to set up and run their own small cluster. “They are thrilled to come to Emory and within two weeks run their computing jobs on our shared cluster,” he says. “That has definitely become a recruiting tool.”