Making the Switch to iSCSI Storage

• By Dian Schaffhauser
• 01/31/08

Client by client, iSCSI is claiming ground against fibre channel in the external storage wars. For Queensborough Community College in Bayside, NY, that day of reckoning came a week or so before Thanksgiving 2006 when the school's fibre channel storage system went down.

But that wasn't the first warning the Office of Information Technology had about the need for a possible switch in platform. According to information systems specialist Bryan Farr, that signal had occurred a few months earlier during a meeting with the school's previous vendor, where the school learned the fibre channel storage area network (SAN) they'd installed just a year and a half earlier was coming off end of life in the not-too-distant future.

The Queensborough campus, part of The City University of New York (CUNY), serves 12,000 students a semester from atop a former golf course, with about 15 buildings over 35 acres. A gigabit Ethernet network with Cisco switches powers between 140 and 150 servers, most from Dell. More recently, the school has begun buying IBM BladeCenter servers running VMware Infrastructure 3 to migrate legacy applications off of computers nearing their end of life. Rather than purchasing new individual servers for each application, QCC is pooling them on VMware.

While the main student information is housed at a CUNY central office, the community college maintains e-mail for staff, faculty and students, as well as application databases for various departments, a total of about 2 or 3 terabytes worth. Farr said e-mail consumes the biggest share of the data store.

And that was the clue to Farr and his colleagues that something had gone wrong. Suddenly, people were calling because they couldn't access their e-mail. "It disturbed a lot of things," he said. "We had to call departments and tell them what happened, explain to people, 'Yeah, it's down. We're working on it.'"

All they could tell was that the main controller for the SAN had an issue. Along the way the problem--whatever it was--poisoned the failover controller. They were, as Farr described it, "dead in the water."

It would take three days of being on the phone constantly to vendor support to work through the technical problems. The computer company sent a support person, but, like the internal staff, he had to sit and await instructions. "One of their tech support [people] was saying it was hardware," recalled Farr. "So they sent new hard drives. We popped those in. Didn't bring it up." Next, software and drivers were checked. Eventually, with a lot of recoding, the SAN was resurrected.

But the team knew it had to change out the campus' storage switches. That posed a follow-on problem. Not only would the school have to swallow the expense of replacing two switches, but if it stayed with fibre channel it would have additional costs related to moving from its existing 1 gigabit fibre channel infrastructure to the newer 2 gigabit. Also, nobody on staff was fully trained on fibre channel. "We knew how to get through some of the volume parts," said Farr. "But setting up targets and fibre channel worldwide addressing--we knew nothing on that level, which you'd need a technician or somebody fully certified to install."

iSCSI, which was built to run in an Ethernet environment, was suddenly on the table. With a modest emergency budget of about $40,000, a group consisting of executive director George Sherman, e-mail administrators Ralph Romanelli and Tony Hong, as well as Farr, started a search for a replacement SAN strategy by viewing vendor webinars. The choices quickly were whittled down to SANs offered by LeftHand Networks and those offered by another manufacturer.

The school chose LeftHand, said Farr, for two reasons: It could buy storage in smaller increments; and each unit is a separate entity, which means that if one goes down, the others presumably will continue running. The initial purchase included three entry-level LeftHand NSM 160 storage units along with five QLogic fibre channel host bus adapters (HBAs) to use with legacy servers that lacked gigabit capabilities.

The new hardware was delivered in December and installed early in January. Farr said it was well documented and had a small learning curve. The only real obstacle--a tiny one--surfaced when the on site installers failed to check for a new version of LeftHand's SAN/iQ software, which had come out about a month earlier and included a slightly modified interface.

Since that initial installation, the school has purchased three additional LeftHand units. "It was easy to add them right into the cluster," said Farr. "As soon as they were plugged in, [the system] started replicating to the new units."

Also, although bandwidth usage really never goes much above half a gigabit, the school has noticed a considerably reduced time for doing backups with iSCSI devices.

Eventually, Queensborough will unplug its existing fibre channel solution, once a few lingering applications are moved to the newer storage system.

Best of all, the new setup works. One day, when Farr was "playing" with a configuration on one of the IBM blades, he inadvertently disconnected it from the network. "Since it boots off the SAN, it couldn't see the boot." Farr cranked up the SAN snapshot ability, went "back" in time, rebooted the server, and up it came. Operations didn't go down; they failed over to another blade as they should. "It actually worked," said Farr. "Just like in the commercials."