Traffic Watch: Monitoring Network Flow at SUNY Geneseo

• By Dian Schaffhauser
• 04/09/10

Two or three times a week, Rick Coloccia receives an e-mail from the Research and Education Networking Information Sharing and Analysis Center. The mission of REN-ISAC is to watch over the cyber security operations of Internet2, the high-performance network run by a consortium of education, research, and other entities. To protect the integrity of its bandwidth, when REN-ISAC perceives that a particular member institution's network is generating traffic indicating the presence of a virus, malware, or some other malicious software, it notifies its administrative contact at that site. As network manager, Coloccia is that contact for his institution, the State University of New York (SUNY) Geneseo.

It's Coloccia's job to respond to the contents of those e-mails, which may report on the activities of a single computer or three or four on campus that are, as he put it, "crudding up the network."

Frequently, that errant traffic is being generated by student-owned computers. Geneseo has about 5,200 students, of which 3,200 currently live on campus in residence halls. As Coloccia pointed out, residence halls are students' homes. "They need to do in residence halls what they might want to do at home. The things students want to do on their personal computers go far beyond what we expect faculty and staff to do on their desktop computers. They install any software they feel like installing. They expect our network to support the software they find from anywhere and everywhere."

The practical upshot of that is that the residence halls look more like "the wild Internet than our computer lab network or academic network," he said.

Likewise, wireless networking is pervasive on campus. A user can move from the residence hall to the dining hall to the classroom and into an administrative building without losing network connectivity. The wireless network has traffic generated not only by the administrative staff and faculty but also by those student computers that the IT organization doesn't control. "There's noise on the network we don't recognize. There's traffic we're not familiar with," Coloccia said. "When the traffic generated from students becomes a problem, we need to figure out what it is and where it came from."

The network team uses a couple of open source utilities from Tobi Oetiker--MRTG, the Multirouter Traffic Grapher, and RRDtool, the Round Robin Database Tool--that use simple network management protocol (SNMP) to show how much traffic passes through every network interface on campus. "That will make eyes turn," admitted Coloccia. "We poll every single network interface on campus--all 10,000 of them--for bandwidth and error traffic." The usage reveals patterns in the amount of raw traffic flowing and the amount of errors passing through every port, but it doesn't provide much visibility beyond that. "We could see that something happened and when, but we could not identify what," he said. "It was virtually impossible to look at what happened last night at 10 with that system."

The university considered buying network management tools to make sense of those patterns and identify anomalies; but recognizing an anomaly when it happens requires setting a baseline. "It's not difficult to create a baseline in an administrative environment because I know that the computers on the desk in the purchasing office are going to do the same things day in and day out, Coloccia explained. "It becomes easy for me to build the baseline and create a deviation that will [generate an alert]: 'When things happen on that network that look different from this, alert me.'" It's extremely difficult, he said, to create a baseline for residence hall traffic, "which is why we haven't done it. There are too many false positives. We spent too much time talking to students, who'd say, 'Yeah, I downloaded a program. What's wrong with that?' And they're not wrong."

In 2007 Coloccia visited the Plixer International booth at the Cisco Networkers conference. The company was offering a free trial version of Scrutinizer NetFlow and sFlow Analyzer, which he decided to try. This set of tools captures network flow data and then can point to precisely which devices are consuming network bandwidth and what form that traffic takes. Another tool from Plixer, Flow Analytics, logs that information and provides additional reporting features, including historic snapshots.

"With Plixer software we can put an Internet address into the software and it will tell us what kind of traffic was moving at a certain time as opposed to that some traffic moved at a certain time. That precision is very important," Coloccia said.

As an example, he cited what happened a few weeks ago with two computers generating massive amounts of traffic in the university's student lab. "In my previous situation, I would have been able to identify precisely when the traffic happened, but I'd have no record as to where that traffic was going," he said. "I'm now able to put that computer's Internet address and timeframe into Plixer software, and it will tell me where that traffic came from, where it was going, and what protocols it was." That provided enough information for him to be able to track down why that traffic was happening and take the necessary next steps. In that particular case, software related to Apple Multicast DNS configurations on a couple of new Apple Mac OS X 10.6 machines was creating the traffic. "We're still not sure why at this point, but we know when it happened, and we know how it happened. We know what machines were responsible. And now we can be more proactive about the settings on these machines."

But what really helped Coloccia sell the university on making the investment in Plixer's "very affordable tools" was its ability to counteract legal threats from the Recording Industry Association of America, the Motion Picture Association of America, the Business Software Alliance, and even HBO.

"They effectively pay people to attempt to download music from the Internet. Whenever the source happens to be a university, they send a nasty-gram here to the person registered to the IP Address," he said. "Here's it's the CIO. She'd turn around and forward those to me. They basically say, 'We have proof we can download this copyrighted piece of material from your site. Stop.'" Whenever Coloccia receives one of those notices, he said, he goes to the NetFlow data and attempts to confirm the accusation. "As many times as the notice would pan out, it would also not pan out. More than once they've tried to tell me they could download music from a printer. I write back and say, 'That network address isn't valid on my network. You didn't get that traffic from me.' The NetFlow tool allows me to substantiate those notices."

On those occasions when the notice is warranted, the person responsible receives a warning. But the university has a very short list, Coloccia said, of activities that could get a student banned from the network. Spreading malicious software is one. In that event IT sends a note that tells the user to have the computer fixed within three days--which could be done with assistance of the help desk. If the malware isn't eliminated in that timeframe, the user is blocked from the campus network.

Abusive bandwidth usage and downloading copyrighted content--music or movies--are other activities that could lead to a ban. Almost every day Coloccia checks e-mail from Plixer's analytics software that tells him who moved the most traffic in and out of campus for a specified period. "The No. 1 outbound traffic should always be the Webcam," he said. "If it's not, then there's a computer that's doing something it shouldn't be."

For example, a researcher had downloaded a massive data set and was using BitTorrent to share it out to anyone else who wanted it. He didn't realize that he was generating a tenth of all outbound traffic on the network from the campus until Coloccia informed him.

But beyond helping IT identify abuse of bandwidth, the software also assist SUNY Geneseo in network uptime. According to Coloccia, NetFlow keeps track of how many flows are moving through devices. If he's having trouble with an appliance, such as an intrusion prevention system or a packet shaper, the Plixer software helps him determine whether the number of flows is exceeding the limits of that network device. When that happens, the appliance could fail entirely, shutting off Internet connectivity to the campus. "Because I have a database of all the traffic coming out of every single port on campus, based on the IP address, I'm able to figure out what network port is providing service to that device and shut it down," he explained. Prior to installing the Plixer software, tracking the source of those kinds of problems was difficult and time consuming.

Two years after implementing it at his institution, Coloccia is sold on network flow analysis software. "Something that I was without for many years, I now can't imagine being without. It's just allowed me to see so clearly inside the network, to find problems much more quickly. It's made my job so much easier to do."