On the Wire: How Western Kentucky U Manages 40,000 Ports

The IT team at Western Kentucky University manages 40,000 ports on 700 switches across five campuses. Network Services Manager Jeppie Sumpter said it was a "fairly manual process" until a couple years ago. Dian Schaffhauser explains how the team automated most of that process.

• By Dian Schaffhauser
• 04/18/12

While wireless network management is a major time sink for college and university network administrators and a rich target of product development for multiple vendors, the wired network and its underpinnings--such as switches and cabling--are still required to keep the wireless infrastructure running. The final word on managing that physical network hasn't been written yet. Schools are still making improvements for how they do that and sorting out what's best to move to wireless and what should remain on the wire.

The Network Services team for Western Kentucky University addresses the networking needs of 21,000 students and 2,200 staff and faculty members. The set-up spans five domestic campuses, including the main one in Bowling Green.

While the institution's administrative and academic operations are still predominantly wired, a couple of years ago the campus crossed the 50-50 split for residential users using wireless network access instead of plugging an Ethernet cable into their computer or other device. Today it's largely wireless access for those who live on campus. But unlike some universities that have pulled out the wired ports within residence halls altogether, Western Kentucky still offers wired ports. As Network Services Engineer Jeppie Sumpter explained, "We've had several discussions with our housing group. To date it has been decided that we'd keep providing both services. We want to provide our students with a flexible, quality computing environment."

In fact, he added, IT anticipates a "small resurgence" in the use of wired connections as it promotes the idea among users that fixed devices--such as smart televisions--should ideally use a wired connection for optimal performance and in order to free up the wireless network for mobile usage. But, said Sumpter, "We will support them either way."

That network management effort currently encompasses 40,000 "hot" wired or always-on ports on roughly 700 switches. The biggest hassles have revolved around security, and up until a couple of years ago, addressing those challenges has been a "fairly manual process," Sumpter said.

How manual? As an example, some incident would surface where a machine on the network was causing problems or a user was caught doing something he or she shouldn't have done. Network Services would get an IP address and the date when the activity took place and track down the errant machine or user. The process for doing that was to figure out what switch port the incident took place on, which, in turn, would identify what building was involved. Then Sumpter or a colleague would go onsite to the communications closet where the network switch was located and literally trace the cross-connect for that port to figure out what room the activity originated from and go to that room to sort out what had happened.

From Manual to Managed Effort
As it has upgraded the network, the network services team began the tedious process of compiling data on every switch port on every plate in every room for every building. With port data in a database, the network administrators wouldn't have to go out and do on-site tracking anymore; they could simply look up information to identify troublesome machines or users, and then take further action.

As a proof of concept project the team worked to automate the process further by developing software tools to collect port data; but staff realized how tricky the job would be in an environment like theirs, which had switches from multiple vendors. In spite of the fact that the gear followed standards, differences among the variety of hardware meant it was going to take a considerable amount of work to write the modules to pull that information out of each switch. "Quite frankly, we didn't want to tie up our resources doing a lot of that," Sumpter noted. "It was important to us, but you [have to] prioritize, and we had other fires burning that we needed to pour our resources into."

Sumpter estimates the switch port database now contains about 80 percent of the university's locations. The holdouts, he said, are "legacy buildings with older switch gear in them that we just haven't had the time or budget to get to yet in terms of upgrades. But we're hoping to bring them into the fold soon as well.

The team was always "on the lookout" for a commercial solution that would do the port lookup work for them. Eventually, SolarWinds, the vendor that provides their primary network management system, came through for them. Western Kentucky uses SolarWinds' Orion Network Performance Monitor to monitor and analyze network routers, switches, servers, and other networked devices and to send alerts to notify the team about potential and current problems. But it's the company's User Device Tracker (UDT), made generally available in May 2011, that caught their attention for tackling the port management too.

One obvious advantage is that the software integrates with the other tools within Orion to provide a unified system of management. "I think the common buzzword people throw out nowadays is a 'single pane of glass,'" Sumpter said. "It's all in one spot, and it makes things easier to deal with."

UDT's main features are:

• It locates endpoints on the network;
• It identifies where an endpoint has been connected in the past;
• It sends alerts when a specific endpoint joins the network;
• It performs port usage and capacity analysis across the entire network; and
• It performs device-specific network capacity analysis.

In the months since the university has begun using UDT, "it's already proven its worth," Sumpter said. For instance, a Trojan machine or "bot" will surface on the network, and the network team will trace its location so it can be pulled off the network and have forensic work done on it.

UDT has also been useful in tracking down machines connected to the wired network and accused by copyright-holders of being involved in copyright infringement. In those situations, the user will still be able to access on-campus network resources but not off-campus ones. "We have mechanisms that give them a Web page that tells them they're not allowed to have Internet access until they've resolved this situation with our security office," Sumpter explained.

In cases where a device has been stolen and the MAC address identifying that particular device is known, the IT staff will put an alert on the wired network in case it surfaces there to be able to identify where it accessed the network from. (Typically, however, Sumpter added, stolen devices, primarily laptops and tablets, will be used on the wireless--not wired--network, in which case the team will use Cisco's Mobility Services Engine tools to figure out where it's located.)

Another problem where UDT comes in handy is in dealing with rogue DHCP servers, which generally get set up in resident's halls. "Somebody will plug in a little Linksys device they got at Best Buy, and they've plugged it in backwards, and so they start giving out IP addresses for everybody in that residence hall," Sumpter said. "In places where we aren't handling those issues with switch features, we'll typically use something like UDT to track that down to the ports. And rather than run into the room, wherever it was, we'll probably shut that port down. It's unfortunate that we have to turn somebody off, but the alternative is that they're messing with lots and lots of other people."

When a port is shut down, he noted, the help desk is notified so that when the person calls to find out why he or she can't get onto the network, support staff will know what happened.

Future Uses for Port Management
Network Services has pondered new future uses for UDT. For example, in an environment where people move equipment "all over the place," doing the annual computer inventory can be tricky, Sumpter said. UDT could be used to track down missing gear or even to compile the inventory for all machines based on when each last accessed the network.

As the university expands its use of voice-over-IP phones, UDT information could be used to help update the database that maintains those locations. Then, Sumpter pointed out, when somebody uses a phone to make a 9-1-1 call, "We'll have accurate location information and can be proactive about keeping up with that." Although products have been introduced to handle just that job, he added, "We don't want to implement what would essentially be duplicate systems, both doing very similar things, because they would also both be pounding on our network equipment, pulling all that same information out, and it's not very efficient."

Ultimately, a port management tool such as SolarWinds' User Device Tracking saves manual steps, Sumpter declared. "Many of us in higher ed are [working] with small teams. And we're dealing with situations where it's been a little bit easier for us to invest the capital in some of these products and tools than in personnel. We do that so we can free up those [people] to focus on more important things than running around in comm closets and tracing stuff down."