Machine Hunt: User Forensics at Salt Lake Community College

• By Dian Schaffhauser
• 03/12/10

With the Internet antics of 54,000 students and 3,000 faculty and staff members, security analyst Brandon Johnson at Salt Lake Community College can easily chew up a day to figure out whose computer on campus is doing the dirty work of a botnet or what user was logged into a particular computer at a given time when law enforcement or HR comes calling. In fact, Johnson said, the need to match a security incident to a particular computer and user crops up about a dozen times a week.

Traditionally, making that correlation has been difficult. As an example, when the police arrive with a subpoena for information--most frequently because an IP address originating at the college campus has surfaced when a user accessed some Web site with content that is currently considered illegal--it's Johnson's job to work with them to identify which computer was involved. However, IP addresses originating from the college look broadly the same to the outside world. "We have a global [network address translation] (NAT) set up, which means in a lot of our labs and staff and faculty areas, all [traffic] goes out with one IP address," Johnson said. Pinpointing the internal source IP and port can be challenging.

Pinpointing a Drip from a Fire Hose
Law enforcement typically provides an IP address, a destination port, and a timestamp. That's of general use. Johnson needs to apply those details against the netflow data the college collects and match them up with a specific machine that was active with that type of traffic at that precise moment. To give perspective, he estimated that the network processes around 100,000 flows every minute. "We have five or six gigs of traffic transiting our network every five minutes," he said.

He homes in on the offending machine through DHCP scopes, a range of valid IP addresses available for assignment or lease to client computers on a subnet--a lab or an office--then goes through the logs of numerous Active Directory domain controllers or lease logs to see what user was logged in.

If the offending machine is on a wireless network, the pursuit is even more demanding because there are open authentication areas on campus--comparable to wireless hotspots. The IT organization is attempting to eliminate those, but with all the activities that can go on at a sizable, multi-location institution, it's like a game of whack-a-mole. "We'd set up an account for a conference or somebody coming on site and for some reason the account wouldn't get expired or deleted. So we'd have people handing those credentials out," Johnson said. "At the library you can walk in and sit down to a computer and start using it with no authorization. There are a lot of labs like this as well. They're not open labs, but faculty request that lab coordinators have the machines logged in prior to instruction to save some time."

The problem with open authentication, as Johnson said he sees it, is that there's no accountability. "I'm not saying that everyone is going to do something wrong, but for when somebody does something wrong, how would we track who was sitting at that computer? We don't have any sort of [access control] at the doors, no cameras in the labs. This is a typical college environment. It's not high security."

IT is making a valiant effort to crack down on open authentication areas and force users to log into machines, but that's a slow process owing to user pushback. Not everybody at the college wants the imposition of log-in restrictions placed on wireless access to the Internet.

Once the machine in question in a legal matter is identified, it's handed over to the police as evidence. "Then that box disappears. We'll never see that hardware again," Johnson said, adding that replacing it becomes an insurance matter.

A Better Way To Tie a Machine to a User
For a while Johnson pondered writing a script that would attach itself to every login event to write a user's IP information to a text file. "But then we'd have to throw that into a database and write an interface for it," he said. "So we weighed that against getting something canned and ready to go and maintainable."

That canned something turned out to be Locate, a network appliance from eTelemetry that automates switch port and device identity. Locate keeps a record of who has used what IP address, MAC address, and switch port over a given period. The console of the system displays that information in real time, along with log-in details fed by Active Directory, such as name, department, and location.

Installation took a couple of hours, Johnson said, consisting of pointing a sniffer port on the Locate server to crawl authentication traffic coming from Active Directory and adding the Web-based management application to the network. Every domain controller has a client that reports login events for an account back to the appliance. In deployments where another LDAP is running, Locate can sniff Internet traffic from a switch port using MAC address correlation. After configuration, Locate performs its IP-to-people mapping by examining network traffic and pulling information from the switches being managed.

Now, when a request comes in from law enforcement, once it has been validated by HR, Johnson goes to the Locate appliance, enters the details provided by police, and determines who's mapped to the computer tied to that IP address for that time. The process takes minutes, not hours.

Besides reducing the amount of time dedicated to forensic work related to legal matters, the appliance has also come in handy for a situation that's even more common: identifying which computers on the network are responsible for anomalous activity, typically caused by malware infections. "I can look on my intrusion prevention system and see that somebody is making multiple rapid SSH connections to another box," Johnson said.

Where previously he might have been tempted simply to shut down all network activity for that user, now, he said, he follows a more measured approach. "We can't cut people off from doing their work," he said. "We need to call them and get more specifics to see if their [computer] is acting weird, if they've done something recently to cause an infection. Then we get a technician out there to repair the machine." Most often, he added, the problem begins through uninformed installations of applications or toolbars, or the user tells him, "Somebody sent me a Hallmark greeting card."

Although the primary use of the appliance is to help Johnson with his correlation work, it also offers other benefits, he said. For example, it can be used to generate reports on specific IP addresses in a lab to quantify how much the lab is used. That helps IT know how to allocate computers for different levels of demand.

Locate's reporting can identify the use of one IP address acting as a proxy for multiple Active Directory authentication points. Machines being used as kiosks for Webmail will show up in the list, but so will anomalies, such as a PC in the business office having "17 accounts mapped to it," Johnson said. "Certain things like that can get a little weird. What's the correlation? That can set you on a discovery path of what exactly is going on with that machine."

The current installation of Locate at Salt Lake Community College handles staff and faculty traffic only--as a bit of a test to see if it really addressed the needs laid out by Johnson. The current college license covers 2,500 accounts. To add those 54,000 student users--which Johnson expects to do in the future when budget allows--would require adding two or three more Locate appliances, he estimated.

According to the company, the Locate appliance starting price is $7,500. That covers tracking of up to 500 user accounts. Pricing scales from there based on user licenses.

The test has been successful. "If you're spending more than four or five hours a day trying to correlate IP addresses with accounts, if you're spinning your wheels trying to do forensics, just trying to find out where a machine is, you may be able to track that IP address down to the port, but to know who is using that computer hooked to that port is a great savings of time," Johnson concluded. "While it doesn't prove that the person [logged in] was the person using the computer, at least it gives us a starting reference."