Incident Response at UT Austin

• 05/09/06

By Mary Grush

An interview with VP for IT Dan Updegrove

The recent break-in to an administrative database at the McCombs School of Business at UT Austin (TX), discovered April 21, may have compromised the personal data of a very large number of individuals. Early reports stated there were about 197,000 records in the database. We caught up with VP for IT Dan Updegrove for an update and some of his impressions after the first two weeks of the institution’s incident response.

Dan Updegrove VP of IT, UT Austin

Though the affected server did not fall under central IT, you’ve been involved in helping the business school and the university cope in the wake of this. Do you have any updates on the situation, especially on the extent of the damage?

The numbers of exposed individual records and exposed Social Security Numbers are somewhat less than originally reported, but still large. Through a press conference on April 23rd and communication on April 25th to everyone on the database with an e-mail address (roughly 185,000), we sought to publicize that McCombs affiliates were at risk and should establish no-cost Fraud Alerts immediately, pending further notification about which of their personal data, if any, had actually been disclosed. We also established a special hotline with a toll-free number that received 4,000 calls in the first week. After exhaustive analysis of database logs, the McCombs School is providing individuals with specific data disclosure information (starting May 8th). Updates on the incident and the university’s response can be found [on the McCombs School of Business Data Theft and Identity Protection Web pages].

What are some of the key steps in the process of reacting to such a breach?

When a security breach is discovered (or suspected), several processes should be engaged immediately:

-- Contact the institution’s Information Security Office (if they were not the ones who discovered the problem) so their technical expertise and incident response protocols can be engaged immediately. The incident response plan would be expected to include not only forensic analysis of data, systems, and networks, but also communication with executive management, law enforcement, legal affairs, and public affairs.

-- Unless advised otherwise by ISO, take the vulnerable/breached machine off the network. Under certain circumstances it may be advisable to keep the host on the network to enable ongoing investigation of the source of an intrusion, in which case special precautions must be taken to protect data on the machine. One approach is to replace institutional data with a bogus dataset that may serve as a “honey pot” to keep the intruder engaged while the ISO and/or law enforcement track the rogue activities.

-- Take immediate steps to avoid deletion of system and network logs, which can be immensely valuable for determining not only the source of the break-in, but also what damage has been done and what data may have been exposed or tampered with. Since such analysis in a complex case can take weeks, logs cannot be allowed to expire according to a routine schedule measured in days.

UT Austin experienced a major database intrusion about 3 years ago. It exposed about 45,000 people’s data. You were there. How do these two experiences compare, and are there any lessons that stand out from either or both? Were there any changes made at UT Austin as a result of the earlier incident, and if so, why didn’t they stop the recent break-in?

Since this incident is the subject of an ongoing law enforcement investigation, specifics about the origin and method of attack cannot be disclosed at this time. The most obvious similarity between the two incidents is that Social Security Numbers were exposed, which is especially frustrating since the university launched a major effort in 2002 to remove and/or protect SSNs in all our information systems. SSNs are no longer used as university ID numbers for students, faculty, or staff, and several million dollars have been spent to remediate databases and applications that formerly depended on SSNs for query access, database indexing, or database linking. Unfortunately, as the break-in revealed, we had not completed the effort for all central and distributed systems.

Was the McCombs School of Business server at greater risk by not being under central IT? D'es this signal a need for more control from central IT? Would central IT have been able to tighten up safeguards to the point that the perpetrator wouldn’t have succeeded?

A more important question, we think, is how soon can we re-architect our information systems so that academic and administrative units that require specialized applications can build them using access to more secure, centrally-managed data, rather than downloading databases for local information processing.

To what extent are UT Austin staff and students asked to take personal responsibility for their own personal data and information? To what extent can they?

I believe this is a shared responsibility between the university and the users. The university must have sound policies concerning data, systems, and network security supported by adequate resources for applications development, systems and network management, personnel training, and user support. Users must take precautions both to protect their data in university (and other) databases — selecting robust and unique passwords and using virtual private networks in wireless zones, for example — and to ensure that their own personal computers and home networks are securely configured and maintained.

What d'es it take to clean up a big security breach like this? What is the cost, especially related to any measures UT could have taken to prevent it?

For the 2003 incident, the university documented over $170,000 in costs to respond to the break-in, which was dominated by printing, mailing, and related staff compensation costs to notify the 45,000 affected individuals. This d'es not include the substantial staff time required to brief the prosecution, prepare for trial, and testify in Federal Court, which resulted in a June 2005 conviction, five years probation, and full restitution of the above costs to the university. It’s obvious that the university will incur greater cost this year to notify the larger number of affected individuals.

Do you have any insights you can share with CIOs, especially those who work in higher education environments? Are there special challenges for the higher education environment, and what should higher ed CIOs be considering now, to protect their institutions?

Too many institutions still rely upon the SSN as the university ID number, or have not expunged no-longer-needed SSNs from central and local systems. Distributed systems containing sensitive information should be minimized and specially protected. Vulnerability assessment cannot be limited to operating systems, but must extend to databases and Web application interfaces. Intrusion detection systems must be deployed and fine-tuned, as the Internet is an increasingly hostile environment. Finally, since creating an effective institutional response during an actual security breach is extraordinarily difficult, it is essential to develop — and test — incident response protocols in advance, just in case. For other guidance, see the Educause Security Task Force Web site.

Mary Grush is the editor of this newsletter.