University of Memphis: Cooperation, Communication Key to Security

By Robert Jackson and Dr. Mark N. Frolick

Perhaps the best way to understand how security issues can affect a learning organization is to experience them first-hand. Robert Jackson, Systems Administrator at the University of Memphis, had that opportunity when a Microsoft SQL server was affected.

Warning Signs
The University of Memphis IT department has several groups that are responsible for various functions. The Intel Server Support Team (ISST) consists of server administrators who are responsible for the security and well being of the Windows-Intel servers, and service administrators are responsible for applications that run on various server platforms. The compromised server was running the Windows NT4 operating system with service pack 6, MS-SQL 6.5, and IIS 4 in addition to an older version of a Web programming language, PHP.

In 2002, ISST received a warning message from the server-monitoring software regarding disk space on the affected server. After working with the Web services team, ISST discovered large amounts of disk space being consumed by file structures hidden within the Windows recycle bin. This hidden file structure was enough proof that the server had been compromised. The issue then became how to deal with taking an important server off the network.

Enforcing Policy
The director responsible for infrastructure was notified immediately. After evidence of the compromise was presented, ISST and the director agreed the server had to be disconnected from the network. Proper officials within the department were notified of the server’s compromise and finally agreed that it should be disconnected from the network. The decision was particularly difficult because it was the university’s online knowledge base and had been growing in popularity following a series of promotions by the department. Once the server was taken off the network, recovery efforts were started.

Because debates ensued about whether the hacked server could be returned to service, 12 hours were required to restore the server: There were attempts to recover data from the server instead of backup; time was required to rebuild the server, as well as to reinstall all necessary applications. Clear security policies and procedures could have eliminated the confusion that occurred during this phase.

Forensics
A forensics investigation revealed hackers gained access to the system through a blank password on the "sa" account of MS-SQL. Although the service administrators stated a password did exist for that account, the ISST group determined there were log entries indicating the "sa" account had been used to compromise the server. Upon connecting to the server with the open "sa" account, the hackers used the xp_cmdshell procedure, the result of a default MS-SQL installation, to execute appropriate commands to gain full access to the server. Once full access was obtained, the hackers installed an FTP server on the machine and began to utilize the university’s bandwidth and storage capacity for illegal means.

Teamwork and cooperation, two of the main tenets of the learning organization model, were called into question when ISST presented the results of the forensic investigation. The goal of any forensic investigation should be to inform and educate, not to place blame.

Other Vulnerable Servers
Realizing there were probably other servers on campus running MS-SQL, the director of infrastructure directed ISST to perform scans of other servers to determine the university’s vulnerability. Although four additional servers were located with no "sa" account passwords, this turned into a political issue for the ISST group when their actions and methodologies for disseminating information were questioned. This is another example of how a security policy could be used to improve communications among various groups within the department. By setting clear guidelines within the security policy, all parties would know what to expect in the event of a security exposure.

The political fall-out from the compromised server resulted in a meeting with the server administrators, service administrators, and IT management to discuss security policies and procedures. The meeting highlighted the challenges faced in a learning organization when the teamwork and cooperation aspect of the model is confronted with a server compromise.

Communication and Cooperation
Several lessons from the compromised server can be applied to learning organizations. First, it is very important that management include security as part of the mission and vision for the IT department. Without appropriate security policies and procedures, it will be difficult to ensure stable computing environments. Security policies and procedures for addressing compromised servers must be in place. Secondly, equilibrium between experimentation and security standards must be established. It may not be appropriate to deploy an application into a production environment unless appropriate security testing has been performed. Finally, teamwork and cooperation must be stressed during times of security exposure, especially when a server has been compromised. Server administrators must work with service administrators to return a service to production as quickly as possible. At the same time, service administrators must understand the importance of securing, and keeping secure, the production environments upon which services depend.

Communication between groups is one of the biggest challenges when striving toward secure environments and when dealing with security breaches. As previously stated, teamwork and cooperation also play a part in ensuring the organization works together. If stakeholders know what to expect when a server is compromised, predictable and dependable reactions can help with a smooth recovery effort.

Dr. Mark N. Frolick is the Western & Southern Chair of Information Systems at Xavier University. Robert Jackson is a Systems Administrator at the University of Memphis. For more information, contact Robert at [email protected].

Featured