University of Memphis: Cooperation, Communication Key to Security
By Robert Jackson and Dr. Mark N. Frolick
Perhaps the best way to understand how security issues can affect a learning
organization is to experience them first-hand. Robert Jackson, Systems Administrator
at the University of Memphis, had that opportunity when a Microsoft SQL server
was affected.
Warning Signs
The University of Memphis IT department has several groups that are responsible
for various functions. The Intel Server Support Team (ISST) consists of server
administrators who are responsible for the security and well being of the Windows-Intel
servers, and service administrators are responsible for applications that run
on various server platforms. The compromised server was running the Windows
NT4 operating system with service pack 6, MS-SQL 6.5, and IIS 4 in addition
to an older version of a Web programming language, PHP.
In 2002, ISST received a warning message from the server-monitoring software
regarding disk space on the affected server. After working with the Web services
team, ISST discovered large amounts of disk space being consumed by file structures
hidden within the Windows recycle bin. This hidden file structure was enough
proof that the server had been compromised. The issue then became how to deal
with taking an important server off the network.
Enforcing Policy
The director responsible for infrastructure was notified immediately. After
evidence of the compromise was presented, ISST and the director agreed the server
had to be disconnected from the network. Proper officials within the department
were notified of the server’s compromise and finally agreed that it should be
disconnected from the network. The decision was particularly difficult because
it was the university’s online knowledge base and had been growing in popularity
following a series of promotions by the department. Once the server was taken
off the network, recovery efforts were started.
Because debates ensued about whether the hacked server could be returned to
service, 12 hours were required to restore the server: There were attempts to
recover data from the server instead of backup; time was required to rebuild
the server, as well as to reinstall all necessary applications. Clear security
policies and procedures could have eliminated the confusion that occurred during
this phase.
Forensics
A forensics investigation revealed hackers gained access to the system through
a blank password on the "sa" account of MS-SQL. Although the service administrators
stated a password did exist for that account, the ISST group determined there
were log entries indicating the "sa" account had been used to compromise the
server. Upon connecting to the server with the open "sa" account, the hackers
used the xp_cmdshell procedure, the result of a default MS-SQL installation,
to execute appropriate commands to gain full access to the server. Once full
access was obtained, the hackers installed an FTP server on the machine and
began to utilize the university’s bandwidth and storage capacity for illegal
means.
Teamwork and cooperation, two of the main tenets of the learning organization
model, were called into question when ISST presented the results of the forensic
investigation. The goal of any forensic investigation should be to inform and
educate, not to place blame.
Other Vulnerable Servers
Realizing there were probably other servers on campus running MS-SQL, the director
of infrastructure directed ISST to perform scans of other servers to determine
the university’s vulnerability. Although four additional servers were located
with no "sa" account passwords, this turned into a political issue for the ISST
group when their actions and methodologies for disseminating information were
questioned. This is another example of how a security policy could be used to
improve communications among various groups within the department. By setting
clear guidelines within the security policy, all parties would know what to
expect in the event of a security exposure.
The political fall-out from the compromised server resulted in a meeting with
the server administrators, service administrators, and IT management to discuss
security policies and procedures. The meeting highlighted the challenges faced
in a learning organization when the teamwork and cooperation aspect of the model
is confronted with a server compromise.
Communication and Cooperation
Several lessons from the compromised server can be applied to learning organizations.
First, it is very important that management include security as part of the
mission and vision for the IT department. Without appropriate security policies
and procedures, it will be difficult to ensure stable computing environments.
Security policies and procedures for addressing compromised servers must be
in place. Secondly, equilibrium between experimentation and security standards
must be established. It may not be appropriate to deploy an application into
a production environment unless appropriate security testing has been performed.
Finally, teamwork and cooperation must be stressed during times of security
exposure, especially when a server has been compromised. Server administrators
must work with service administrators to return a service to production as quickly
as possible. At the same time, service administrators must understand the importance
of securing, and keeping secure, the production environments upon which services
depend.
Communication between groups is one of the biggest challenges when striving
toward secure environments and when dealing with security breaches. As previously
stated, teamwork and cooperation also play a part in ensuring the organization
works together. If stakeholders know what to expect when a server is compromised,
predictable and dependable reactions can help with a smooth recovery effort.
Dr. Mark N. Frolick is the Western & Southern Chair of Information Systems
at Xavier University. Robert Jackson is a Systems Administrator at the University
of Memphis. For more information, contact Robert at [email protected].