It Was the Best of Times, It Was the Worst of Times . . .

• By Terry Calhoun
• 09/10/03

First, we had the budget cuts and were probably hitting this semester understaffed. Then, in many higher education-intensive parts of the country, the lights went out. Following that was the worm, Blaster. Then came, Sobig. And then the students came back, bringing with them their infected computers.

More than one CIO was quoted in the news media recently saying that this was the most difficult time ever to be IT staff on campus. And it was challenging, no doubt about it. But we think that, on campus after campus, the IT staff came through with shining colors. There was a lot to handle, but it was handled well!

So perhaps the most famous words of Dickens apply in all their discordant meanings: "It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way—in short, the period was so."

Impacts on the campus information technology infrastructure probably affected the campus-wide community, in general, as much as any one thing ever has. Read the following to yourself while imagining that you are listening to "Weekend Edition" on NPR and, as voices report from these campuses, there is stirring background music going on:

• At Auburn University lines of students wrapped around support offices two weeks after the Blaster worm first struck, initially downing the residential network.
• Temple University provided students with free anti-virus software, which is probably why—along with 90,000 warning e-mails and 27,00 paper flyers—only 400 of 35,000 student computers appear to have been infected.
• Duke University was set up to filter out virus-laden e-mail (it filtered out 2.5 million) and only a handful of machines on campus got infected.
• Harvard University puts incoming e-mail through a virus filter—in its arts and sciences department, 36,000 infected messages were stopped in the first 9 hours of the software’s implementation.
• Brown University’s network registration tool scanned newly-connected machines and instructed the owners without needed patches and updates to go get them—that was about half of all students.
• George Mason University cut off all residence hall access to 3,600 students for a while—after many students failed to sign statements that they had run anti-virus software and placed patches.
• At Columbia University a system-wide spam filter protected computers from viruses, even so consultants were also busy working in the dorms with student machines on site. Due to strong rules on computer user rights, however, student computers at Columbia and Barnard were not quarantined from the network.
• At the University of Maryland, returning students going into the network were directed to a Web site telling them to apply patches for the Blaster worm; those who did not do so within a fixed period of time were kicked off the network.
• At the University of Virginia, about 800 student-owned machines were kicked out of the network by security "bots" and were not allowed back in until obtaining CDs and loading up on protection.
• Oberlin College suffered "near meltdown" on August 21 due to students returning to campus with infected computers—9 out of 10 Windows machines were infected.
• University of North Texas was cleaning off 16 computers every hour and a half—and charging students $30 to do it. Students were not permitted to log into the network without first proving they had clean computers.
• Vanderbilt University shut down connections to 1,200 computers after finding out that as many as one-fourth of all student computers were infected. It took days to get service to them all turned back on.
• Salisbury University shut its residential network completely down for a day, this after a 2-week period spent cleaning off 500 university computers.
• MIT shut off service to infected computers and blocks traffic to and from suspected machines.
• At the University of Illinois, a team of 30+ network technicians worked on students’ desks to patch and check student computers in the residence halls. Some students faced a week’s delay in getting permission to get connected.

Wow. And those scenarios were played out on several thousand campuses across North America.

Now the analyses begin. How much did it cost us in overtime and lost productivity? Yes, we really do need to take a look at our policies about the responsibilities of users and how much support we will provide—and at what cost—to student machines. On some campuses, networks were shut down for days. On others, there was little impact. Why? In some cases it was due to long-standing procedures and thoughtful implementation, ahead of time. On other campuses, fast-thinking IT management whipped teams together and somehow managed to grab students’ cooperation and thinking to head off larger disasters.

Next week, we’ll have a guest opinion from J'e St. Sauver of the University of Oregon, who’s been looking at what happened, where. He will share his opinions about why some campuses got hit so hard and some didn’t—and also share some lessons learned.