Guardians of the Data

With 1,300-plus databases under its management, the U Massachusetts Medical School has plenty to worry about when it comes to keeping data secure; but this database team has put in place change management practices that go a long way in protecting the resources in its care.

• By Dian Schaffhauser
• 05/16/12

The 11-person database and reporting systems team led by Director Ray Lefebvre at the University of Massachusetts Medical School supports a lot of applications. First, there's the medical school itself. On top of that, it also supports the business systems for Commonwealth Medicine, the health care consulting division of the medical school that provides health and human services for the state of Massachusetts. Between those two organizations, Lefebvre's database administrators, database developers, report developers, and technology specialists support 4,000 business users, 2,500 faculty, 1,100 medical school students, and about 10,000 remote application users on 25 enterprise-class business systems, 350 workgroup applications, enterprise reporting, and 350 Web sites under the med.edu umbrella. Data resides in roughly 1,100 to 1,200 SQL Server databases, 125 Oracle databases, and about a dozen MySQL databases.

In an era where every school faces the internal mortification, unbudgeted expense, regulatory nightmare, and public embarrassment of a possible data breach and students, patients, faculty, and staff face the potential plight of identity theft, an environment like this is ripe for a fall. In fact, keeping the data secure is among the primary challenges Lefebvre's operation faces during the work day.

At the same time, however, the Medical School wants to ensure--like universities everywhere--that students and researchers have access to resources on the network and the Internet as they need them.

Closely Guarded Openness
There are several approaches IT takes in order to protect the many data assets under its care while still granting secure access to users.

It starts with multiple levels of firewalls. The database servers are behind two firewalls. The applications are behind their own firewall, as are production, quality assurance, testing, and other areas covered by the team.

That leads to another security measure: "very heavy network segmentation," as Lefebvre describes it, "between the front ends and the back ends and the outside world--the big bad Internet, we like to call it."

Users can, for instance, gain public access to the Internet from within the university network. But when they need to access data that has personal identifiers or protected health information, the user has to go through extra security "checks and balances." That applies to those outside the department--the students and faculty--and inside the department, the database team members.

The group relies on role-based administration managed by Microsoft's Active Directory in order to minimize the "attack vector" inherent in database accounts access. And accounts for accessing data aren't simply handed out when somebody asks. Explains Lefebvre, people have to go through a very formal vetting process of requesting access through a ticketing system and having the request reviewed by security personnel.

Also, the database team uses whatever is at hand to segment duties. For that the database staff uses a number of tools. A primary one is Quest Software's Toad. Toad has been around for years. The original name stands for Tool for Oracle Application Developers, but it now also supports working with Microsoft SQL Server, Sybase, IBM DB2, and MySQL.

According to Lefebvre, by using different editions of Toad, the organization can separate duties. Standard practice is for data administrators to use "the least privileged approach to database accounts." "Assuming you're doing that," he says, "only the very privileged are seeing the databases that they're supposed to see. And those higher privileged accounts are in very few people's hands." With Toad, for example, production database administrators (DBAs) can access all of the environments with auditing control in place through the Toad DBA Suite. Developers can access only the development and test databases with the Toad Development Suite.

As of version 11, introduced in fall 2011, Toad allows for a database team to implement read-only access to production environments. The advantage of that is that it prevents developers who may have both sandbox and production databases open from making changes to the production one inadvertently. "There's no bleed-over between them," says Lefebvre. "You can't copy production right into test, for example."

The Change Advisory Board
Lefebvre comes out of a Sarbanes-Oxley environment, where public companies are expected to comply with a strict set of regulations, including internal procedures to ensure that financial disclosures are accurate. So change management is just standard operating procedure to him. When he joined U Mass in 2007, he began formalizing the database-focused change management process to address compliance with HIPAA and SAS70 regulations, which require change tracking and auditing capabilities. In the intervening years, the processes have really matured at the school.

A change advisory board (CAB) scrutinizes all change requests. That team has representation from every discipline in U Mass Medical School's information systems department, including help desk, network systems, desktop services, security, data center management, and a bunch of other areas. The team meets weekly.

Most of the five to 10 production changes made in a day involve backend code changes to applications, such as adding a new field to a database report or updating a front end application that isn't delivering the data desired by the users. Even as those changes are vetted by the CAB, there's a pass-off that occurs among the various parties working to implement them. "Developers can do the changes in dev and test. Then when they want them done in quality assurance and the production environment, they have to go through their DBA brethren, who make the changes in there," says Lefebvre. "So the developers in dev/test use Toad to do all the developing and the debugging work, and then my DBA staff uses Toad to do all the deployment work."

If a change is required between regular weekly meetings, CAB participants vote electronically via email. When a proposed change is hotly contested by the CAB, it's escalated to a CIO-level person, who weighs in to act as the tie-breaker.

Once CAB gives approval, the database team implements the change. But on top of that, it's also recorded. In this part of the change process, screen recording is captured as a database person performs a specific change. That recording is saved and available for replay, such as when a change goes awry and needs to be dissected or an auditor comes in and wants to track the steps taken for a specific request for compliance's sake.

"They can actually see that we recorded it, and we have it on a calendar that we use for scheduling all of our work through as well," Lefebvre notes. "It's pretty formal. It blends right in there."

Those recordings, done with Screenrecorder, a free Microsoft tool, are stored in a secure shared storage drive as Widows media files averaging about 10 megabytes in size each. Lefebvre points out, "We've been doing this for about two years, so we've got gigabytes of storage behind it. We're committed to it."

Following the Right--not the Easy--Way
A third major aspect of protecting data at the Medical School is to use data masking and de-identification using an inexpensive datamasking tool. "If you break into our development tests or QA, you're getting fake data," Lefebvre explains. That happens any time a production database with personal data in it leaves a "hyper-secure" network segment and goes to a less secure segment that's more accessible. That includes, for example, scrambling birth dates, first names, and last names. "At one time it was real data, but now it's changed to look like fake data. And we don't even do anything with Social Security numbers anymore," he adds.

Often Lefebvre and members of his team find themselves seeking out inexpensive ways to simplify and automate the little chores that make up the database work they're plowing through. They've found Idea Pond, a community site specifically for Toad users to share ideas and questions, a useful source for recommendations. That's where the pointer to the data masking tool came from, as well as for the screen recorder.

Lefebvre has hawked the idea to Quest of including a screen recording function within the Toad toolset, but that hasn't happened yet. "I would like it to be able to click a button, say, 'This is a production change; please record it for me.' That would be nice."

In the meantime Lefebvre and his database team will continue seeking out new ways of securing their operations--at times suffering kickback from some of the very users they're dedicated to protecting. "I think we might be on the leading edge around maturity. The reason I say that is because sometimes researchers and educators will say, 'How come I have to do this? Such-and-such over at this university doesn't have to do anything like this--they can just make the change,' or 'They can just do what they want.'" Sounding like a parent, his response to whiners is often: "Well, just because they don't have to doesn't mean it's the right thing."

By leveraging the tools and technology available to his staff, Lefebvre reports that to date there's been no data leakage. "The steps we've taken help me sleep better. If I weren’t doing all that, I would worry."