Linking Data to Decision-Making
Five information technology leaders explain how they use metrics to measure the impact of IT projects or cost-justify new investments.
Pennie Turgeon, vice president for information technology and chief information officer at Clark University (MA), has a theory that CIOs are defined by the types of problems they are asked to solve. "I want to solve university problems, not IT problems. That's the lens I have on different projects when they come my way."
Whenever possible, Turgeon looks for data to support the decision-making process. When those decisions involve infrastructure, she partners with business unit leaders to get the data to justify a decision. "We have spent a lot of time looking at data and trying to decide how we can use it to make better institutional decisions and start to develop a culture that looks more at data and relies less on gut decisions," she said.
In one example, Clark participates in the TechQual+ survey to better understand what end users expect from IT organizations. "The first time we did it we got slammed on wireless. We knew we were going to be," Turgeon recalled. "We used that data to say this is where we need to make improvements if we are going to give students what they expect out of a modern university." The survey results were used to justify a four-year, multimillion-dollar upgrade project.
Benchmarking Against Peers
Leah Lang, director of analytics services for Educause, says the CIOs she works with are looking for data to help make investment arguments to other executives on campus. "That is the No. 1 use case we hear," she said. Educause has been gathering financial, staffing and service benchmarks since 2002 through its Core Data Service. CIOs look at peer organizations to assess their spending and staffing, to adjust the size of their departments and to make the case for budgeting and staffing requests. In 2016, Educause began gathering data about what campuses are spending on in-house infrastructure vs. external providers.
"We are in a data-driven decision-making culture now," Lang said. "We are no longer in an environment where you can go on instinct. So even though most of the time this data supports your instinct, unless you have the data to support your argument, you are vulnerable in that argument."
Jack Suess, vice president of information technology and CIO at the University of Maryland, Baltimore County, said that he uses the Core Data Service to look at staff numbers in specific areas. "I am trying to understand if I am aligning my staff resources similarly to my peers," he said. Here is one example: UMBC has more employees in telephony than other campuses its size because it is one of the few campuses that still handles its own internal wiring. "Digging a little deeper, because we are a relatively young campus we have been doing a lot of building, so we manage to save the campus money by doing this work in house versus going outside," Suess said. "That is where the Core Data Service can be helpful. It allowed me to explore and ask good questions."
But Suess emphasized that you have to know the local context to really understand the data. He noted that UMBC has historically been underfunded for its mission in the Maryland university system. "When I look at my IT budget compared to other peers around the country, there are places we are a little lower funded," he said. "But if I looked at the registrar's office of financial services, they would look the same way. So before I run into the CFO's office with this data, I have to understand the funding context."
UMBC has also been taking advantage of the new Educause Benchmarking Service (launched last year) to assess the maturity of universities' technology deployments.
"For instance, it allows me to see we are doing quite well in the areas we have decided to prioritize in cybersecurity," Suess said. "If I had prioritized certain areas and we were not doing well, that would scare me. There are other areas, such as cybersecurity training personnel, where I know we are equal to or behind our peers and I am OK with that, because we have made deliberate decisions that we are not going to put significant investment there."
Suess shares the benchmarking data with UMBC's president, Freeman Hrabowski III, who wants UMBC to be seen as more of a thought leader in analytics and cybersecurity. "Looking at the data, we were able to show him that we were doing well in both those areas. We have been doing a lot of little things over last five to six years and the benchmarking shows these have paid off. Without the benchmarking, all he has is my word."
Tracking Metrics
Florida International University is in the midst of implementing a university-wide business intelligence and analytics system. Vice President and CIO Robert Grillo said the effort grew out of a desire he shared with other campus executives for greater analytical capabilities to track metrics in finance, human resources and enrollment.
Grillo said one key to successfully making the case for such investment is not to sell it as an IT project. "I think it has been more and more difficult for any CIO to try to make a case for investment just for IT," he said. "If you are not tying it directly to the outcomes that the institution is striving for, your case becomes quite weak."
Grillo very rarely goes to the board or a committee to present a technology solution without having a partner on the business unit side with him. For the BI project, he sought out leaders in departments such as human resources. "They saw that they could write algorithms that would show them the cost of a graduate student post-doc employed through human resources but also getting stipends from finance," he said. "Those are the types of analyses that were very difficult to do without having a data warehouse built to pull all the data into one area. The provost understood that we were living in a world of performance metrics, and if we didn't understand the data we were going to be dead in the water. These tools were necessary."
Metrics are a similar force for change at the University of Wisconsin-Madison, where Sarah Buszka, a process improvement specialist and engagement coordinator in the Division of Information Technology, is working to help colleagues develop more data-driven approaches to evaluation and governance.
Buszka works in the User Services department, which includes the help desk, departmental support and a technology store. She has developed dashboards to help coworkers start thinking about communicating the value of those services to campus leadership, faculty, staff and students. "We are going through a change in IT governance and a change in CIO," she said. "That is introducing chaos in our landscape and there are a lot of questions being asked of us that were not being asked before."
"A lot of our challenges arise from the fact that in the past we weren't using data or metrics," Buszka added. "But just because we have always done something one way does not mean we should continue to do it that way."
The new governance group and new CIO will be asking lots of questions, she acknowledged. "My goal is to help folks get ahead of that. How can they be a part of this change instead of having change happen to them? We want to be proactive in creating a dashboard to communicate some of those high-level metrics around operations, strategy and the value added to the mission of the university."
Buszka's goal is to get beyond using just numbers such as help desk tickets to communicate value. "The number of tickets generated by a help desk for a year is a good metric," she explained, "but I want to get more granular: Who are the customers? Students? Faculty? Staff? How is it changing? We can bring in customer feedback from focus groups and advisory boards about how they want to interact. That pulls in that qualitative piece."
Her team has had conversations about what data to present and how to present it. "What I found to be most effective for me was leveraging pictures and visuals," she said. "For my group, I create a one-page document that highlights operational metrics and value metrics. Traffic lights — red, yellow and green — signify the status on things. This is very simple. It is not line items buried in spreadsheets, which is what we were traditionally doing. At a glance I can see green, we are good, and red, we are holding. There is not a lot of room for misinterpretation."
The dashboard Buszka helped create is shared directly with John Krogman, the division's chief operating officer, who reported to now retired CIO Bruce Maas. "It let him start asking more questions of our group," Buszka said, "which lets us dig deeper and bring some more data back to him, so he can understand the context of what is going on. It also caused him to start asking questions of folks in other groups that report to him."
Measuring Impact
Buszka recently shared the stage at an Educause event with Edward Gray, a systems integration and support specialist at the University of Mary Washington (VA).
Gray said his department has discussed linking metrics and measures up through critical success factors to the goals and objectives of the organization. "Having a clear linkage through that chain helps make sure you are measuring the right things and they are informing decisions based on where you want the organization to go," he said. What you are measuring is easy, he added. It is why you are measuring it that is not communicated as well.
Gray gave one example from the University of Mary Washington: A few years ago its developer group was spending a lot of time on one particular department's reporting requests. The IT department made the decision to start tracking time spent on which projects and for whom so it could prove that this group was monopolizing developers' time. "Begrudgingly the developers started tracking their time, but they understood the rationale for why they were being asked to do it," Gray said. If the supervisor said "track what projects you are working on" and left it at that, it would have been met with more trepidation, because time tracking is usually linked to changes in head count, he said. "By providing the why behind that measure, the supervisor was able to get better buy-in from the developers to collect the metrics they needed to go back to the customers and say, ‘you are monopolizing our time. We need to scale back on your requests,'" Gray explained.
According to Clark University's Turgeon, the important thing is to focus on the value drivers you can use to evaluate the impact of IT-enabled solutions. Sometimes there is a quantitative measurement, such as the number of classrooms renovated. Did you do the number you said you were going to do in the time you said you were going to do it within budget?
Other investment costs are more difficult to quantify. Clark had been considering replacing its ERP system with a cloud-based solution such as Workday. "There would be an opportunity cost involved in taking people away from other things we want to do and putting them on a migration project," Turgeon noted. On the other hand, there is a cost of staff members doing things in a very inefficient way because the current systems haven't been modernized enough. "Sometimes we try to quantify things that aren't quantifiable," she added. "We've got more analysis to do on that, but we did make a decision to stick with our current ERP for the next three years, giving us some time to do more in-depth financial analysis on what we would really gain and where we capture it."