Finding the ROI of Online Programs

It's time to get serious about growing online education programs if your institution wants them to be sustainable. This Arizona State University research project is examining the costs and benefits of scaling digital learning in order to share replicable return-on-investment mechanics and tools for any school willing to take on the challenge.

• By Dian Schaffhauser
• 07/19/17

Are schools overpaying for the creation and delivery of their online courses and programs? Is the investment worthwhile? If the expense could be brought down through scaling, would that make online education more sustainable?

Those are the kinds of questions that may be answered in a research project nearing completion at Arizona State University's EdPlus Action Lab. The lab is headed by Lou Pugliese, a senior innovation fellow at the university and the founding CEO of ed tech stalwart Blackboard. Its current focus is on researching digital learning efficacy, alternative design and delivery modalities and adaptive learning.

What differentiates the lab from other initiatives of its ilk is an emphasis on seeking outcomes that can be put to work. "I got into this research business when I came to ASU about a year-and-a-half ago," explained Pugliese. "What I found is that most of the research that's out there — no matter what — is not really actionable. There wasn't a takeaway from much of the research. That's why I changed the dimension of the Action Lab to make it useful and implementable."

Beyond Best Practices

During a session at this year's ASU+GSV Summit, Pugliese led a panel discussion on the topic of "ROI Analysis in Digital Teaching and Learning" to share some of the preliminary findings from a research project run by the Action Lab and the Boston Consulting Group and initiated by the Bill & Melinda Gates Foundation.

The hypothesis was that online courses and programs have certainly matured, but many institutions haven't really grown their online efforts in ways that would allow them to benefit from the economic and academic efficiencies inherent in scaling.

The idea of the study, said Pugliese, was to look at the characteristics of diverse types of institutions and then pull together a synopsis of the research "that could be easily digested by different types of institutions and people" — and that would allow them to compare their operations to others that have succeeded in scaling.

"It's more than a best practices report," he noted. "It's really specific mechanics of the economic and operational efficiencies in these institutions that we surveyed that would directly translate [to other institutions]." The stress is on replicability, he pointed out: "If you can't replicate the practice, then there's not going to be cost savings, even at scale."

What to Measure?

Institutional participants were carefully selected for the study. To kick off the project, the research team examined different cost scenarios associated with most scaled digital environments, Pugliese said. That part of the work examined:

• Instructional delivery;
• Infrastructure of the physical classroom vs. online;
• Student support;
• Technology licensing;
• Proctoring costs;
• Other elements of technology, such as "bolt-on components";
• Course development labor;
• Course refresh costs;
• Management and administration; and
• Marketing related to student acquisition.

Then the researchers drilled down on institutional criteria. Those selected to participate have implemented digital learning at scale and are "relatively mature," said Pugliese. The percentage of digital learning undergraduate students vs. their face-to-face counterparts is "fairly significant." And each institution produces what would be considered "best-in-class student outcomes" and serves "a large number of target population students."

Because the Action Lab was working with the Gates Foundation, which has "really good relationships with schools through past research work they've done," Pugliese said, the team was able to take a subset of those schools and target them for "immediate access."

"Getting things up and running to do this research is hard," acknowledged Pugliese. By poking through the Gates portfolio, the researchers could fast-track that part of the work and identify those candidates "that fit a particular category: Did they bend the cost curve in lowering cost? Do parts of their portfolio have mixed institutions in terms of types — four-year, community college, private, for profit, etc.?"

Once the shortlist was developed, he added, the lab fine-tuned it further by examining undergraduate headcount, percentage of undergraduates online, how many years they've been implementing their programs, relative graduation rates, relative retention rates, percentage of Pell students, percentage of non-white and percentage of admitted.

The researchers felt it was important to get a cross-section of institutions that could also cover a number of criteria "We went through a lot of universities that were very willing and anxious and wanted to [participate in the study]," noted Pugliese. "They just didn't have the capability to do it. They didn't have the people."

Those left standing were six in number: Arizona State, the University of Central Florida, the Kentucky Community & Technical College System, Georgia State, Houston Community College and Rio Salado College in Arizona.

From there, the work turned to identifying practices that the researchers consider "replicable."

Sharing Tools and Fundamentals

Pugliese, who is the principle investigator for the research, shared some early findings.

First, the definition of who learns online "has changed radically," he said. While technology used to address the four-year undergraduate degree earner, now the population is considerably differentiated. As a result online practices require "methods and workflow and processes on how to do that." The final research report will share detailed examples of how different kinds of schools handle those elements of their programs, as well as tools that will allow readers to analyze their own institutions and do benchmarking against various metrics "around cost of efficient and effective instructional delivery, operation and maintenance, student support costs, those types of things."

One area that's ripe for "significant improvement in cost and efficiency" is the use of "centrally delivered operating models of digital teaching and learning," said Pugliese. "The ability to move from a decentralized approach to a centralized approach, where there's a higher degree of quality control and it's necessary to have strong professional development, is a definite trend in lowering cost overall." However, he added, "We found across the board in many cases, how you do that migration from a decentralized approach to a centralized approach isn't easy."

Another trend was the switch from outsourced OPM (online program management) to developing and promoting courses and programs inside the organization. "Institutions [that] historically are using OPM providers are increasingly taking that in-house for higher quality services to implement at scale," Pugliese observed. Among others, both Arizona State and U Central Florida have gone through that transition.

For both OPM and centralization, the report will share some "basic fundamentals" in terms of what has to be done, where investments will need to be made, and what the timeframe will be. "These areas are "fairly complicated," he said. "To put that in some sort of toolset we think will be most helpful."

The Catch-22 of Scaling Up

Along with the report, the research group is still developing its "dissemination strategy." That will involve not just sharing the report but also putting on workshops at institutional forums over the next year or two, said Pugliese, "where we can have a small quorum sit down for a how-to-do-it session." The gatherings will show how to use the statistical and financial data tools for doing benchmarking and comparisons.

While much is still in development on the project, one thing is clear: It's important to view digital learning activities through the lens of return on investment.

"There is still some reticence to want to embrace [digital teaching and learning] at scale," Pugliese explained. "Some institutions are still looking at this in a way that's very isolated for certain types of programs or certain types of low-risk opportunities. Some look at it from the perspective of having to keep up with other institutions in terms of marketing. Many of them, I think, even after years haven't really seen the benefit because it's a Catch-22. They haven't implemented it at scale and can't see the benefit. And they can't see the benefit until they implement at scale."