Online Learning

Harvard Tailoring the MOOC Experience With Adaptive Learning

• By Dian Schaffhauser
• 02/02/17

Harvard University has begun experimenting with the use of adaptive functionality in one of its massive open online courses (MOOCs). The initial finding is that students using the adaptive assessments learned more than those who didn't — and spent less time overall getting through problems.

Adaptive technology uses information gained as the learner interacts with the system to change up how a concept is presented by level of difficulty, order and types of help provided.

The experiment took place in a single HarvardX course, "Super-Earths and Life" (now available as an on-demand course), deployed in the current academic year.

As explained in a research paper on the project, the course includes several subsections, each with homework made up of several problems. Students were randomly split between an experimental group and a control group, each with about 435 people. Individuals in the control group were given a "predetermined, non-adaptive set of problems on a page." The experimental group received problems served up by an adaptive tool, the Student Centered Adaptive Learning Engine (SCALE), produced by TutorGen. All the problems in the course were "manually tagged" with various learning objectives, and those that made up the adaptive assessments were also tagged with level of difficulty: advanced, regular and easy.

SCALE estimated the student's mastery of a learning objective each time he or she gave an answer to a question tagged with that objective, and used that factor to determine what level of problem to feed the next time around. When a student finished a problem, a bit of code in the edX content would send a parcel of data about the learner and his or her response to a web application created by Harvard that integrated with the edX learning management platform. That, in turn, was "processed and sent to SCALE." When the learner moved to the next problem, the web app would query SCALE in real time for the next recommended question and serve that to the student.

The engine stopped serving problems for a subsection if it determined that the user's mastery level for a given objective was "sufficiently high" that it needed no more verification.

To be able to measure the effects of the adaptive experiences on learners' mastery of the material, pre- and post-assessments were added to the course and given to the students in both groups.

The Super-Earths and Life course was chosen, said instructor Dimitar Sasselov, a Harvard professor of astronomy, for two reasons: "The class covers a very interesting topic"; and "because it involves physics and biology — high-level science." In a prepared statement, he noted, "Students on an advanced level are able to push further, and the adaptive course is the ultimate way to accomplish that."

The researchers found differences between the experimental and control groups in several areas:

• The experimental group achieved a larger knowledge gain (19 percent higher) after completing adaptive assessments, compared with non-adaptive settings in the control group.
• The experimental group students also tended to move faster through the course material, taking 4.37 hours on task vs. 4.8 hours for the control group members.
• The adaptive problem students showed more persistence in course assessments, making more attempts on advanced problems (2.99 vs. 2.41).
• They also tackled fewer problems overall than the control group students (3.99 vs. 5.42).
• However, the research established "no statistically significant differences" in the rates of course completion or certification between the two groups.

"The idea is to tailor the experience so that students are more likely to get what they need," explained Dustin Tingley, VPAL research faculty director and a Harvard professor of government. "The broader mission is to make sure that students are really benefitting from the online learning experience." VPAL is Harvard's Office of the Vice Provost for Advances in Learning.

VPAL researchers and HarvardX technologists worked with TutorGen to implement the adaptive learning technology within the MOOC. Mary Jean Blink, the company's chief operating officer, shares a byline on the research. VPAL expects to continue its research on adaptive learning functionality in future HarvardX courses.

The complete findings of the project are available on the Harvard VPAL website here.