Online Learning

Coursera Adds A/B Testing to MOOC Platform

• By Dian Schaffhauser
• 07/10/18

Coursera is now allowing for A/B testing on its MOOC platform. This lets instructors and researchers test a single variable to see how it affects some aspect of the course, such as participation. As a Coursera blog article explained, the testing is done by "randomizing enrollees into different versions of the same course" and then seeing what impact the change-up makes on the "learning outputs."

For one recent experiment, for example, an instructor wanted to examine the impact on interaction in the MOOC by gender-based "situational background cues." As Christopher Brooks, director of learning analytics and research in the Office of Digital Education & Innovation and a research assistant professor at the University of Michigan, explained in a video, he teaches a large number of introductory Python and data science courses. He wanted to find out whether different course imagery can cause "a difference in [students'] sense of belonging." A single condition that he tested displayed women working in a data science environment in the background of a lecture, while another used men doing the same.

The variation in conditions influenced the levels of interaction, according to the data found in the clickstream logs and discussion forums, but no difference in retention or completion, which, according to Brooks, was what he and his research team "anticipated based on [the research] literature."

The benefit of tackling this kind of testing in a massive open online course can't be overstated, he said. "On campus if I were going to do A/B testing, I'd generally have to use a couple of different sections and maybe I'd get 100, 150 or even 200 students. But here on the Coursera platform, I get tens of thousands, and these learners are from a global subject pool. They're not just our regular students on campus but they are more representative of a world population."

Other kinds of testing referenced by Coursera include the use of videos vs. slideshows vs. text to determine the optimal content format; the use of formative assessments vs. summative testing to understand the impact on learning outcomes; and the impact of using motivational efforts, such as stating goals upfront, for improving course progression for certain groups of students.

According to Coursera, instructors are conducting "more than a dozen experiments live on the platform today." "By empowering our partner community to experiment, we hope to accelerate the pace of pedagogical innovation on the platform," the organization added.