edX MOOC Research Gives Clearer Picture, Challenges Assumptions

• By Dian Schaffhauser
• 04/01/15

If massive open online courses are goldmines of data, surely, edX must be the mother lode. MIT and Harvard University have just published a 37-page draft report that summarizes a multitude of findings from two years of hosting 68 courses on the popular MOOC platform. That encompassed 1.7 million participants, 10 million "participant hours" and 1.1 billion "participant-logged events." edX is a non-profit learning platform founded by the two institutions in 2012. (Those courses offered on edX by Harvard are available through HarvardX; those from MIT are available on MITx.)

The research team, led by Andrew Ho, a professor at the Harvard Graduate School of Education, and Isaac Chuang, a professor of electrical engineering and computer science and senior associate dean of digital learning at MIT, discovered the following:

• Growth is steady. Across all courses, cumulative enrollment has risen at a steady pace of 2,200 participants a day;
• Participation mostly declines in repeated courses initially, then it stabilizes. In 11 courses with repeated versions, participation declined by an average of 43 percent from the first to the second version. For the five courses that had a third version, participation was essentially unchanged from the second to the third version;
• However, declining enrollment isn't a foregone conclusion. HarvardX's introduction to computer science course, CS50x, actually doubled in size from the first version to the second. According to the researchers, this was due to a doubling of the course administration window as well as the additions of support for asynchronous participation (students could take it over a year-long period) and certification;
• Curriculum determines participation. The two institutions assigned courses to four rough categories: computer science (CS); science, technology, engineering and math (STEM); humanities, history, religion, design and education (HHRDE); and government and health and social sciences (GHSS). The CS courses ruled with an average participation of 68,000 vs. 19,000 for the other categories;
• Certification follows its own pattern of participation. According to the report, the CS and STEM courses had average certification rates that were about half that of HHRDE and GHSS courses — 7 percent and 6 percent vs. 14 percent and 11 percent, respectively;
• Certification rates explode when participants pay for the privilege through an "ID-verify" process — 59 percent for verified students compared to 5 percent for non-verified students, on average, across 12 courses. An ID-verified certification requires successful completion of the course and verification of identity through a photo ID. The other form of certification is referred to as an "honor code certificate of achievement," which certifies for free that the person has completed the course successfully but doesn't require a photo ID;
• Certification is a draw for more than half of MOOC participants. From a survey about intentions researchers found that 57 percent of respondents expressed a desire to earn a certificate; nearly a quarter did so. Among participants who weren't sure or who had no intention of earning a certificate, eight percent ultimately did so anyway; and
• Teachers like MOOCs. Among 200,000 participants who responded to a survey about teaching, 39 percent self-identified as a past or present teacher; 21 percent of those teachers reported teaching in the course topic area. This suggests, the researchers noted, "that even participants who are uninterested in certification may still make productive use of MOOCs."

What wasn't evident in the findings was a wide diversity among learners. As the researchers stated, the people accessing MOOCs "are disproportionately those who already have college and graduate degrees." While they don't consider that a problem, it does call into question the oft-repeated adage that MOOCs will make higher education accessible to the world, including people in its disadvantaged corners.

"These free, open courses are phenomenal opportunities for millions of learners," Ho said in a statement. "But equity cannot be increased just by opening doors. We hope that our data help teachers and institutions to think about their intended audiences and serve as a baseline for charting progress."

Also, the researchers have taken a stand against the idea that energy needs to be put into improving completion rates of MOOCs based on who signed up. In fact, the report redefined the MOOC population of learners from those who registered for courses but took no subsequent action to those who participated by logging into the course at least once.

"While increasing completion has been a subject of interest, given that many participants have limited, uncertain or zero interest in completing MOOCs, exerting research muscle to indiscriminately increase completion may not be productive," Ho explained. "Researchers might want to focus more specifically on well surveyed or paying subpopulations, where we have a better sense of their expectations and motivations."

The researchers recommended that institutions put more emphasis on formalizing the "flow of pedagogical innovations" between the MOOCs and their face-to-face counterparts on campus. "The real potential is in the fostering of feedback loops between the two realms," said Chuang. "In particular, the high number of teacher participants signals great potential for impact beyond Harvard and MIT, especially if deliberate steps could be taken to share best practices."

The research project may develop "top 5" lists to highlight certain course details, such as variations in interaction, scale and demographics. The MOOC with the most female representation, for example, is a museum course from HarvardX called "Tangible Things," while MITx's computing courses attracted the largest global audience.

"These courses reflect the breadth of our university curricula, and we felt the need to highlight their diverse designs, philosophies, audiences and learning outcomes in our analyses," noted Chuang. "Which course is right for you? It depends, and these lists might help learners decide what qualities in a given MOOC are most important to them."

A draft version of the report is available on the Social Science Research Network.