SJSU MOOC Study Reveals Achievement Gains but Low Retention Rates

• By John K. Waters
• 09/03/13

San Jose State University has published the findings of a study of its recent experiment with for-credit Massive Open Online Courses (MOOCs). Student performance improved between the spring 2013 pilot program and the following summer program in all three of the initial classes offered (elementary statistics, college algebra, and entry level math), with the percentage of students earning a C or better in two of those classes surpassing comparable on-campus student performance.

However, the overall retention rate for the summer program dropped to 60 percent, compared with 83 percent this spring, which reflected "SJSU's decision to be more flexible when students signaled to instructors that they needed to drop the course," the university said in a statement.

The study, funded by the National Science Foundation, looked at the SJSU Plus program, which was developed by SJSU faculty working with leading MOOC provider Udacity. The initial program ran in the spring 2013 semester and consisted of the three courses, which were structured as MOOCs, though they were limited to 100 students per class. Half those students were already attending the university, and half were non-matriculating students, including veterans, community college students, and even a few high school students. SJSU added two courses to the program over the summer — "Introduction to Psychology" and "Introduction to Computer Programming" — and attracted nearly 2,000 students, including students from other countries.

All five MOOC courses were co-developed by university instructors and Udacity, which itself originated as an experiment at Stanford University. Udacity is one of a trio of MOOC providers currently dominating the market, which includes Coursera, a startup founded by Stanford computer science professors Daphne Koller and Andrew Ng, and edX, a joint effort of Harvard and the Massachusetts Institute of Technology (MIT) to create an open-source online learning platform. SJSU also provides edX courses as optional resources for its professors who want to use them for flipped classes.

In July the school announced plans to take "a short breather" from its MOOC experiment to evaluate the model and revealed disturbing pass rates for the first three courses of between 23.8 and 50.5 percent.

"We entered into these pilot programs to test these modalities with our faculty and our students," Ellen Junn, provost and vice president for academic affairs at SJSU, told Campus Technology at the time. "Part of that process is to collect data and then to analyze and interpret that data to get a better sense of how these online learning environments affect student learning, faculty workload, and so forth."

Last week the university issued a statement in which it listed a number of insights gathered from this evaluation of the six-month pilot program. On the positive side, online video proved to be such an effective educational tool that some SJSU instructors are incorporating SJSU Plus materials into their campus-based courses, the school reports. Also, the online model seemed to foster student connections "with each other, instructors and instructional assistants through every means available: text, e-mail, phone calls, chats and meetings," the school said.

On the downside: Students "who are unfamiliar with the demands of college courses" need help preparing for and keeping up with MOOCs. Also, students have asked for more flexible pacing of the classes beyond the traditional semester schedule. "Customized scheduling is unprecedented at SJSU," Junn said, "but we would like to explore this option."

SJSU has added orientation programs for all five of its MOOCs, as well as "tools that help students gauge their progress." Also, the instructors will check in with MOOC students more often in the future.

"Yet with these new results, we still aren't there," Udacity founder Sebastian Thrun admitted in his blog. "There remains so much more that needs to be improved. The summer pilot was the second iteration of a new approach. To all those people who declared our experiment a failure, you have to understand how innovation works. Few ideas work on the first try. Iteration is key to innovation. We are seeing significant improvement in learning outcomes and student engagement. And we know from our data that there is much more to be done."

Junn said SJSU plans to incorporate the lessons learned from its pilot program and resume offering SJSU Plus courses in January 2014.