Combining Data from Multiple Digital Learning Tools Produces Better Predictions

• By Dian Schaffhauser
• 08/09/18

A study undertaken jointly by the University of Maryland, Baltimore County and education technology companies Blackboard and VitalSource found that early activity with digital tools is a "strong predictor" of passing a class — even more so than the grade point average a student enters the class with. The organizations used student activity data from the university generated through Blackboard's learning management system and VitalSource's digital course materials and combined with the use of the Caliper Analytics standard from IMS Global Learning Consortium. IMS Caliper Analytics enables a variety of learning tools to return data that can be analyzed in aggregate.

The idea for the project, according to a report on the results, was to overcome a common objection made by institutions when they decline to "do" learning analytics: that data needs to be all-encompassing, otherwise the school will only get "partial understandings" of what a student is doing. For instance, if somebody isn't logging into the LMS, perhaps he or she is using the study time working with the curriculum.

At each quartile of activity, additional activity in Blackboard Learn and VitalSource cranked up the likelihood that a student would pass his or her course. Source: "Improving Student Risk Predictions: Assessing the Impact of Learning Data Sources," from the University of Maryland, Baltimore County, Blackboardand VitalSource.

The study involved seven courses, including math, physics, Spanish and statistics, encompassing some 838 student enrollments; the largest course had activity from 284 students; the smallest had 26. These specific classes were chosen because the instructors had been using VitalSource for multiple terms and used the Blackboard Learn gradebook. Student activities were tracked for four weeks, determined to be the amount of time that would still allow for interventions to help at-risk students.

The study found that UMBC students who showed high engagement with digital learning tools were 200 percent more likely to pass their courses than less active students. More specifically, incoming C students with low engagement and no access to VitalSource had a 37 percent likelihood of passing a class; similar students showing the highest level of engagement with Blackboard Learn and VitalSource had a more than 90 percent likelihood of passing a class.

Of course, as the report's researchers pointed out, there's no "directly causal relationship." The relationship between a good grade and effort is complex, and students need to do more than just access their online materials to boost their grades. For example, the report noted, "it could be that students with lower GPAs who access course materials very frequently are motivated and excited about a course, whereas a student with a lower GPA at a lower level of activity may lack motivation and confidence in the materials."

The project also uncovered additional findings:

• Patterns of activity differ significantly among courses;
• Learning activity data turns out to be a "more powerful predictor of achievement" than either demographics or educational background;
• Students with high levels of activity in multiple learning tools can dramatically increase their chances of successfully passing a class; and
• By combining data from multiple learning tools, schools can do a better job of making predictions about student achievement.

UMBC found the initiative compelling on several levels. "To my knowledge, this project represents one of the few empirical studies to look at student success through combined, digital tool usage data," said John Fritz, UMBC's associate vice president for instructional technology. "In addition, the IMS Caliper Analytics standard made it technically possible for a university and two ed tech companies to pursue shared interests, which is rare in its own right. We learned a lot and appreciated the collaboration very much."

The full report, with many more details, is openly available on the Blackboard website.