2019 Impact Awards
Visualizing Academic Risk
After years of researching the factors that impact student success, Marist College developed an open source early alert dashboard that turns interventions into a proactive, rather than reactive, process.
Category: Student Systems and Services
Institution: Marist College
Project: Marist Universal Student Experience (MUSE)
Project lead: Edward M. Presutti, assistant director of data science and analytics
Tech lineup: Apereo, IBM, MariaDB, Red Hat, RStudio
Marist College is not new to learning analytics. In 2011, the Poughkeepsie, N.Y, liberal arts college participated in the Open Academic Analytics initiative as part of the Educause Next Generation Learning Challenges (NGLC) grant funded by the Bill and Melinda Gates Foundation. In the years since, the college has continued to build on its efforts to create an open source academic early alert system.
In 2018, Marist's Data Science and Analytics Department rolled out a tool offering a dashboard view of academic risk that faculty members can access through the institution's Sakai-based learning management system (LMS). Dubbed the Marist Universal Student Experience (MUSE), the system provides a proactive rather than a retrospective or reactive mechanism to improve the chances of student success.
Marist has done quite a bit of research into what impacts a student's success in a specific course, noted Edward Presutti, assistant director of Data Science and Analytics. "Whereas many efforts in higher education address generalized student success across the whole curriculum, this is more focused on how successful they are being in a specific class. The challenge is how to take whatever data you have and convert that into a metric for what a student's effort might be."
How It Works
By clicking on the tool icon for MUSE in the LMS, faculty members can bring up a dashboard of their class. They see a visualization of the students in their class and each student's corresponding risk status. They can view more detail on an individual student's activity and/or select an option to e-mail the student with their concerns, along with resources that are available to the student.
Presutti emphasized that the design had to be clean and simple. "When we launched this last fall, we wanted to make sure it was not adding to the instructors' burden and it certainly hasn't. We wanted to provide them something that is optionable for them to begin with, and secondly is a one-click icon that brings up a view of their classroom and their students instantly."
Professors are now able to clearly see student events as they relate to LMS use. Some professors are using the system to help determine participation metrics and refine their grading schemes regarding class participation. Some early adopters have been very proactive about using the tool, Presutti said. "We did a very small study in the spring that showed that an intervention of any kind — even just notifying the student that they may be at risk —has an impact on their final grade, with some significant performance increases."
Besides studying classroom activities in the LMS, the predictive risk model also pulls other data from the student information system. Financial strain can impact student success, as do other signifiers of engagement. Are students participating in clubs or playing sports and how many sports? Do they have on-campus employment or work-study? Campus employment might indicate engagement, Presutti said, whereas work-study might indicate a financial need.
Data Integration
Although the dashboard is easy for faculty to use, setting up the process to pull all the data together on a nightly basis to do the analyses requires considerable work behind the scenes. Like most campuses, Marist has a diverse set of systems. "Although we run Banner, not everything is integrated," Presutti said. "The enrollment management system is not 100 percent integrated with Banner. There are import/export utilities that we have to use with the LMS. The technical challenge is to understand all the diverse systems you have and come up with a mechanism so that you can effectively extract them and coordinate that into a good unit of analysis to use for training."
Because Marist has many student workers in data science, it chose to "roll its own" extract, transform and load (ETL) process using an IBM Netezza appliance. Data science at Marist is also an open source shop. "Our dashboard is now part of the Apereo open source community as is the research and everything we have done here," Presutti stressed. "We want to make sure everything we have done is open source, so our whole aggregation process runs on a [Red Hat] Linux box." Other open source technologies in the mix include RStudio and MariaDB.
Indeed, the original grant from the Gates Foundation in 2011 required that this learning analytics process would be open source and available to others, so that was a driving factor in how Marist structured its project. Presutti believes the focus on open source tools makes it easier to collaborate with other universities. For instance, through collaboration with North Carolina State University, a predictive model has already been implemented as part of that institution's student success initiatives. Also through a collaboration with the Joint Information Systems Committee (JISC) in the United Kingdom, a number of institutions there have piloted the models.
Although there are some tech hurdles to be overcome, one of the most challenging aspects of the effort is not technical at all. "It is getting buy-in from data custodians and getting the endorsement and support of senior management," Presutti noted. Certain data you just will never get and you shouldn't, he added. For instance, healthcare data is highly protected. But other key factors of student success are less obvious — like sports participation. "So when it comes to what we refer to as ethically sensitive data, we have to go through the process of deciding whether it is ethical. That is one of the challenges for any institution."
Next Steps
So what is next for learning analytics at Marist? "The next step has to be retention and prescriptive analytics," Presutti said. "We can identify students that are potentially at risk, but can we know which intervention steps are most beneficial to them? That is really the next direction. We are not there yet with an integrated intervention strategy. But that is where we are going."
Of course, there are already departments on campus that support the student population. "It is not the role of data science to tell them how to do things," Presutti said. "Whatever we do with them, it can't be a heavy lift for them. The same way we created the dashboard for faculty, it has to be something easy to use and that doesn't add to their workload."
Return to Campus Technology Impact Awards Home
