Monitoring the PACE of Student Learning: Analytics at Rio Salado College
At Rio Salado College, where all 41,000-plus students attend classes online, instructional priorities include a strong emphasis on personalization--helping nontraditional students reach their educational goals through programs and services tailored to individual needs. To achieve this personalization, the college has implemented weekly starts (in which students can choose to start a class at the beginning of any of 48 weeks throughout the year), 24/7 technology and academic hot lines, easy access to online advising, and now a Progress and Course Engagement (PACE) system for automated tracking of student progress--with intervention as needed.
Several institutions have developed learning analytics tied to their course management systems, specifically to provide early interventions that can help at-risk students. Detecting "at risk" behaviors requires a tracking system and sophisticated data modeling. Michael Cottam, Rio Salado College's associate dean over instructional design and new program development, spoke with Campus Technology about Rio's PACE system and the college's collaborations with other institutions to share predictive data models.
Mary Grush: What has Rio Salado College been doing in the area of learning analytics for early intervention?
Michael Cottam: Over the past couple of years we've analyzed data from our LMS, to try to isolate the behaviors of the successful student--and those of the student who is not so successful. We wanted to be able to identify at-risk students early because the earlier in a course you can predict whether a student is going to be at risk, the more time you have to put interventions in place--support structures, contacts, and so on that could help mitigate the risk.
As we crunched data from tens of thousands of students, we found that there are three main predictors of success: the frequency of a student logging into a course; site engagement--whether they read or engage with the course materials online and do practice exercises and so forth; and how many points they are getting on their assignments. All that may sound simple, but the statistics we encounter are anything but simple. And we've found that, overwhelmingly, these three factors do act as predictors of success.
Grush: What system do you use, and how does it work?
Cottam: We call our system PACE: Progress and Course Engagement. The reports we generate show green, yellow, and red flags--yes, like a traffic light--so that instructors can easily see who is at risk. We can predict, after the first week of a course, with 70 percent accuracy, whether any given student will complete the course successfully (with a grade of "C" or better). That's our "eighth day" at risk model. A second model includes weekly updates using similar predictive factors.
Grush: Have you implemented this across all courses?
Cottam: To date, we've run several pilots, and we are just about to implement PACE across all our online courses.
Grush: What are the actions taken when those colored flags come up?
Cottam: We identify the level of risk for every student in a course, which helps us to focus instructor, advisor, and other institutional resources on quickly helping the ones who are most at risk. There are many possible courses of action. For example, faculty and advisors may call students identified by PACE as at-risk to try to find out the reasons those students are struggling and to determine ways to help them. That contact can sometimes make a big difference but, of course, sometimes not.
Grush: Did you develop this system in house? Are you collaborating or sharing your work with other institutions?
Cottam: Yes, PACE was developed in house. But we are collaborating with other institutions, such as Purdue, where that university also developed its own system for early intervention, a system they call Signals.
We are also participating in what we feel is a very important Gates-funded project of WCET, to validate the Predictive Analytics Reporting (PAR) Framework. Several WCET member institutions are aggregating more than 400,000 student records (with no personally identifiable data) that will be used to investigate questions surrounding student progress, completion, and loss prevention. Ultimately the findings will inform data models that can be leveraged by local systems such as our own PACE system.
Grush: Which other institutions are participating in the PAR project?
Cottam: Institutions include the Colorado Community College System; the American Public University System--APUS's Phil Ice is the PI for PAR; the University of Hawaii System; the University of Illinois-Spingfield; and the University of Phoenix. Vernon Smith here at Rio is leading our institution's participation in the PAR effort.
Grush: Besides ways to detect at-risk students, are there other valuable insights institutions could get from these learning analytics?
Cottam: One direction that I hope we can take with learning analytics is to look more deeply into the effectiveness of specific learning designs and learning materials in our online classes. If we can gather meaningful data on how our students interact with the course materials and then can correlate student actions with assessment outcomes, we have a chance to find out which design elements contribute most to student success. Then, we can make data-based decisions to improve our courses.
Some of the elements of learning analytics and predictive analytics are already finding their way into new versions of learning management systems, and I believe that trend will continue. However, another area that is particularly exciting to me is the potential for combining analytics with open educational resources. If there are OER learning objects on the Web with common assessments, and we are able to gather data on their effectiveness for a large number of learners and institutions, the impact of learning design improvements can extend beyond a single class or section. Shared data and transparency about learning design and learning outcomes has the potential to change the way we approach student success.
Grush: What's in the future for learning analytics?
Cottam: Analytics have been used in the corporate sector for many years. Look at Netflix and Amazon, companies that have used what they know about their customers to improve their experiences and to increase sales. In education, we are not looking for sales, but for student success, and I believe the impact of analytics can be as great in education as it has been in business and marketing.