All Stakeholders Must Engage in Learning Analytics Debate

• 05/31/16

Whether they like it or not, members of the educational technology community and faculty members should engage in debates about learning analytics algorithms, transparency and data access. That was the message delivered by George Siemens, executive director of the Learning Innovation and Networked Knowledge Research Lab (LINK) at the University of Texas at Arlington, during a recent Future Trends Forum discussion with futurist Bryan Alexander.

"You can hate something and still recognize that it is an important factor to think about," Siemens stressed. "You can think analytics are crap, but you can't deny they are influential. If we want a future that embodies values that are important to us, then we have to be active participants in the sociotechnical and economic spaces driving that change."

Siemens, known for his pioneering work with massive open online courses, is the author of Knowing Knowledge, an exploration of how the context and characteristics of knowledge have changed and what it means to organizations. He said the open educational resources (OER) community has missed the boat somewhat by ignoring the potential of analytics. "The textbook companies lost the battle for openness to the OER movement, but they have won the war because they have dropped the object being the content and instead have turned their attention to the intelligence around that content," he said. "Companies like Pearson and others recognized early on that you can't win free, but you can win against value layers. In a sense, we are fighting a ghost of an opponent because they are not putting their resources into the openness space to fight against the OER movement. Their resources are being directed to a completely different battlefield, which is why I keep arguing we need to be at the table."

Alexander asked Siemens for both short- and long-term prognoses for what is likely to happen with learning analytics. In the short term, he replied, we will see lots of companies trying to make money from analytics, but he doesn't see much that is useful coming out of it. "But the long-term potential is quite significant, because it will have an influence on everything we do in university settings," he said. It touches on resource allocation, teaching practices and how students learn, he noted — but will that be a good thing?

Education is a multi-dimensional, complex system, Siemens said, and yet many people propose single-variable solutions. "So add data analytics to this mess? You are going to keep having the same problems," he said. "You might be able to push a particular agenda that you have confidence in numerically, but not necessarily have a better system. So in the short term you are going to have a lot of vendors popping up. There are a lot of stakeholders who want quantitative or empirical data about how we are doing as a system. Yet everything we know about effective learning isn't reflected through analytics; it is reflected through quality of teaching, care, compassion and human consideration."

Noting that education is a highly contested space, Siemens said there are powerful forces that have an interest in structuring change in such a way that it has some potential for economic value — forces such as the startup community. "I have nothing against the startup community, so I don't mean to be critical about that," he added. "But it is just one narrative. And if you have individuals who have an unfettered ability to enact change or at least drive an agenda, and you don't have people aware of what is happening, you will have exactly the situation we have nationally and internationally around data access. Our data is being used as a commodity, and we are being monitored at a level we are not comfortable with. The best time to have paid attention to that was 15 years ago when it was starting to happen, and I think the same issues are arising with education. The best time to pay attention to the factors at play really was a while ago."

Siemens stressed that he is not opposed to learning analytics. In fact, he was the founding president of the Society of Learning Analytics Research. "I am well aware of what is possible with analytics and I am excited about the prospects," he said, but he fears the startup community will see it as "a wedge to cause this rumored disruption in education."

While analytics are not necessarily good or bad by themselves, they can certainly be used to favor one view of what learning is over another view. "One view might be that our ability to quantify and measure and sort students could be done better with analytics," he said. "On the other hand, analytics could be used to improve the ability to identify students who are at need and improve our ability to quickly provide support and intervene. I would like to think that if we do analytics well, we build a profile of the learner that the learner owns. We develop this idea of an open analytics platform where researchers see the algorithms, and people can see the bias in those algorithms."

Siemens also likes the idea of computed curriculum, where a course doesn't exist in advance — it is only created as a learner starts with a learning experience. "And once we connect their profile with the subject matter they are dealing with, that curriculum is computed to provide them with the best resources to support their existing learning and increase how quickly they achieve a degree or recognition," he said.

Currently, he added, universities are confined by so many structures that are time-based: the credit hour or how many years it takes to get a master's or bachelor's degree. "I don't see why someone from industry couldn't come back and get a master's degree in three weeks if they have the capability," Siemens said. "That is where we will see the greatest potential — if we start asking questions about which attributes of our current educational system are no longer aligned well with what we are capable of doing with analytics and technology."

Relationships with Analytics Vendors

Siemens noted that when the first learning management systems were introduced, university leadership often didn't have the capability to plan for the technology, which meant that the vendors drove the decision-making process. "And we are seeing the same thing happen with analytics. If you are not comfortable with the role data and analysis can play in decision-making or in teaching and learning, then ultimately, we have a power imbalance where the vendor makes the calls."

Looking at how some of the vendors are approaching the landscape, Siemens said he believes we are "going to see the most incredible lock-in we have ever had." He predicts this is the last generation in which universities are going to be looking at a new LMS. "Once you integrate your learning systems with your business practice systems, you don't change anymore. When was the last time someone changed their accounting system? You do it once every 20 years and it is hell. The learning analytics systems vendors are integrating with systemic and organizational software, and you are never changing again."

And once you are driven by analytics, he said, you are going to be driven by one fundamental factor: Can we control the data? If a faculty member uses toolsets that give a little more control to the individual and don't afford the organization the data it wants in order to make its models function, everything will be locked down, he asserted. "Pedagogically, analytics could become a tremendous disabler of pedagogical innovation as much as an enabler of more effective practices."

An Ethics Guide for Analytics?

During the Future Trends Forum session, Susan Adams, an instructional designer and faculty development specialist at Oregon Health and Science University, asked Siemens if he knew of any good ethics guides to how universities use analytics.

Siemens responded that the best guide he has seen so far was developed by the Open University in the United Kingdom. "They have a guide about how it will be used in the learning process, driven from the lens of learning rather than data availability," he said.

"Starting with ethics is important," he continued. "We should recognize that if openness around algorithms and learning analytics practices is important to us, we should be starting to make that a conversation with vendors. I know of some LMS vendors where you actually buy back your data. Your students generate it, and when you want to analyze it, you have to buy it back. So we should really be asking if it is open. If so, we can correct inefficiencies. If an algorithm is closed, we don't know how the dials are being spun behind the scenes. If we have openness around pedagogical practices and algorithms used to sort and influence our students, we at least can change them."

But today we don't even have that basic level of openness around the data, he said. "And even worse, we are not going to have that openness around the software systems, because most academics who don't like learning analytics just step away from the conversation. They don't try to shape it. That is why I have been very pleased with people like Audrey Watters and others, because they at least raise the concern and raise the flag."