As the Learning Management System (LMS) rapidly evolves, becoming more and more collaborative and integrated into the campus classroom, how can we measure where that evolution is taking students? How can the effectiveness of the social and collaborative technologies that are part of a good LMS be measured?

From an LMS provider’s standpoint, the more open and flexible the LMS, the more it can be integrated with other programs for robust analysis of student activity and interaction.  According to Lou Pugliese, president of online learning solutions provider  Moodlerooms, that kind of integration is needed. Technologies exist to measure student data and interactions on a large scale, Pugliese says: The focus now is how to effectively collect data and conduct reporting on-demand within the LMS. “Over the past ten years, the LMS has managed to record the most basic of student interactions and activity, but we’ve barely scratched the surface in enabling universities to analyze data on an institutional level,” says Pugliese. “However, new developments in analytical technologies will provide educators with the ability to measure interactions within the ever-popular collaborative tools present in today’s LMS environments. Moving beyond simple traffic reporting to more comprehensive online behaviour analysis will be critical to make more effective intervention decisions.”  

For further insights, we spoke with Stephen Downes, who works as a senior researcher for the National Research Council of Canada and specializes in online learning, new media, and pedagogy. Downes’ daily newsletter, OLDaily, is distributed to thousands of subscribers worldwide. In this interview, he discusses the continuing evolution of the LMS, how collaborative tools can be better used for online learning -- and how student participation might be assessed in the future.

Downes is a guest speaker at an upcoming Campus Technology webinar on Oct. 21 on “Facilitating Social Interactions: Measuring Engagement and Promoting Academic Success within the LMS,” which will also feature Pugliese, along with Colin Beer of Central Queensland University’s Curriculum Design Unit. You can register for this event here.

Campus Technology: How do you see open source learning management systems evolving over the next few years?

Stephen Downes: I’m seeing two major trends -- the first is that these systems are becoming more and more integrated with others, especially university student information systems. I see more of that in the future – centralized authentication and so forth, and an increase in things that we’re already integrated with, such as OpenID. There will be more and more of that. 
The second major trend is a proliferation of specialized modules. Look at Moodle. There are modules for RSS aggregation, for integration with LAMS [Learning Activity Management System], for integration with Second Life, and a host of other things. The net effect of those two trends is that the learning management system is becoming less a standalone product, and more a product that is integrated with a wider ecosystem of services and applications.

Do you see that integration as a good thing?

It’s a good thing, but it also presents challenges. Now that you have a way for a piece of data to move from Facebook to a learning management system to a student information system, there are real issues when it comes to security and privacy. There are certain expectations of learning management systems, and security is one of them. Privacy is another – the idea that these systems are not going to be open to the entire world.  Managing the connections with [other types of software] while maintaining those expectations is going to be a significant challenge.

If we approach learning management systems as collaborative tools, what best practices you can suggest for facilitating learning among students using an LMS?

[There are many, but] one that springs to mind is this: Don’t put too many people in the same space. Say that we’re running a massive online course with 1,200 people [enrolled].  The only way to manage a course like this is to [divide up] the students so we don’t have 1,200 people trying to comment in the same space. Instead, we encourage them to use their own blogs for that.

The tendency [with collaborative tools] is to try to bring everybody into a single environment in order to foster collaboration, but my preference is to foster the collaborative activity itself outside the environment, and to use the environment only for reporting and communication.

There are some really superb collaborative tools out there – Google Docs for example.  If you’ve got 100 people in your class and you want them to collaborate – well, move them into groups of [perhaps] eight or so.  Have them collaborate and then come back to the main area….  You don’t need to do everything in the collaborative environment itself, and in fact, there are many reasons why you shouldn’t do everything [there].

People think of collaboration as lining everybody up under the same banner. But collaborations work not when there is unanimity but rather when there is diversity. Having people perform different roles that draw on their different strengths, and having each person bring what is unique to themselves to the table, then valuing that contribution and finding ways to synthesize those individual contributions – that produces a stronger, more rewarding collaboration.

Speaking of the LMS as a collaborative tool, should instructors concentrate on framing the conversation using these tools, or should students be allowed to control the content much more than they would in a traditional classroom setting?  What do you think leads to more organic growth in the discussion and in student engagement?

I’m not sure I would express the possibilities in such sharp terms -- either the teacher frames or the students control. I prefer to think of the teacher, especially in non-presentation and non-content-based types of courses, as more as a co-conspirator, if you will. [In those cases, instructors] are like students [in a way]. Students should look to them as an example of what should be done, but without doing exactly what has been done. 

Of course, a lot of learning is imitative, so it becomes important for the instructor to say, “I’m just representing one way of doing this.  This is what I do,” then perhaps point to people in the community as examples.

As we discuss the LMS and collaborative learning, how can analytic technologies be used in combination with an LMS to measure student engagement? Is that something that’s being done effectively now?

There are different tools for measuring learning engagement, and most of them are quantificational. The obvious ones [measure] page access, time-on-task, successful submission of question results – things like that. Those are suitable for a basic level of assessment.  You can tell whether students are actually doing something. That’s important in certain circumstances.
But to think that constitutes analytics in any meaningful sense would be a gross oversimplification.

There is a whole set of approaches having to do with content analysis. The idea is to look at contributions in discussion forums, and to analyze the kind of contribution. Was it descriptive? Was it on-topic? Was it evaluative? Did it pose a question?

Each of these contributions can be identified by markers in the sentences. [The software then] looks for these markers in the discussion list and produces a profile of what level the student is working at. You can map that to, say, Bloom’s Taxonomy to help evaluate at what level the person is working.

Again though, these are rough indicators, and indicators only… There’s going to be much more that isn’t detected by these systems.

Beyond these, how do you best measure student engagement and achievement in online learning environments?

The answer is, “By not relying on any single measure.” Different kinds of assessments give you different results; you want a variety of different assessing methods to give you a broader perspective of an individual.

We’re going to have to find a mechanism whereby people can present aspects of their work for evaluation… maybe to different bodies such as the community as a whole, a review panel, a special interest group, or a university professor. You’ll get multiple assessments of the same body of work. The overall assessment may be some compilation of that. That’s down the line, however.

Moodlerooms
1101 East 33rd St., Suite A306
Baltimore, MD 21218