Better Off With or Without Your CMS? 5 Kinds of Assessment That Can Really Help

• By Stephen C. Ehrman, Steven W. Gilbert
• 06/26/03

Course management systems are expensive and changing rapidly. And while their educational benefits are rarely obvious, their costs are quite apparent. Do you have any way of knowing how much educational value your college or university currently derives from its use of a CMS? Do you have a strategy for increasing its value in a course or across the curriculum?

Perhaps it would help if you could see more of what is actually happening when the CMS is used. There are five questions we've found to be helpful in establishing the value and improving the effectiveness of a CMS, whether its Angel, Blackboard, CHEF, or another letter of the alphabet. Each question leads to a different study design, and all five produce data that could help faculty make more effective use of the CMS while guiding support staff who work with them.

Question #1: Watching Ourselves in the Mirror
How have the ways faculty used our CMS improved the effectiveness of teaching and learning at our institution?

One way to answer this question: Hold a mirror to the academic program and let all faculty members see just how much (or how little) impact CMS use has on program quality. Provide some facts and encourage debate about what they mean and what to do next to improve learning.

Perhaps some faculty doubt whether a CMS has any value at all. In a recent study, Cheryl Bielema and Robert Keel (2003) at the University of Missouri-St. Louis (UMSL) compared two groups of courses that both involved some face-to-face classes on campus: One group made extensive use of the CMS, while the second made less. Bielema and Keel surveyed students in both sets of courses, asking them how often they engaged in certain instructionally important activities.

Students in classes with high use of the CMS were more likely to say they were involved in the kinds of teaching and learning practices endorsed by Chickering and Gamson's "Seven Principles of Good Practice in Undergraduate Education." For example, they were statistically more likely than students in the courses with little CMS use to report that they received instructor comments on assignments quickly, communicated with their instructor, completed assignments on time, and in general were more engaged in the course.

What were the consequences of these practices? The students in the CMS-intensive courses were also statistically more likely to predict that they each would take another course at UMSL and complete their degree there. Bielema and Keel concluded, in part, "responses from these students indicated that faculty be encouraged to continue integrating [CMS] features and that future syllabi include expectations for student work online."

D'es the Bielema-Keel study indicate that faculty at your institution who make use of the course management system are also promoting better teaching and learning practices and helping students toward graduation? Not necessarily! That's what appears to be happening at UMSL but it may—or may not—be happening at your institution. There is no substitute for doing your own version of this study to find out how your faculty and students are using the CMS.

Question #2: Focusing the Lens
What specific changes in support strategy could improve the value of a CMS?

The first question fosters improvement by focusing attention on the kinds of teaching and learning that typically produce better outcomes. It also answers the question, "How well are we doing so far?" In contrast, the second question produces more direct suggestions for how to improve those outcomes.

This study design is based on several assumptions. First, that faculty choices about how to use the CMS (e.g., for interactive online discussions) have a great influence on whether, and how, the CMS contributes to program quality. Second, that if faculty do not value a particular teaching approach (e.g., seminar-style discussion) they will not typically choose to use a CMS to do that kind of teaching online. And third, if faculty initially find it difficult or impossible to use the CMS effectively for a particular educational purpose, they usually won't keep on trying to use the CMS for that purpose.

The initial step in improving your support strategy is to list some specific teaching-learning activities where the CMS could potentially be of great value. These might include accessibility (distance learning); self-paced materials; libraries of student work annotated over time; structured seminars involving students and people outside the institution; online quizzes; and intensive, focused, moderated seminar discussion to help students learn a new way of thinking about the course content

Let's focus on online, small-group, asynchronous discussions. To use data to guide institutional efforts to increase the use of the CMS to support such discussion, it makes sense to begin by asking faculty how much priority they give to seminar-style discussion in specific courses. Suppose survey findings show that many faculty place relatively low priority on seminars, especially if the courses in question have more than 20 students. As a result of this finding, the teaching-learning center or the CMS support staff might decide to avoid wasting time trying to encourage faculty to use the CMS for this purpose, or try to increase faculty use by distributing information about how the effectiveness of large courses can be improved by supporting (online) discussion among students.

Of course, it's highly likely that such a survey would also show that many other faculty do believe that small group discussion is important and appropriate for many of the courses they teach. For these faculty, the logical next question is whether they have already tried using the CMS to support, enrich, or extend such discussions in these courses. If their answer is no, then one way to improve program quality is to publicize this potential use of the CMS.

The next question in this sequence would ask about the experiences of faculty who've tried using the CMS to support online seminar-style discussion. Faculty who report especially successful experiences should be contacted to see if they would be willing to share their experiences with colleagues. Those whose experiences were frustrating should be asked more questions to see if the reasons for the problem can be discovered.

This is a win-win study design. Every possible pattern of faculty response can help guide institutional investment in the CMS and its support. The result: better support for faculty in doing the kinds of teaching they value the most, plus greater return on the investment in the CMS.

Question #3: Taking the Long View
How are our uses of the CMS helping to increase enrollment or retention?

Many institutions use a CMS in part to try to increase enrollment or to help students take more courses each term. But without data, there is sometimes needless debate about whether a program is working for these purposes. At Southeast Missouri State University, for example, the administration had questioned whether the distance learning program was really helping the university serve students at a distance, or whether it only helped students who lived on or near campus.

Starrett and Rodgers (2003) discovered, however, that more than half the students in the program were from outside the county and less likely to be served by the university without these courses. Their findings helped reinforce the case that faculty development aimed at improving online instruction should be continued as a way of enabling the institution to serve students across its service area.

Some studies also provide guidance about the kinds of teaching that can help improve retention. The Bielema-Keel study quoted earlier included the finding that, in courses using the CMS to improve teaching, students were more likely to predict that they would take additional courses and finish a degree at the university. At your institution, studies could also track students to see what they actually do: Is retention improved when students take courses where the CMS is used to support the seven principles of good practice?

Data from studies can also be used to suggest policies to improve retention. For example, a study by Henke and Russum (2001) of IBM staff members receiving online training first showed that work schedules were responsible for some attrition. If students who were forced to drop out were given a second chance to resume study at the same place—but on a different schedule—they might rejoin the course and finish it successfully. IBM did implement the new policy and follow-on studies showed this solution was indeed increasing student retention and success.

Question #4: Controlling Costs
How can we use our course management system to help control costs?

A CMS costs money and time, but it can sometimes be used to support change in academic programs that in turn save money or reduce stress on faculty and staff. Evaluation can play two roles: It can discover opportunities for such savings and it can provide evidence of whether savings have been achieved.

For example, Pope and Anderson (2003) report on how the University of Pennsylvania analyzed the costs of its departmental laboratories for undergraduate engineering students. Their study of the operating costs of those old labs provided guidance for reorganizing lab instruction. The steps included using a CMS to provide equipment training, safety training, and administrative support. This use of the CMS reduced time demands on faculty and graduate students, while also freeing them to spend more time on coaching students on engineering ideas.

Many studies provide evidence that specific, CMS-enabled program changes do reduce costs. Henderson, Brown, and Meyers (2003), for example, found that the same investments in course design that had aided course quality also helped cut the costs of developing and teaching those courses. One of the largest collections of such evaluated projects was supported by the Center for Academic Transformation ( www.center.rpi.edu). These projects, which focused on reducing operating costs and which often used CMSes, usually were able to produce substantial savings in per-student costs. However, the analyses did not include costs of operating the CMS and other institution-wide operating costs.

Question #5: Lowering Barriers
How can faculty identify and lower barriers to desired CMS use by students?

Suppose a faculty member wants students to collaborate online. Faculty members we have talked with over the past few years have identified more than 40 different reasons why at least one student in a course might not work effectively with other students online. Some of the barriers relate to technology (e.g., Internet connectivity problems, etc.), but most of the barriers relate to other attitudes or other non-technical factors in the course (e.g., student d'esn't believe collaboration will help that student learn; directions for the assignment weren't clear to the student, etc.).

The good news is that many of these barriers are easy to remove if the faculty or institution knows that the barrier is hindering a student's work. One way out of this problem is for faculty to survey students about factors that may be making it difficult or impossible for each of them to do group work online.

The findings from these five kinds of studies can guide efforts to increase the benefits of a CMS. Help your institution learn more about the CMS it's already using—with the results, you can guide future adjustments and improve your CMS.