Assessing Student Work in the Open Educational Resources Era
Direct interaction between students and teachers, as in “face-to-face,” or “on the ground,” or “in real life” (whatever we call people being in physical proximity to each other these days, also sometimes called--with an embarrassed downward look--“seat time”), has always been considered prime time. It was and is the time when teachers get to know their students most fully and when the teacher’s considerable influence is at its utmost. No multimedia resource can trump this human interaction, but technology-enhanced instruction has entered the learning equation, particularly to support learning outside the traditional classroom walls. And other work outside the classroom, such as field work or other forms of experiential learning--often supported by technology--is part of today’s learning environment as well. So, the big question as we try to understand what students learn becomes: How do we know what broad mix of learning experiences away from the teacher is best for learners (from among all the choices today)?
Multimedia and More
The availability and quality of open educational resources (OER) now--see “An Open Mind” April 8, 2010 at the New York Times, for example--reminds us that learners have many more options for important learning opportunities than ever before. Many of these resources are produced within academia, so they are guided and informed learning experiences created with enthusiasm. And this is mostly because of the vivid and immediate way resources can be created now with minimal help from media experts; every laptop can be a production studio and a post-production editing suite. The claims over previous decades that multimedia improves learning seemed hollow for much of that time--when many argued that all teaching should be enhanced with multimedia. But now, they are no longer so hollow, and the claims are generally less sweeping. Those among us who have believed that online learning is often the lowest of the low-impact learning experiences may begin to change their minds once they explore the prepared materials now available online. OERs should be considered an essential part of the learning mix. As examples, look at MIT Visualizing Cultures, or at a page from the MERLOT Mathematics Portal that presents calculus problems and flash tutorials to help solve them.
High-impact learning experiences, such as problem-based learning, experiential learning, service learning, internships, work-study, study abroad, co-op learning, field study, and others, are also part of the current broad mix. These experiences are a growing proportion of a student’s mix of learning experiences, and many are facilitated by technology. In addition, many colleges and universities use online learning communities to help develop what has been proven to be true, that studying in groups of students increases, sometimes dramatically, the pace and quality of learning. In all cases, active, participatory learning seems to add much more learning impact than classroom attendance only.
Integrating a Broad Mix of Learning Experiences
But how do we assess or even understand what students learn from this broad mix of learning experiences? And how do we integrate it all?
Scenario: Let’s imagine that at a particular institution a student’s learning experience in academic subjects is made up of 25 percent in-class interaction, 35 percent individual study, 20 percent study group, and 20 percent high-impact learning activities. Do all of these experiences lead toward a synthesized understanding of a domain, or do they remain unconnected? How can a teacher know enough about the 75 percent of learning occurring outside her experience to assess that learning? Do the teacher’s assessment and evaluation instruments capture that 75 percent?
The reason for these questions is that we have moved beyond the need or even the desirability to stay with the simple lecture-reading assignment dichotomy. With so many more ways for students to learn, that simple dichotomy seems less and less appropriate. It is not that good students won’t learn in that dichotomy, but that by limiting the class to it, we are missing ways to increase inspiration and imagination about how the learning is connected to the world.
Expanding on the 75 percent question, the time students are away from the teacher in our scenario, are the various modules (20 percent, 25 percent, etc.) of student learning experiences in my imagined institution disconnected and unrelated? Are they reinforcing each other? Is the teacher aware of these experiences, and, if so, does she have any way of gaining insight into the nature and quality and results of these experiences?
Continuing the scenario with another question, if 75 percent of learning is occurring away from the teacher, shouldn’t the teacher use her 25 percent of the time helping students make the most of their 75 percent? In other words, instead of lecturing, why not help students learn how to plan their out-of-class experiences in such ways as helping them identify questions to pursue, helping the collaborative groups with their projects, guiding students in assessing the evidence to solve problems they’ve been assigned, and so on? The learning opportunities and digital tools are there to support this more distributed and varied learning scenario, so it makes sense for teachers gradually to adjust and fit into this scenario.
Evaluation: Moving from Standardized Tests to Student Commentary
Ultimately, however, the teacher has to evaluate this broad mix of learning experiences, which may be more likely to be different for each student. One could argue, of course, that no matter if the learning design is exactly the same for all students such as in the traditional classroom, each of them still had or has a different experience because of their varying abilities and backgrounds. But, still, it is, admittedly, unusual to have a class where students follow different paths by design. How do you assess work that varies in kind and quality based on the mythical institution I’ve described?
We’ve often solved this dilemma by giving a standardized test so we can norm students and provide what we believe is a grade that roughly characterizes the achievement of each student. Now, with learning experiences so much more varied and accessible--essentially with living textbooks all over the landscape--and with the ability of students to comment while they are engaged in those living textbook experiences, the idea of a standardized test seems more and more removed from reality.
We have evaluated after-the-fact for so long, we teachers may not be aware that we can evaluate while learning during a particular assignment is still in process. One institution, Brigham Young University in Idaho, requires students to submit a question to the teacher from the assignment before class so that those questions can come up during class. The whole class is based on questions students have sent.
Students commenting on their work (during the work-in-progress, just as the work is handed in, and then commenting on their own comments later) is in fact a new way to standardize evaluation. With some norming of how teachers evaluate student comments, grading can arise from that evaluation process. The work is varied, but student comments on the work is a standardized mental process.
Student comments are most often called “reflections.” In a portfolio, the student can be working on reporting the results of field work, get to a sticking point, and use an informal reflection in writing to help her think through the problem. The informal reflection--like a think-aloud protocol--is then associated with the work. Since the reflections provide a window into the thinking process of students while they work on problems, the reflection is an extraordinarily valuable data point to use for assessment. This process is being used already at a number of colleges and universities and some of them claim the reflections, or what I was calling “comments,” are a more valid and authentic basis for assessment than the collection of work in the portfolio.
Creating a standardized test to try to encompass work that varies--our learning ecology now encourages a much greater variety--is unrealistic. But requiring student comments at specified points in the whole process of an assignment, including culminating comments, provides richer data to evaluate. In this case, the teacher is evaluating student’s ability to comment on the meaning of the work, a step toward meta-cognition.
Comments made by the students about the work they do is a natural outcome of their efforts doing the work. Evaluating those comments using a process of inter-rater reliability to create a normed evaluation set of criteria is more appropriate to this age than the testing methods we have used.
Opportunities for learning in various modalities are everywhere. Such a multitude and such a bounty opens the door for students to find many more ways to discover and experiment and do something active. But the bounty also poses a challenge for assessment. Student reflection on their work offers one way to meet this challenge.
[Photo by Trent Batson]