MERLOT: Peer Review of Instructional Technology
Faculty may have enough problems producing high-quality instructional materials without the challenge of evaluating the quality of the content, potential efficacy in teaching and learning, and usability of available digital materials. MERLOT (Multimedia Educational Resource for Learning and Online Teaching) is a project that addresses these difficulties through a cooperatively developed, free, Web-based resource where faculty can easily find digital learning materials with evaluations and guidance for use. The project engages faculty and their discipline communities in the scholarship of teaching. A strategic priority of MERLOT is the peer review of online learning materials, a process that will help ensure that learning materials within the MERLOT “collection” address significant theoretical or research issues and are contextually accurate, pedagogically sound, and technically easy to use.
Perhaps most importantly, MERLOT provides a mechanism for professional recognition for the faculty developers. “We finally have a way of recognizing faculty efforts and contributions other than by counting journal articles,” says Cathy Swift, Professor, Marketing and Director of Center for Global Business, Georgia Southern University, and MERLOT Business Discipline Team Co-Leader.
Establishing Discipline Communities of Reviewers
The MERLOT project has established peer review teams of faculty who have demonstrated:
- Expertise in their discipline
- Excellence in teaching
- Experience in using technology in teaching and learning
- Connections with their discipline’s professional organizations.
The faculty were selected by the 23 systems and institutions of higher education that are partners in the MERLOT project. MERLOT’s strategy is to use institutions of higher education in establishing discipline communities of reviewers and then transitioning the responsibility for peer review to the professional discipline organizations.
MERLOT provides facilitation and training in planning and conducting the evaluation of instructional technology to the discipline-specific peer review teams. Discipline team leaders coordinate the workload among the faculty as the discipline teams perform the reviews, add materials to the collection, and design the collection’s categorization scheme. The peer review process typically involves each learning material being evaluated by three faculty members from the team who have relevant expertise. Faculty reviewers write individual evaluations and then send the reports to the discipline leader, who integrates them. The resulting peer review report contains a description of the learning goals, targeted student population(s), prerequisite knowledge/skills, type of learning material, summary of procedures for using the software and technical requirements, an evaluation and observations on the quality of the content, potential effectiveness as a teaching-learning tool, and ease of use of the materials. Comments and recommendations only for the author are also provided. MERLOT provides password-protected Web sites for posting documents, listservs, and threaded discussions, enabling the discipline teams to communicate and coordinate their work.
Evaluation Standards
The faculty on the discipline teams are provided with three evaluation standards for online learning materials by MERLOT. Opportunities for applying the standards are provided at the MERLOT workshop where the faculty teams develop discipline-specific implementation of these standards.
Quality of Content: There are two general elements to quality of content. First, do the learning materials present valid (correct) concepts, models, and skills? To evaluate the validity of the content, the reviewers rely on their expertise. Second, do the learning materials present educationally significant concepts, models, and skills for the discipline? To evaluate the educational significance of the content, reviewers decide if the content is core curriculum within the discipline, difficult to teach and learn, and/or a prerequisite for understanding more advanced material in the discipline.
Potential Effectiveness for Teaching and Learning: Determining actual effectiveness requires actual use of the digital learning materials by students and faculty and the systematic assessment of outcomes. Evaluating potential effectiveness asks the reviewers to judge, based on their expertise as teachers, if the digital materials are likely to improve teaching and learning, given the ways faculty and students could use them. The reviewers are provided an established set of principles of effective learner-centered education developed by the American Psychology Association along with MERLOT guidelines and any discipline-specific guidelines. This evaluation dimension is conditional on how the digital learning materials are used.
Ease of Use: The basic question underlying the ease-of-use standard is, “How easy is it for teachers and students to use the digital learning materials for the first time?” A summary of usability standards are provided as a guideline.
Complementing Peer Reviews with User Comments
The peer review process is designed to inherit and improve on the validity and reliability of the evaluation processes present in the peer review of scholarship. MERLOT has a second and parallel review process that will complement the formal peer reviews. Anyone who signs on as an individual member of MERLOT can contribute User Comments. The user-centered review process has precedence in a number of highly used Web sites, such as Amazon.com, and will allow individuals to provide observations and evaluations on the learning materials within MERLOT. A User Comments page is connected to every item of learning material with guidelines for feedback.
During 2000-2001, twelve discipline teams will be implementing the peer review process for music, math, biology, business, psychology, physics, foreign languages, history, information technology, teacher education, chemistry, and health science/nursing. To learn more about MERLOT, click on “About MERLOT” at www.merlot.org.
Syllabus Case Study: Putting the Mystery Back Into Language Teaching
Students might think their required foreign language classes are “murder,” but on one Southern California campus the term takes on a whole new meaning. Imagine a class in which the content is one part Murder on the Orient Express and one part the board game “Clue,” all conducted in Spanish or French. That’s pretty close to what Walter Oliver and Terri Nelson of California State University San Bernardino have created for use in their intermediate language courses. The project, an interactive murder mystery emphasizing critical thinking, task-based learning, and collaboration, was a finalist in the Paul Allen Foundation contest for best online university course of 1998.
The Misterio en Toluca and Un Meurtre à Cinet mysteries follow a similar format. All material is in the language of study, and students are expected to communicate solely in that language, not in English. Using a combination of e-mail, listservs, and a Web site, students work through a series of clues and name the murderer. Although Oliver and Nelson deliberately designed the program so that there is no one right answer (anyone could have done it), there is plenty of evidence available for students to use in defending their choice for the culprit. On the course Web site are police reports, wills, train schedules, newspaper articles, birth certificates, census information, autopsy reports, letters, and suspicious prescriptions. Some of the material is authentic; in some cases, the authors have fabricated facsimiles of actual documents. Students are required to tell the truth and answer any questions put to them.
Each student in the group is given an identity, which can be embellished as the semester proceeds. Seven characters whose personalities are richly described have critical roles; one of these is most likely to be the perpetrator. For instance, there is Jacques LaCroix, who is married to Charlotte, but having a clandestine affair with the victim Virginie (who, incidentally, just announced to him that she is pregnant). There’s also the mysterious Bernard Bovy, who may or may not be related to Virginie but stands to gain financially from her untimely death. There are also seven peripheral characters, primarily relatives or friends of the others. Up to 14 students can participate at one time.
The mysteries are set in Toluca, Mexico, and Cinet, France. An illustrated town map serves as the home page for the mystery, linking students to clues, background information, and the Internet. Students participate in four rounds of play, collecting and sharing clues as they go along. The clues point in a number of directions, implicating at one time or another each of the seven unsavory characters in the core group. The idea is to get students interested, says Oliver, who has been teaching Spanish for over 30 years. He and Nelson (who teaches French and originated the idea) wanted to “create a meaningful context for students so that they could create useful language for communicating with each other.” Typically in foreign language classes, says Oliver, students create inauthentic dialogues with each other when they are required to practice conversation. The students are bored and don’t feel invested in speaking the language. “They don’t give answers to questions that are real or very interesting. Giving them a context for speaking draws them in and encourages them to use real language.”
The class meets regularly for conversation practice, but all other instruction, including grammar and vocabulary study, happens outside of the classroom, online. Using the Web has a number of advantages. The Web site acts as a distribution site for evidence of the crime, some of which students are told to keep secret, and some of which they are required to reveal to the entire class. Students use the town map to gather evidence or link to further information, for instance jumping from the site’s mayor’s office to real government sites. The Web site also is a source for cultural information about Mexico or France. Students learn what daily life is like in those countries as they access the same kinds of material a local person might need and travel (virtually) to many historical and culturally rich locales in the area. (Toluca is an actual town in Mexico. Cinet is not, although it is modeled on a typical French town.) They learn, for instance, that in France, train schedules are written for one-way travel only, so they have to remember to pick up a schedule for the return trip.
Oliver and Nelson recommend that instructors planning to teach the course give weaker students the identities of the central, richly-described characters. “The weaker students need more support and tend to become more confident when they start with more material. The better students can extemporize more easily and don’t need as much background to get started. They tend to be very creative with their characters,” says Oliver. Starting with only a limited amount of information on each character, students add details and personality, making the mystery fresh every time it is played.
Because students are writing “real” letters and documents instead of artificial exercises, their foreign language writing is getting better. Oliver and Nelson also find that there is a great deal of peer mentoring through the listserv. “Students teach each other. If one sends a message that is full of grammatical errors, another student might reply by repeating the question in his or her answer but using a corrected version, to let the student know how to write it correctly. It’s a subtle and kind way of helping each other,” says Oliver.
At the end of the course, students are required to submit multimedia presentations in which they make a case for their choice of murderer. The presentations have been quite clever, according to Oliver, including newscasts, legal briefs, and even a videotaped confession. One group of students created a town newspaper, which not only offered news, but also incorporated ads for many of the products, such as perfume, that appear in the mystery documents. Although students are a bit disappointed to discover that there is no “right” solution to the crime, they enjoy trying to convince the others that they’ve made the right choice.
Oliver and Nelson have taught more than 20 sections of the course over the past few years. Oliver, who now directs the California State University Languages Initiative to form consortia for teaching foreign languages throughout the state system, says students have given it rave reviews. He notes that they devote more time to the murder mystery than they have in previous traditional versions of the course. Says Oliver, “Students complain to me that they spend too much time on the course, but then they admit it’s because they get so caught up in the story, not because it takes longer to do the assignments.” The material can be completed within a quarter or semester with intermediate students. Misterio en Toluca and Un Meurtre à Cinet are now copyrighted products distributed by Heinle and Heinle. For more information, see the Web sites http://quixote.csusb.edu/misterio/intro.htm and http://voila.heinle.com/cinet/title.html.
