2016 Innovators Awards

Bringing Peer Review Tech to the Classroom

The My Reviewers peer review software platform provides document management tools, resources and workflows for students, instructors and administrators in college and university writing programs.

• By David Raths
• 07/13/16

Category: Teaching and Learning

Institution: University of South Florida

Project: My Reviewers

Project lead: Joe Moxley, professor of English

Tech lineup: Developed in-house

Peer review is often a key part of writing programs in higher ed, but many learning management systems lack the features to facilitate the reviewing process. So when Val Ross, director of the Critical Writing Program at the University of Pennsylvania, searched for a software tool to manage peer review for faculty and students, she was pleased to find My Reviewers, a system created by Joe Moxley, an English professor at the University of South Florida.

"The software he developed is far more sophisticated in the types of metrics and in handling multiple reviews, instead of just a one-to-one setup you often find," Ross said. "It is a great pedagogical tool for helping writers understand audiences and for developing the interpersonal and cognitive skills of interacting with people about their work."

Moxley, professor of English and director of composition at USF, has been working on the concept of peer review software for writing programs for more than a decade. USF began piloting My Reviewers in the English department's freshman composition courses during the 2009-2010 academic year. Since the peer review features came online in 2012, students at USF have conducted more than 88,811 peer reviews. Beginning in the fall of 2014, My Reviewers also was adopted by the university's Professional and Technical Writing Program.

Moxley said there are several problems inherent in college writing programs that My Reviewers can help alleviate. He noted that on most campuses, students in first-year writing courses are asked to conduct peer groups in class — but there's no easy way for instructors to sort through the peer reviews to distinguish unhelpful reviews from strong ones. Many students are unsure of how to conduct peer reviews, lacking models of effective comments or a system that holds them accountable. My Reviewers provides a way to optimize timely feedback from instructors and peers, Moxley explained. It enables instructors to more effectively administer peer review and assess the quality of peer-to-peer feedback in a digital environment. And real-time analytics allow instructors and administrators to make evidence-based curriculum changes, identify at-risk students, and research instructor efficacy.

How it Works

In a nutshell, here is how My Reviewers works: Students upload texts to cloud storage; then, using a suite of document management tools (sticky notes, text boxes, highlights, hyperlinks), teachers provide feedback by inserting comments, providing endnotes, and assessing texts via rubrics that calculate scores through user-defined weights. Students perform peer reviews using the same tools. Comments and rubric scores are aggregated for assessment purposes; learning analytics track student progress by rubric criteria and project performance as well as measure curriculum effectiveness. Administrators mentor teachers using real-time evidence. The system has a permission-based architecture and can be configured so that access to student documents is viewable by a single instructor, multiple instructors or by mentors and instructors.

Moxley said that just having all the student writing and comments in digital form will help researchers better understand the writing process, collaboration and revision. "Historically there has been a lot of research regarding the helpfulness of both teacher and student commenting. One thing that digital tools afford is a much simpler way to analyze that," he said. Also, in a digital space it is easy to aggregate all the comments you received over the period of a semester, both from peers and instructors, which can help students see patterns in their own writing.

In the first-year composition courses at USF, students write three papers, and for each of those papers, they write three drafts. The instructors and peers give feedback on those drafts, and the software aggregates those comments. The revision plan workflow pulls peer and instructor reviews and offers research-based, heuristic questions so that students engage with query-based feedback instead of directive feedback. A "Community Comments" feature provides a handbook regarding style, mechanics and grammar.

Students are graded on their review comments as well as their own papers. "In our first-year composition program, we attribute 30 percent of the grade to the peer review," Moxley said.

USF charges students a fee for the use of the software, but Moxley points out that it is a good deal because it also includes e-books. "When they were charged for books alone it was $150," he said. For the use of My Reviewers and the e-books, the charge is $47.

Digitizing the writing review process also is of interest to administrators focused on student success, Moxley said. "Everything is recorded by instructors," he noted. "We can use that information to track student success over time. If we have students who haven't turned in work or are doing poorly, people involved in mentorship can reach out and find out what is going on. We give reports to the office of institutional effectiveness, and they use them each year for accreditation reporting."

Beyond the English Department

Moxley and collaborators at several other universities, including Val Ross at Penn, have received $677,811 in funding from the National Science Foundation to study the use of My Reviewers in STEM courses.

For example, science faculty at the University of Pennsylvania are exploring how the technology can help them integrate more writing assignments into their courses. Doing so has been a challenge, Ross said, because busy faculty don't have much time available for grading papers. "We thought that with training on this peer review software, the students could actually be responding to each others' work in a way that advances their knowledge of peer reviewing, which they will be doing forever in some fashion," she explained, "while developing social skills involved in peer reviewing and building their subject-matter knowledge, because part of real peer review is to critique each other's technical claims and problem-solving."

She said peer review could help some students who are struggling with the sciences, because it immerses them in social learning as opposed to solo problem-solving. "So those who learn through social interaction and are more satisfied in a social community are likelier to be retained," Ross said. "That is our hope and one of the questions we are studying."

Marketing the Software

Although USF owns the intellectual property for My Reviewers, Moxley has formed a limited liability company and licensed the technology in order to make sales to other universities. His startup received a follow-on grant from an NSF program called I-Corps to help entrepreneurs commercialize technology funded by NSF.

So far, a dozen schools have tried the software. "I invited some schools prematurely, I think, because we weren't quite ready," Moxley admitted. "It took us a long time to develop a bug-free toolset."

Moxley said some other universities have created peer review software, but those programs are not as full-featured. "The way we aggregate peer review data with the analytic reports and portfolio process, no one else has that," he said.  "We also place an emphasis on collaborative writing. The modern workplace requires collaboration skills. My Reviewers permits students to write as a group and evaluate other teams' portfolios."

One of the business challenges, Moxley said, is that people have a tendency to expect digital tools to be free. "We have spent over $1 million on development," he said, starting with $250,000 in startup funds from USF and then the fees charged to students using it at USF. "This involved a lot of investment," he said.

Penn's Ross believes My Reviewers has great potential as the field of writing studies moves toward being more of a social science. "Here you have an ideal marriage of pedagogy and metrics," she said. "I think for us it is not only the peer reviewing, but also how to train the faculty and new instructors in assessment work. When you are looking at inter-rater reliability and consistency of evaluation, the tool is tremendous for that."

Return to Campus Technology Innovators Home