Study: Instructor Quality 'Clearly' an Important Factor in Student Success

• By Dian Schaffhauser
• 01/17/17

A project led by three researchers at the University of Michigan has attempted to tease out the impact of individual instructor effectiveness in colleges and universities. The results of the study were recently published by the National Bureau of Economic Research.

To standardize the setting for the study, the researchers turned to data supplied by the University of Phoenix, considered the largest university in the world. U Phoenix, a for-profit institution, offers both online and in-person courses. In this project, the focus was on instructors in a college algebra course (MTH/208, covering College Mathematics I and II,) delivered in both formats and required of all students in bachelor of arts degree programs. Most of the faculty are non-tenured and part-time.

The courses themselves are highly structured. The classes typically last five weeks and students take just a single course at a time. Instructors use "centrally prepared" curriculum and assessments, and grading is done automatically through course software. There is one specific difference between the two formats. Face-to-face faculty can introduce their own learning tools, give extra exams and homework or add other elements that aren't part of the structured offering. Most of the online teachers use the course materials and software "as given." However, in both formats instructors can weight course components as they choose for the final grade.

About 60 math sections are started every week, and instructors are allocated to them randomly. Students only find out which instructor they'll have a few days before the course begins, at which point they're already enrolled. For the brick-and-mortar campuses, the assignment is less random and most are too small to run different sections concurrently.

Typically, instructors are hired through a main committee, which checks that the candidate has the right degree. Qualified candidates attend a five-week standardized training course, which includes delivering a mock lecture for in-person teachers and a mock online session for online instructors. Salaries are "relatively fixed" but do vary a bit based on degree and tenure.

To track teacher performance, U Phoenix requires its instructors to take a yearly refresher course on teaching methods. Annually an evaluator will sit in on the course to monitor the quality of the instructor. Also, an in-house data analytics team tracks key performance indicators, such as average response time to questions asked through the online platform, student complaints or systematic issuing of grades that are too high or too low. If the evaluation uncovers a negative deviation from university standards, the instructor is put on notice. If performance doesn't improve, the university doesn't rehire the instructor.

The study analyzed information on 2,343 instructors who had taught 34,725 sections of MTH/208, covering a total of 396,038 student observations, between July 2000 and July 2014. Student performance on the final exam (or some variation) was used as the outcome measure.

What did the research discover? The better the instructor, the better the grades, especially in in-person courses. As the report stated, "A one-standard-deviation (SD) in instructor quality is associated with a 0.20 SD increase in course grades and a 0.41 SD increase in final exam scores in the follow-on course, as well as a 0.13 SD increase in the number of credits earned within six months." That's comparable to the performance gap between students entering the class with a 3.0 vs. a 2.0 GPA and larger than the gap between 25- and 35-year-olds. "Instructors are clearly a quite important factor in student success," they noted.

Effectiveness grows "modestly" as the instructor gains course-specific teaching experience, but it's not reflected in pay. Nor is effectiveness as reflected in student assessments. In fact, suggested the report, as more institutions, both public and private, turn to "contingent faculty," they should consider updating how they manage instructor evaluation and compensation.

As the researchers concluded, tuning of personnel policies and practices "for recruiting, developing, motivating and retaining effective postsecondary instructors may be a key, yet underdeveloped, tool for improving institutional productivity."

A working draft of the research is available through the NBER website here.