Assessing Assessment
- By William H. Graves
- 12/30/04
Technology can play a role in assessing learning— and more than a labor-saving
role at that.
D'es technology play a role in the assessment of student learning? The answer
at first blush is “yes.” After all, many instructors use the testing
tools available in course management systems or more specialized assessment
software to develop and save test banks and then to automate the generation,
administration, and grading of quizzes and exams from those test banks. Such
tools can certainly be labor-saving for instructors, but some political observation
about the assessment of student learning will lead to a potentially deeper role
for assessment technologies in encouraging and evaluating student learning.
Assessment: The Political Angle
Oversight and policy-making bodies from the institutional to the federal level
believe, reasonably, that learning outcomes should be a key indicator of institutional
performance in higher education. Accordingly, they expect nonprofit higher education
to assess and report student learning. In fact, some policy makers are threatening
to regulate funding subsidies or tuition levels, in an effort to encourage accountability
in assessing and reporting learning outcomes (and costs) for informational or
comparative purposes. Frankly, this accountability expectation may be more reasonable
than most higher education leaders will concede. But avoiding its intrusive
implications may require a proactive response from higher education. Never fear;
some proactive responses are in the making.
Getting Proactive
In its 2004 edition of Measuring Up, the National Center for Public
Policy and Higher Education (www.highereducation.org),
for the first time, gives a “Plus” in learning to five states: Illinois,
Kentucky, Nevada, Oklahoma, and South Carolina. These states have developed
comparable learning measures by participating in a national demonstration project
conducted by the National Forum on College-Level Learning, with funding from
The Pew Charitable Trusts (curry.edschool.virginia.edu/centers/collegelevellearning).
In contrast, the 2000 and 2002 editions of Measuring Up gave all 50 states an
“Incomplete” in learning because there were no comparable data that
would allow for meaningful state-by-state comparisons in this category. At the
heart of this issue is a basic learning-assessment practice in the academic
enterprise.
Tenure, a keystone in nonprofit higher education’s governance model,
is broadly interpreted to grant any tenure-track instructor the academic freedom
to determine course content coverage within the scope of a course’s catalog
description, to develop and administer course examinations, and to assign course
grades. So, at many institutions, the assessment of course learning g'es no
further than the course instance and the grades assigned by its instructor.
The institutional assessment of learning stops there.
Yet many instructors are now creating an engaging mastery learning model of
study and feedback by changing their assessment strategies toward an almost
continuous pattern of quizzing, practice testing, and testing that takes advantage
of labor-saving testing software.
I called attention to this strategy in my
November column, in the context of the “common course redesign strategy”
pioneered by the National Center for Academic Transformation (NCAT) and 30 partner
institutions. They have demonstrated that it’s possible to assess a traditional
multi-section high-enrollment course versus its redesigned counterpart to a)
improve learning outcomes through diagnostic adjustments, and then to b) continue
the assessment on an institutional, longitudinal basis provided that testing
is standardized across the (redesigned) course as a whole. Clearly, though,
to move beyond some degree of internal institutional standardization (in the
interest of internal longitudinal comparative assessment and improvement strategies)
will require bolder departures from the prevailing instructor-centric model
of assessment.
Bolder Moves
The common course redesign strategy applies particularly well to the college-prep
(developmental) and college-level basic skills courses and a few of the highest-enrollment
introductory courses on any campus. These courses are taught at almost all institutions
and cover essentially the same material. A consortium of institutions—a
community college district or a state higher education system, for example—could
adopt the above longitudinal assessment strategy across the consortium by standardizing
testing in selected common courses. But an even bolder move would be to decouple
teaching from testing in selected common courses. Assessment organizations such
as ACT (www.act.org), College
Board (www.collegeboard.com),
ETS (www.ets.org), and others
offer standardized tests (and course materials) in a number of skills- and intro-level
common courses, and most of these are or soon will be available in an Internet-delivery
format, sometimes involving sophisticated assessment methodologies beyond the
true/false and multiple-choice formats. What’s more, the reductions in
direct instructional expenses resulting from common course redesign—an
average of 40 percent for the 30 NCAT partners—could more than offset
the expenses of externally provided assessment, when those assessment expenses
could not be passed along to students.
Stumbling Blocks and Solutions
The problem is, many institutions are undoubtedly hesitant to publish learning
outcome data that can be compared to the same data from other institutions—the
growing expectation of policy makers. Perhaps some recent developments in another
aspect of institutional learning assessment could inform this comparative issue:
Retention and graduation rates are the most common externally reported measures
of learning. Yet, they reflect many factors other than learning, while accounting
for learning only by assuming that retained and graduated students must have
learned something to justify their credit-recorded success. There is no comparative,
standardized basis for reporting these rates, and they are not universally provided.
But Alexander Astin, writing in the Oct. 22, 2004 issue of the
Chronicle
Review (“To Use Graduation Rates to Measure Excellence, You Have
to Do Your Homework,”) recently reported some hope for more meaningfully
reporting graduation rates. Astin made a case for reporting actual versus “expected”
graduation rates as a basis for taking into account institutional type and the
socio-educational profile of its first-year class. Possibly, this work will
find application at the level of the set of courses most closely linked to retention
and graduation—again, the college-prep and college-level basic skills
courses, along with a handful of college intro courses, which taken together
account for a high percentage of all enrollments, and could be externally assessed
as per our discussion above.
Technology to the Rescue
Yes, technology can play a role in assessing learning; more than a labor-saving
role. Today’s course management systems and assessment technologies make
it possible to:
- Deploy a continuous feedback loop of study-assessment-mastery in a variety
of courses.
- Develop a common (standardized) institutional, longitudinal assessment
strategy for high-enrollment courses taught by multiple instructors—a
strategy that assesses the course as a whole, not each of its instances taught
by different instructors.
- Participate in a consortium to extend the (preceding) whole-course, common-course
longitudinal strategy across institutions.
- Decouple teaching from testing in basic skills areas and selected intro-level
disciplinary and professional courses by using nationally “normed”
assessments (and materials) available from trusted external assessment providers.