How Do You Measure Success? <br>Lessons on Assessment and Evaluation from the LEAD Center
If you are a college instructor who is investing significant effort (and assuming
risk) in using innovative technologies in your courses, or if you are an academic
staff person who supports such college instructors, you surely want to determine
whether, and if so why, students are learning better, more, and/or differently
as a result of your efforts. If you are an instructor, you can obtain the answers
you seek by using assessment techniques. If you are someone who supports instructors,
you may be able to help these instructors by using evaluation methods.
Most people in the academic community tend to use the terms evaluation
and assessment interchangeably. Researchers at the University of Wisconsin-Madison's
Learning through Evaluation, Adaptation, and Dissemination (LEAD) Center use
these terms to describe different, though related, processes.
Evaluation refers to applied researcher efforts to help clients achieve
their goals for a project (whether a course, a program, or an organization).
LEAD researchers help clients decide what data should be obtained, gather and
analyze that data, and then report the resulting information in formats that
the clients can use to improve their project and/or demonstrate to others the
value of the project.
Assessment refers to faculty efforts to obtain information about how
and what students are learning in order to improve their teaching efforts and/or
to demonstrate to others the degree to which students have accomplished the
learning goals for a course. At the LEAD Center, both evaluation and assessment
entail the following processes:
- Articulation of goals
- Specification of the strategies intended to achieve these goals and the
reasons that these strategies are expected to be effective
- Agreement on the evidence that will convince specified individuals that
the strategies have achieved the goals
- The gathering, interpretation, and use of information.
Evaluation Processes at the LEAD Center
If you have the opportunity to work with LEAD evaluators, they will help you
through these four processes, as the faculty who teach the sophomore-level engineering
course at the University of Wisconsin-Madison (CS 310) will tell you. LEAD is
currently evaluating CS 310, a course that focuses on teaching students how
to solve problems in engineering using a variety of computer applications. For
the first time this past fall, all lectures, exercises, and course materials
for CS 310 were available online through the course's Web page and a newly developed
online video-presentation application called eTEACH. eTEACH allows students
to watch and review lectures and related materials on their own time and at
their own pace. Live lectures have been replaced by team labs, where students
engage in small-group problem solving. The results of LEAD's year-long formative
evaluation will be used to improve both the course and the eTEACH computer program/application.
The LEAD Center works only with clients who undertake larger, long-term projects
and have a strong commitment to evaluation. Once the center has determined that
the client meets requisite characteristics, LEAD researchers meet with clients
to discuss key questions such as:
- Which goals for student learning or project success d'es the client want
to evaluate?
- What will the client and the client's colleagues accept as evidence that
the goals have been achieved?
- How much emphasis shall be placed on understanding student learning processes
and the organizational and cultural factors associated with project success?
- What data-gathering methods are feasible for obtaining information about
both processes and outcomes?
- Given the limitations of the research design, funding, and timetable, what
kind of formative and summative feedback processes and products will optimize
the achievement of goals?
During the project, researchers at LEAD use a combination of qualitative and
quantitative social science methods, with strong emphasis on inductive analysis.
Data-gathering methods include surveys, open-ended structured interviews and
focus groups, observations and video recordings, and longitudinal student databases.
Assessment Resources
Due to expertise gained in their work evaluating college-level educational
improvement efforts, several researchers at LEAD have worked with the National
Institute for Science Education's College Level One (CL-1) team to produce assessment
resources that can help faculty ascertain how well their strategies to improve
student learning are working. The Field-tested Learning Assessment Guide (FLAG)
and the Student Assessment of Learning Gains (SALG) instrument are located at
the CL-1 Web site: www.wcer.wisc.edu/nise/cl1/.
The FLAG provides a collection of classroom assessment strategies that rests
on a strong foundation of empirical research and has been tested by extensive
use in the classroom. The FLAG is designed around Angelo and Cross's concept
of "Classroom Assessment Techniques" (CATs). These are self-instructional
modules that introduce techniques for assessing progress toward conceptual,
attitudinal, and performance-based course goals in science, mathematics, engineering,
and technology (SMET) disciplines.
Upon invitation by the CL-1 Team, a national group of leading SMET assessment
scholars accepted the challenge of developing CATs for their assessment specialties,
resulting in a set of twelve CATs that are field-tested and evaluated. The FLAG
also includes an introductory primer, an interactive engine that links faculty
goals with the most appropriate assessment techniques, and a searchable database
of assessment tools, which continues to be expanded.
A particularly popular CAT within the FLAG is the Student Assessment of Learning
Gains (SALG), which uses the Web to offer faculty a quick and easy way to obtain
both mid- and end-of-semester feedback from students. The SALG is accessible
to anyone with a browser and is offered as a free service.
The LT2 Web Site
LEAD researchers also have enjoyed the opportunity to develop case studies
for the College Level One team's faculty development resource on effective use
of learning technology. This resource, Learning Through Technology (LT2), is
located at www.wcer.wisc.edu/nise/cl1/.
LT2 is designed to answer questions such as, "What can I do with learning
technology that I can't do now? What are the nuts and bolts of using learning
technology? How can I use learning technology so that my students really learn?"
In particular, it seeks to serve SMET educators who believe it is important
to develop the ranks of future scientists and a technical workforce, prepare
teachers to be scientifically knowledgeable, and help all students become scientifically
literate members of our society by making appropriate—indeed, transformative—use
of the new computer-based learning technologies. The LT2 site is not designed
to serve individuals seeking resources on distance learning or on how to translate
traditional course content into electronic formats. It offers in-depth case
studies, lively first-person accounts, "hallway conversations" about
technology, and links to articles and more resources, including a taxonomy of
learning technologies.
Using the LEAD Center
The UW-Madison established the LEAD Center in fall 1994 to provide third-party
evaluation research in support of educational improvement efforts at both undergraduate
and graduate levels. The center has a client-driven and student-focused approach
to evaluation research. LEAD clients are faculty or staff at UW-Madison or institutions
that are collaborating with the UW. Furthermore, they are individuals who:
- Can provide or work with LEAD to obtain the resources—usually grants—to
pay the full cost of the evaluation research
- Have well-articulated goals for deeper and more relevant student learning
- Are developing and testing more effective strategies for achieving these
goals
- Are committed to obtaining and using feedback on student learning experiences
and outcomes to improve teaching and fine-tune goals
- Seek to understand the various factors that are necessary to more effectively
institutionalize and disseminate their efforts.
For more information about LEAD and the evaluation and assessment projects
affiliated with the center, visit www.cae.wisc.edu/~lead/.
References
T.A. Angelo & K.P. Cross, Classroom Assessment Techniques, Jossey-Bass Publishers
(1993). Hestenes, D., Wells, M., and Swackhammer, G. (1992). "Force concept
inventory." The Physics Teacher, 30, 141-158.
Assessment and
Technology in Physics
Curt Hieggelke is
a national leader in the development, use, and dissemination of innovative
computer-enhanced introductory physics teaching methods. A "tekkie" from
way back, Hieggelke has transformed his introductory physics courses at
Joliet Junior College in Illinois into meaningful and exciting learning
experiences for his students—whether they are aspiring engineers, scientists,
health professionals, or non-science majors. Key to his success are (1)
the use of computer-based labs that actively engage his students through
real-time acquisition and analysis of data, connections to real-world
events, and visualization and simulation, and (2) assessment practices
that constantly inform Hieggelke of the value of his innovations.
In the late 1980s,
Hieggelke realized that he might go beyond using computers merely for
analysis and instead use them to transform the way his students learn
physics. In particular, he was excited about the possibilities of using
electronic probes that interface with a computer; such devices would enable
his students to collect and analyze data themselves, fostering a predict-observe-explain
learning process that Hieggelke felt was essential to getting his students
to understand—not just memorize and regurgitate—important physics concepts.
After obtaining computers
for his students, Hieggelke searched for—and helped develop—a second generation
of software tools that would enable the active learning environment that
Hieggelke sought for his students. He explains, "The first generation
of computer technology was ‘do the old lab experiment, hook a computer
to it, and let the computer do graphing or fitting.'" The second generation
demands the active engagement of students by predicting, observing, and
explaining. "Students are really engaged with the experiments," he says.
"After they set them up, they can interact with them and see exactly how
things changed."
More specifically,
these software tools allow students to visualize patterns of data, use
graphical representations to avoid being lost in the data setup and collection
of details that accompany most lab activities, and experiment easily with
different parameters in the same lab setup.
|
Pretest
Score |
Posttest
Score |
Mean
Score
SD |
49%
16% |
73%
15% |
| SD
= Standard deviation; N = number of students |
| Average
JJC Hake gain |
.47 |
| Average
national Hake gain for students in traditional courses |
.23 |
| Average
national Hake gain for students in interactive courses |
.48 |
Along with his computer-enhanced
teaching strategies, Hieggelke also uses assessment activities and guided
group work activities. He blends these computer-independent activities
and his computer-dependent activities into a synergistic framework for
learning. "The hope is that [the computer work] will feed nicely
into how I am interacting with the students and how the students are interacting
with each other," he says, "even when we are not in lab."
Hieggelke places great
importance on the use of formative assessment tools as learning activities.
He believes that these activities are critical in directly fostering learning,
and in providing information about student learning that instructors need
in order to constantly adjust and improve teaching strategies.
The assessment activities
Hieggelke uses fall into two general groups: activities that he calls
"Tasks Inspired by Physics Education Research" (TIPERs), and pre-/posttests
that have been developed recently by physics faculty around the nation.
The physics education research on which the TIPERs are based has established
that it is very difficult to modify some of the typical beliefs that students
hold, and has ascertained through experimentation that certain methods
of teaching are more effective than others in getting students to make
the appropriate modifications.
Hieggelke uses the
learning tasks or formats that these education researchers have designed
to pursue their research questions as assessment activities in his course.
Details about the TIPERs can be found at http://tycphysics.org.
Hieggelke uses pre-/posttest assessment tools to check whether his teaching
activities are reinforcing each other and providing students with challenging,
engaging, and effective science learning experiences. His data show that
when tested on conceptual understanding, students perform significantly
better than their counterparts in traditional physics courses (see table).
The Force Concept
Inventory (FCI), designed by David Hestenes and colleagues, is widely
used in the physics community to assess student understanding of the basic
issues and concepts in Newtonian dynamics. Questions are multiple-choice
and are written in non-technical language, but answers are included among
attractive distractors that specifically address common misconceptions
about physics.
Has Hieggelke succeeded
in transforming his introductory physics course into meaningful and exciting
learning experiences for his students? The proof is in the outcomes. According
to Alan Van Heuvelen, a nationally recognized physics educator at Ohio
State University's Department of Physics, "When you plot Hieggelke's students'
posttest results on the Force Concept Inventory along with the results
of students taught by other faculty who use the interactive engagement
approach to physics, his students' outcomes compare favorably with those
of students taught by Eric Mazur at Harvard University."
For more information
see the LT2 Web site, www.wcer.wisc.edu/nise/cl1/.For
examples of some of the software that Hieggelke uses in his classes, see
http://vernier.com/cmat/tst.html,
http://vernier.com/cmat/rtp.html,
and www.wiley.com/college/sokoloff-physics/.
