Research

Study Finds Large-Scale 'Nudging' Misses the Target

• By Dian Schaffhauser
• 09/09/19

Nudging strategies may not scale well. That's the finding of a new working paper recently published by the National Bureau of Economics on the practice of reminding students about pending deadlines regarding financial aid and college enrollment.

In two randomized controlled trials reaching 800,000 students, researchers examined messaging, delivery, timing and even access to one-on-one advising to find out how those various "nudging" mechanisms influenced outcomes. Their conclusion: There was no impact.

The study was undertaken by a team of researchers from the universities of Virginia and Pennsylvania and Brigham Young University, Brandeis and Penn State. The project was intended to better understand how "global" nudging efforts could increase completion rates for the Free Application for Federal Student Aid (FAFSA). These included one-way and two-way communications, sending information or connecting students to resources to help them navigate college and financial aid processes and, in some cases, offering "light touch" counseling to students as they made college application, enrollment and financing decisions.

The positive benefits of nudging have primarily shown up in what have been "small-scale," localized studies, according to the paper. The number of students involved in those studies was a median of 6,233. Many of those smaller-scale studies used messages "sent to students at critical financial aid junctures," the paper explained, which could "generate substantial improvements in college enrollment" for low cost. The hope was that nudging would also show promise even as the number of students scaled.

The two trials involved working with two organizations, one nationwide and one statewide, to send information about FAFSA filing to high school and college students.

The first intervention targeted all lower-income and first-generation high school seniors who had registered with the Common Application. This is a national website through which students can use a single application to apply to as many as 20 of 800-plus participating colleges. Students in the treatment group received messages encouraging them to complete the FAFSA early to maximize the financial aid they received. Those messages were varied in multiple ways, through content, delivery channel (mail, e-mail or text), the offer of advising and encouragement to share information socially with friends.

The second intervention focused on students who had applied to college through a state-sponsored portal that allows applications to all the state's public four-year colleges, as well as to some private institutions and community colleges. Students in the treatment group received text messages, notifying them of two important changes in federal financial aid policy: the ability to file the FAFSA starting in October (rather than January) of the year prior to college enrollment and the ability to use income tax returns their families had already filed in the financial aid application process. These messages had variances in timing and the use of infographics to increase visual appeal. The intervention was also varied in other ways, including the populations targeted.

As the researchers stated, "We consistently find no effect these messages [had] on student enrollment or financial aid outcomes." That result was true "across samples, content, timing, visual presentation and offers of personalized help."

Why could nudging work in small numbers but not in large numbers? The researchers offered a few possible explanations:

• First, most of those previous nudging activities involved a "local partner with closer connections to and knowledge of treated students." Those partners could know something about the students that helped with the outreach, and the students themselves might have reacted differently to messages from partners who they knew were "specifically invested in them or their communities."
• Second, the scaling-up for this study required more generic messages and less personalization, which meant students might not have recognized the importance of the communications.
• Third, the current crop of students may just understand FAFSA completion basics better than previous cohorts, making for a smaller population upon which nudging would make a difference.

As the authors of the report explained, "future attempts to scale up [nudging] campaigns should proceed cautiously and with attention paid to these hypotheses."

A digital version of the working paper is available for a modest cost on the NBER website.