State of Student Aid in Texas – 2022

Section 10: Evidence-Based Programs and Interventions

<< BACK TO MAIN PAGE

Interventions to Provide Support and Skills Training Improves Employment Outcomes for Students in Some Two-Year Programs

Given cost pressures at colleges, identifying interventions that can have the greatest impact on student success is vital. Research using random controlled trials provides meaningful insight into the extent to which various interventions are effective in promoting desired outcomes.

Project QUEST is a San Antonio, Texas organization aimed at helping low-income residents complete job-focused higher education programs and become gainfully employed. The organization provides students in specific technical and skill-based programs with a comprehensive suite of support and resources including financial assistance, remedial instruction for placement tests, personal and academic counseling, weekly meetings with a focus on life and study skills, and job placement assistance throughout the student’s time pursuing a degree and employment.

In order to evaluate the effectiveness of these resources, Project QUEST conducted a randomized trial among students who started services with Project QUEST between 2006 and 2008 and followed them for six years. In 2017, the study was extended to assess the impact on participant’s earnings, educational attainment, and public benefits usage in the nine years following random assignment. The evaluation focused on those pursuing skilled technical positions in medical fields (e.g., registered nurses, licensed vocational nurses, sonography technicians, etc.). Students who completed the program saw long-term gains in wages and employment, with participants earning over $5,000 more than the control group annually, on average, and experiencing a 15 percent higher level of year-round employment compared to the control group.

Average Annual Earnings for Project QUEST and Non-Participants

Average Annual Earnings for Project QUEST and Non-Participants

Sources: Project QUEST, Nine Year Gains: Project QUEST’s Continuing Impact, April 2019 (https://questsa.org/quest_impact/).

Individualized Coaching Is Effective for Increasing Attainment and Persistence

A Portland-based company and a member of the Strada Education Network, InsideTrack, contracts with higher education institutions in all sectors to provide individualized student coaching to improve student success. The company conducted randomized trials at multiple institutions where they randomly divided students at a school into two groups, providing coaching to one group and not to the other, in order to observe the effect of the coaching experience on retention and graduation. The National Bureau of Education Research evaluated many of those experiments, selecting those from the 2003-04 school year, to enable comparisons with certain national survey data, and the 2007-08 school year, as the most current year available at the time of evaluation.

The evaluation found that, when controlling for covariates (i.e. age, gender, high school GPA, SAT score), individualized coaching was associated with an approximately five percent increase in retention after six months, a 12 percent increase after 12 months, and a 14 percent increase after 24 months compared to the control group. Groups receiving coaching also saw levels of completion four percent higher than groups who did not receive coaching.

Individualized student coaching, in addition to more traditional advising financial and academic advising and counseling, has the potential to increase persistence and attainment rates and was more cost effective, in the case of this InsideTrack evaluation, than other previously studied methods of increasing attainment, such as increasing financial aid.

InsideTrack Student Coaching Evaluation: Persistence Increases Associated with Individualized Coaching, Compared to Control Group

InsideTrack Student Coaching Evaluation: Persistence Increases Associated with Individualized Coaching, Compared to Control Group

Sources: Bettinger and Baker, The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Advising, March 2014 (http://journals.sagepub.com/doi/abs/10.3102/0162373713500523).

Text Nudges Provide Needed “Summer Melt” Intervention at a Low Cost

At times, colleges want to provide outreach and interventions to a large number of students in order to improve certain outcomes, such as retention and graduation, but often have limited resources. Sending text messages with targeted messaging at key intervals, commonly referred to as “text nudges,” offer an empirically tested method for positively influencing students along a variety of outcome variables. Text nudges have been found to be effective in combatting attrition during the summer following initial enrollment, known as summer melt,** for as little as two dollars per student included in the texting campaign. Text nudges can be more easily scaled to a larger student population compared to more robust interventions such as phone calls or advising sessions, which, while typically effective, tend to be more labor intensive.

Researchers from the University of Virginia and the University of Pittsburgh conducted randomized controlled trials in 2012 at three high schools. The students in the experimental groups received text messages reminding them about college-related deadlines or required tasks (e.g. reminders to access important paperwork and register for orientation). These text nudging interventions were designed to increase the percentages of college-bound high school graduates that matriculate in the fall.

The text nudges were found to be effective for students with moderate GPAs, students who were enrolling in a two-year program, students enrolled in free or reduced-price lunch programs, students with unspecified college plans, and students who had not completed the FAFSA. Two-year programs experienced a statistically significant increase in enrollment with an increase of three percent. This suggests that summer melt text nudges are most effective for groups that may have limited access to other quality college information sources and represent a cost-effective intervention, but may be insufficient when used alone for many groups of students.

Enrollment Gains from Summer Melt Text Nudges

Enrollment Gains from Summer Melt Text Nudges

* Statistically significant at the 0.05 level (p<0.05).
** Summer melt is a term used to describe the occurrence of students indicating their intent to attend a college in the fall but then ultimately not matriculating. Some have defined this term as only including those who did not matriculate at any college while others have defined it as specific to an institution. Students may indicate their intent to attend college through various activities such as expressly saying so on a form to a counselor, paying college deposits, and registering for classes. The activities used to determine intent depend upon the definition being used for summer melt. For the purposes of this study, the researchers used information on students’ expressly stated intentions to attend college and defined summer melt as “the phenomenon that college-intending high school graduates fail to matriculate in college anywhere in the year following high school.”
Sources: Summer Nudging: Can personalized text messages and peer mentor outreach increase college going among low-income high school graduates? Castleman, B. and Page, L., Journal of Economic Behavior and Organization (2015), (https://www.sciencedirect.com/science/article/pii/S0167268114003217).

Text Nudges Can be Used to Improve Two-Year Outcomes During a Student’s Academic Career

The sending of targeted messaging via text messages at key intervals is commonly referred to as a “text nudge”. Text nudges have been found to have positive effects in facilitating increased rates of annual FAFSA completion, and in promoting retention and attainment, for a minimal cost to the institution. A series of studies on text nudges used in varying contexts have suggested that text nudges may provide a low-cost alternative or supplement to other more intensive, and expensive, methods of outreach during a student’s academic career, but are limited in effectiveness for some groups of students.

Researchers from the University of Virginia and the University of Pittsburgh used a randomized controlled trial design to examine the impact of text nudges on FAFSA re-filing rates among college freshmen. Text nudges containing information on where to obtain help with financial aid, important deadlines and requirements, and offering assistance related to financial aid and were sent to a randomly assigned group of community college freshmen during the 2012-13 academic year. Outreach took place over the course of approximately seven months with messages approximately every two weeks. Text nudges designed to provide important information and prompting concerning annual refiling of FAFSA have been found to be highly effective among community college students. Freshman community college students who received text nudges were nearly 12 percent more likely to persist into the fall of their sophomore year and were 14 percent more likely to persist into the spring.

Text messages represent a viable cost-effective option and are a valuable tool as part of a set of strategies to impact academic accessibility, persistence, and attainment; however, used alone, text nudges are likely to be inadequate for the overall student population. While impacts are substantial in some cases, effects are consistently limited to specific groups of students, often those with low availability of resources.

Enrollment Change For Two-Year College Students Receiving FAFSA Re-Filing Text Nudges

Enrollment Change For Two-Year College Students Receiving FAFSA Re-Filing Text Nudges

* Statistically significant at the 0.05 level (p<0.05).
Sources: Freshman year financial nudges: An experiment to increase FAFSA renewal and college persistence. Castleman, B. and Page, L., Journal of Human Resources (2016), (http://jhr.uwpress.org/content/51/2/389.short).

Need-Based Grants Increase Retention, Graduation, and Enrollment

A metanalysis by Snevers and DeWitt (2018) of ten recent studies examined the effects of need-based grants on enrollment, retention, and graduation. This analysis demonstrated overall positive effects of need-based grants across the three outcomes measured in all studies. While these effects were small, they were strongly significant.

More granularly, the study found that need-based grants had a small positive impact on enrollment across studies in different educational contexts and multiple countries. Enrollment among those receiving grants increased by 2.5 percent compared to the control group. When looking at retention, the study similarly found a small but significant positive impact from need-based grants. When compared to their peers, the recipients of need-based grants were 2.5 percent more likely to be retained. This effect was again seen across all studies included in the analysis.

Finally, the results indicated that need-based grants have a positive effect on students ultimately graduating, but not necessarily within the ideal time to degree, such as four years for a bachelor’s degree. Students who received grants were significantly more likely to graduate when looking at longer time to degree; however, no significant results were found for graduation rates within the ideal time to degree. The authors note that this seems to indicate that the effects of need-based grants may be delayed or may increase over time.

Enrollment, Retention and Graduation Change For College Students Receiving Need-Based Grants

Enrollment, Retention and Graduation Change For College Students Receiving Need-Based Grants

Sources: Eline Sneyers & Kristof De Witte (2018). Interventions in higher education and their effect on student success: a meta-analysis, Educational Review, 70:2, 208-228, DOI: 10.1080/00131911.2017.1300874 (https://doi.org/10.1080/00131911.2017.1300874).

Evaluations of Community College Program for Low-Income Students Find Positive Results

In 2007, the City University of New York (CUNY) launched the Accelerated Study in Associate Programs (ASAP) with the aim to improve graduation rates among low-income students. The program provided services, such as tutoring and individual advising, and financial assistance with public transportation, textbooks, and tuition. An evaluation of the program found that retention rates, credit accumulation, and graduation rates were significantly increased among program participants as compared to a control group. The program group had a graduation rate that was nearly double that of the control group after three years (40 percent vs 22 percent) and was still 10 percentage points higher than the control group after six years (51 percent vs 41 percent), showing that the program both increased the graduation and helped students graduate at a faster rate.

The ASAP model was replicated by MDRC as a demonstration project in 2014 at three Ohio community colleges. An evaluation of the Ohio ASAP found similar results to CUNY. Retention rates, credit accumulation, and graduation rates were all significantly higher among program participants as compared to the control groups. As with the CUNY ASAP, the Ohio ASAP saw the graduation rate nearly double among program participants compared to the control group after three years (35 percent vs. 19 percent). Program participants also transferred to four-year institutions at higher rates than their peers in the control group (18 percent vs. 12 percent). As of 2022, the ASAP National Replication Collaborative includes colleges in five other states, in addition to the programs in New York and Ohio, that receive customized assistance to set up their programs. Though there are slightly higher costs associated with implementing a program like ASAP than typical services, these evaluations demonstrate the real-world benefits and can help inform conversations about costs and the return on investment.

CUNY ASAP: Earned Any Degree

CUNY ASAP: Earned Any Degree

Sources: MDRC, The Power of Fully Supporting Community College Students: The Effects of the City University of New York’s Accelerated Study in Associate Programs After Six Years, October 2017 (https://www.mdrc.org/publication/power-fully-supporting-community-college-students); MDRC, Doubling Graduation Rates in a New State: Two-Year Findings from the ASAP Ohio Demonstration, December 2018 (https://www.mdrc.org/sites/default/files/ASAP_brief_2018_Final.pdf); MDRC, Increasing Community College Graduation Rates with a Proven Model: Three-Year Results from the Accelerated Study in Associate Programs (ASAP) Ohio Demonstration (https://www.mdrc.org/sites/default/files/ASAP_OH_3yr_Impact_Report_1.pdf); ASAP National Replication Collaborative (https://www1.cuny.edu/sites/asap/replication/#1605743953673-ac7ec7cc-d7a9).

Financial Literacy Can Predict Positive Future Financial Outcomes

The FINRA Investor Education Foundation’s National Financial Capability Study questionnaire was administered to a panel of participants in 2012 and 2018. The longitudinal data allowed for an analysis of the causal effect of financial knowledge on various financial outcomes. In both years, participants answered a five-question financial literacy scale to assess knowledge of fundamental concepts of finance, including topics like interest, inflation, and risk diversification.

The score derived from this financial literacy scale was found to be quite stable for most participants between 2012 and 2018. Younger participants, between ages 18 and 32 experienced an increase in their score on average while older participants, above the age of 64, experienced some decline on average. Overall, across all participants, financial literacy was relatively stable over time.

Financial literacy was found to have a significant relationship with several financial outcomes. Every one-unit increase in the 2012 financial literacy scale score was associated with a 2.5 percent increase in financial satisfaction in 2018, an 8.0 percent increase in the likelihood of being able to meet an unexpected $2,000 expense in 2018, and a 6.0 percent increase in the likelihood that the respondent had done any retirement planning. Additionally, the financial literacy score in 2012 was not significantly related to three negative outcomes measured: the perception of having too much debt, the use of alternative financial services, and costly credit card behaviors. This suggests that poor financial outcomes may not be related to low financial literacy, and that there are other factors at play that were not captured in this study.

Downstream Positive Effects of Increases in Financial Literacy: Percent Increase in 2018 Financial Behavior Relative to the Mean for Every One-Unit Increase in 2012 Financial Literacy Score

Downstream Positive Effects of Increases in Financial Literacy: Percent Increase in 2018 Financial Behavior Relative to the Mean for Every One-Unit Increase in 2012 Financial Literacy Score

Percent Increase in 2018 Ability to Meet $2,000 Shock Relative to the Mean for Every One-Unit Increase in 2012 Financial Literacy Score

Percent Increase in 2018 Ability to Meet $2,000 Shock Relative to the Mean for Every One-Unit Increase in 2012 Financial Literacy Score

Sources: Angrisani, Marco; Burke, Jeremy; Lusardi, Annamaria; and Mottola, Gary. The Stability and Predictive Power of Financial Literacy: Evidence From Longitudinal Study, October 2020 (https://www.finrafoundation.org/sites/finrafoundation/files/stability-and-predictive-power-financial-literacy-evidence-longitudinal-data.pdf).

Share This Page!

BACK TO TOP