This study—a follow-up to prior CAPR research using the same sample of students—employs a randomized controlled trial to investigate whether an algorithmic multiple measures assessment (MMA) system used at seven State University of New York (SUNY) community colleges led to better student outcomes—for up to four and a half years after randomization—than a system based on test scores alone. The authors estimate the overall treatment effects on enrollment in and completion of college-level math and English as well as effects on other outcomes. They also descriptively examine the outcomes of program group students whose placements were bumped up and bumped down by the MMA algorithm (many students’ placements did not change).
The study is described both in an article in the Journal of Research on Educational Effectiveness (authored by Kopko and Daniels Sarica) and in a CAPR working paper (authored by Kopko, Daniels Sarica, and Cullinan). In the working paper, the authors also present results from treatment effect analyses on race/ethnicity, Pell recipient status, and gender subgroups, and they present a cost-effectiveness analysis as well.
The study finds that the MMA method used at the colleges improved access to and success in college-level courses and that lower cut scores in English rather than math are associated with larger and longer lasting impacts on completion of college-level coursework. In English, program group students had higher rates of enrollment in and completion of (with a C or higher) college-level English courses. Impacts on enrollment and completion in English were statistically significant across all nine terms. In math, program group students had higher rates of enrollment in college-level math courses, and they had higher initial rates of completion. While impacts on enrollment persisted through term nine, there is no evidence that statistically significant gains in completion were sustained past the first term for the full sample.
While MMA improved outcomes among student subgroups, it had little to no impact on gaps in outcomes between subgroups. The study also finds that bumped-up students had substantially better outcomes in both math and English, while bumped-down students had substantially worse outcomes. The results suggest that increased access to college-level courses is the driving factor in the positive outcomes experienced by program group students and that placement into standalone developmental courses can have detrimental effects on student outcomes. In the discussion of the study’s results, the authors make recommendations for adopting MMA at colleges.
An accompanying CAPR brief focusing on the “bump-zone” findings of the study is also available: By the ninth term, program group students in the bump-up zone were about 9 percentage points more likely than their business-as-usual peers to complete college-level math or English.
A CAPR toolkit for reforming assessment and placement using MMA is available.
A version of this paper appears in Journal of Research on Educational Effectiveness.