If a kid isn’t keeping up with peers academically, summer school seems like a no-brainer. Instead of forgetting what they learned during the school year while they’re on vacation for two months, they’re catching up and getting ahead. Which is why it was a surprise when a Rand Corporation study of summer school programs in five urban school districts1 found that this common-sense solution … didn’t really solve the problem.
Rand’s study found that summer school offered modest, short-term improvements in math scores at best, but those improvements faded by the fall. Other metrics — performance in language arts, student attendance and overall grades — showed no meaningful link to summer school. “The effects were pretty underwhelming,” said Megan Kuhfeld, a senior research scientist with NWEA, a nonprofit educational testing and research organization.
Overall, summer school programs didn’t deliver on their promises. But some subgroups did benefit: the students who regularly attended the programs that were better at navigating hurdles like student retention.
It’s perhaps never been so urgent to make educational interventions like summer school work for kids. Two years into the pandemic, children across the nation are behind where they would have been academically if the pandemic hadn’t happened. To help bridge the gap, educational theories will have to adapt to the unique realities of actual kids’ lives and families’ needs. If they don’t, even the best ideas, with tons of evidence behind them, won’t work in the real world.
Kids learned plenty during the pandemic, Kuhfeld told me. The problem, she said, is that they aren’t learning as much or as quickly as they were each year before the pandemic. Nationally, third-graders in fall 2021 were, on average, testing significantly below where third-graders were testing in fall 2019 in reading and math. The NWEA assessments showed these declines extended across third-graders through eighth-graders, too.2
Most of the experts I spoke to said the popular term “learning loss” is a misnomer — it’s not that kids have lost ground, they’re just not progressing as fast. But the slower progression is real, and there are patterns to it. The effects were particularly pronounced among Black, Hispanic and American Indian and Alaska Native students.
In the NWEA data, the median percentile ranks for Black third-graders went down 10 points in reading and 14 points in math. For white third-graders, the median percentile ranks declined by exactly half of that (5 points in reading and 7 in math), while the median percentile ranks for Asian American third-graders fell by 3 points in both subjects.
In addition, there’s evidence of declines in attendance and high school graduation rates, something that could signal a broad sense of emotional disconnection from school. Which, in turn, could help explain slowed learning — or exacerbate it, said Dan Goldhaber, director of the National Center for Analysis of Longitudinal Data in Education Research.
Slowed learning during the pandemic doesn’t necessarily mean kids are doomed, however. In fact, other researchers like Torrey Trust, a professor of learning technology at the University of Massachusetts Amherst, said kids actually learned a lot of things during the pandemic that they might not have learned otherwise. For many, virtual classes meant more time with family, more skills with technology, and for some, even better educational experiences, free from bullying.
The other good news: Research shows that the slower progress documented by these test scores should be able to be fixed with small-group tutoring. “It’s not rocket science,” said Thurston Domina, a professor of educational policy and organizational leadership at the University of North Carolina at Chapel Hill. “You get the kids in small groups and you can really give them customized instruction and focus on them.” When Matthew Kraft, a professor of education and economics at Brown University, reviewed several meta-analyses of the effectiveness of various educational interventions in 2021, he found that tutoring in small groups had a significantly greater effect on student test scores than changes in class size, longer school days or summer-school-type programs.
But while it’s relatively simple for researchers to run studies on classrooms or schools and figure out which interventions produce the best results, it’s hard for educators to take those findings and put them to work across America. The evidence doesn’t produce a solution — it just shows you how hard it’s going to be to craft a wide-reaching solution.
Case in point: those summer school studies. One of the biggest factors affecting the overall failure of summer school programs in the Rand analysis was that only around half the kids who attended one year didn’t come back for the next — and some kids didn’t even attend each day the first year. The kids who attended summer school habitually, for both years, did improve their math and language skills in ways that lasted all school year. But that group represented only about 35 percent of all the kids involved in the study.
So summer school works just fine — if you can get kids to actually go. And that sets up a whole other set of logistical complications that have to be studied and analyzed and implemented. It takes hiring the right teachers who have the motivation and specific interest in teaching summer school, Kuhfeld said. It also takes long-term dedicated recruitment of kids into the programs. Unlike with regular school, students don’t have to attend summer school, so getting them and their families to choose the programs means you have to build both interest and trust — neither of which is a given. And all of this takes money. “There’s a big gap between what should work in theory and what works in practice,” Kuhfeld said.
This kind of effect is depressingly common. When the George W. Bush administration set up a program to compile evidence-based educational resources in 2002, education specialists told me they’d hoped this program — the What Works Clearinghouse — would bridge the gap between academia and classrooms. They envisioned it as a way for teachers to get a better handle on how to use evidence-based interventions in the classroom. “We thought we would punch in third-grade math and get an answer,” said Rachael Gabriel, a professor of literacy education at the University of Connecticut.
But it never worked out to be that simple.
In many cases, researchers I spoke to found that teachers — the people tasked with educating students and bringing those test scores up — didn’t have much control over which interventions they could use and how. Those decisions were made higher up in the chain of administration. A teacher might want to try something and not be allowed. Or they might be excited to try something that was allowed but not be given the funding or staff or bus transport to make it happen effectively.
Making things work in a classroom is different from making things work in a whole district or a whole state or the whole country. That’s something Domina learned when California’s State Board of Education tried to mandate all eighth-graders to take and be tested on algebra. The idea was very much based on evidence, he said. Studies showed that separating some kids into elite math and others into remedial math served to widen inequality and narrow kids’ futures. Giving kids higher expectations leads them to do better. So expanding access to algebra for all should have reduced test-score gaps between rich and poor, white and Black.
But it didn’t. In fact, the opposite happened. Domina sees problems of scale — particularly staffing issues — at the heart of that failure. Offering algebra to everyone meant that schools needed a lot more algebra teachers, and quickly. But there were only so many fully qualified, highly skilled algebra teachers. A lot of kids, particularly the ones in lower-income schools, ended up with teachers who didn’t have as much experience and weren’t as effective at teaching the material, he said.
That story is particularly poignant now. Small-group tutoring can help students catch up on what they didn’t get a chance to learn during the pandemic. But small-group tutoring takes staff — and schools are one of many industries suffering from staffing shortages. Experts like Kraft are concerned that schools might create failing tutoring programs by using irregular volunteers or older students in place of dedicated staff.
Much like students, schools themselves aren’t necessarily functioning at a neutral, pre-pandemic state, either. “The biggest trend I’ve seen in the last 6-12 months is that schools are struggling to get the basics down. Staying open is hard,” said Chase Nordengren, the principal research lead for Effective Instructional Strategies at NWEA. He’s seen many cases where federal funds, which otherwise may have been spent on staffing tutoring programs to mitigate learning loss, were spent instead on things like better ventilation, personal protective equipment and substitute teachers.
“I think tutoring is a really promising initiative,” Goldhaber said. “But we have never tried to do tutoring at the scale that we are trying it today.” Because of that, he said, parents should be advocating for real-time evaluation and course-correction to go along with these learning-loss interventions. There should be tools in place to help teachers know when something isn’t working for their specific school and allow them to make the kind of personalized adjustments we know are necessary to make any intervention effective. But that, again, takes resources.
In the end, it’s not kids’ pandemic test scores that really make researchers feel gloomy about the future of education. Instead, it’s the way educational systems have been set up to fail those kids. Schools have been running with limited resources and little wiggle room for change for at least the past decade, Domina said. “And now we’ve hit a crisis. And they’re not resilient.”