Matthew Kraft, an associate professor of education and economics at Brown University, was an early proponent of giving tutors — ordinarily a luxury for the rich — to the masses after the pandemic. The research evidence was strong; more than a hundred studies had shown remarkable academic gains for students who were frequently tutored every week at school. Sometimes, they caught up two grade levels in a single year.
After COVID-19 shuttered schools in the spring of 2020, Kraft along with a small group of academics lobbied the Biden administration to urge schools to invest in this kind of intensive tutoring across the nation to help students catch up from pandemic learning losses. Many schools did — or tried to do so. Now, in a moment of scholarly honesty and reflection, Kraft has produced a study showing that tutoring the masses isn’t so easy — even with billions of dollars from Uncle Sam.
The study, which was posted online in late August 2024, tracked almost 7,000 students who were tutored in Nashville, Tennessee, and calculated how much of their academic progress could be attributed to the sessions of tutoring they received at school between 2021 and 2023. Kraft and his research team found that tutoring produced only a small boost to reading test scores, on average, and no improvement in math. Tutoring failed to lift course grades in either subject.
“These results are not as large as many in the education sector had hoped,” said Kraft in an interview. That’s something of an academic understatement. The one and only positive result for students was a tiny fraction of what earlier tutoring studies had found.
“I was and continue to be incredibly impressed with the rigorous and wide body of evidence that exists for tutoring and the large average effects that those studies produced,” said Kraft. “I don’t think I paid as much attention to whether those tutoring programs were as applicable to post-COVID era tutoring at scale.”
Sponsored
Going forward, Kraft said he and other researchers need to “recalibrate” or adjust expectations around the “eye-popping” or very large impacts that previous small-scale tutoring programs have achieved.
Kraft described the Nashville program as “multiple orders of magnitude” larger than the pre-COVID tutoring studies. Those were often less than 50 students, while some involved a few hundred. Only a handful included over 1,000 students. Nashville’s tutoring program reached almost 7,000 students, roughly 10% of the district’s student population.
Tennessee was a trailblazer in tutoring after the pandemic. State lawmakers appropriated extra funding to schools to launch large tutoring programs, even before the Biden administration urged schools around the nation to do the same with their federal COVID recovery funds. Nashville partnered with researchers, including Kraft, to study its ramp up and outcomes for students to help advise on improvements along the way.
As with the launching of any big new program, Nashville hit a series of snags. Early administrators were overwhelmed with “14 bazillion emails,” as educators described them to researchers in the study, before they hired enough staff to coordinate the tutoring program. They first tried online tutoring. But too much time and effort was wasted setting kids up on computers, coping with software problems, and searching for missing headphones. Some children had to sit in the hallway with their tablets and headphones; it was hard to concentrate.
Meanwhile, remote tutors were frustrated by not being able to talk with teachers regularly. Often there was redundancy with tutors being told to teach topics identical to what the students were learning in class.
The content of the tutoring lessons was in turmoil, too. The city scrapped its math curriculum midway. Different grades required different reading curricula. For each of them, Nashville educators needed to create tutor guides and student workbooks from scratch.
Eventually the city switched course and replaced its remote tutors, who were college student volunteers, with teachers at the school who could tutor in-person. That eliminated the headaches of troublesome technology. Also, teachers could adjust the tutoring lessons to avoid repeating exactly what they had taught in class.
But school teachers were fewer in number and couldn’t serve as many students as an army of remote volunteers. Instead of one tutor for each student, teachers worked with three or four students at a time. Even after tripling and quadrupling up, there weren’t enough teachers to tutor everyone during school hours. Half the students had their tutoring sessions scheduled immediately before or right after school.
In interviews, teachers said they enjoyed the stronger relationships they were building with their students. But there were tradeoffs. The extra tutoring work raised concerns about teacher burnout.
Despite the flux, some things improved as the tutoring program evolved. The average number of tutoring sessions that students attended increased from 16 sessions in the earlier semesters to 24 sessions per semester by spring of 2023.
Why the academic gains for students weren’t stronger is unclear. One of Kraft’s theories is that Nashville asked tutors to teach grade-level skills and topics, similar to what the children were also learning in their classrooms and what the state tests would assess. But many students were months, even years behind grade level, and may have needed to learn rudimentary skills before being able to grasp more advanced topics. (This problem surprised me because I thought the whole purpose of tutoring was to fill in missing skills and knowledge!) In the data, average students in the middle of the achievement distribution showed the greatest gains from Nashville’s tutoring program. Students at the bottom and top didn’t progress much, or at all. (See the graph below.)
“What’s most important is that we figure out what tutoring programs and design features work best for which students,” Kraft said.
Average students in the middle of the achievement distribution gained the most from Nashville’s tutoring program, while students who were the most behind did not catch up much
Source: Kraft, Matthew A., Danielle Sanderson Edwards, and Marisa Cannata. (2024). The Scaling Dynamics and Causal Effects of a District-Operated Tutoring Program.
Another reason for the disappointing academic gains from tutoring may be related to the individualized attention that many students were also receiving at Nashville’s schools. Tutoring often took place during frequently scheduled periods of “Personalized Learning Time” for students, and even students not selected for tutoring received other instruction during this period, such as small-group work with a teacher or individual services for children with special needs. Another set of students was assigned independent practice work using advanced educational software that adapts to a student’s level. To demonstrate positive results in this study, tutoring would have had to outperform all these other interventions. It’s possible that these other interventions are as powerful as tutoring. Earlier pre-COVID studies of tutoring generally compared the gains against those of students who had nothing more than traditional whole class instruction. That’s a starker comparison. (To be sure, one would still have hoped to see stronger results for tutoring as the Nashville program migrated to outside of school hours; students who received both tutoring and personalized learning time should have meaningfully outperformed students who had only the personalized learning time.)
Other post-pandemic tutoring research has been rosier. A smaller study of frequent in-school tutoring in Chicago and Atlanta, released in March 2024, found giant gains for students in math, enough to totally undo learning losses for the average student. However, those results were achieved by only three-quarters of the roughly 800 students who had been assigned to receive tutoring and actually attended sessions.*
Kraft argued that schools should not abandon tutoring just because it’s not a silver bullet for academic recovery after COVID. “I worry,” he said, “that we may excuse ourselves from the hard work of iterative experimentation and continuous improvement by saying that we didn’t get the eye-popping results that we had hoped for right out of the gate, and therefore it’s not the solution that we should continue to invest in.”
Iteratively is how the business world innovates too. I’m a former business reporter, and this rocky effort to bring tutoring to schools reminds me of how Levi’s introduced custom-made jeans for the masses in the 1990s. These “personal pairs” didn’t cost much more than traditional mass-produced jeans, but it was time consuming for clerks to take measurements, often the jeans didn’t fit and reorders were a hassle. Levi’s pulled the plug in 2003. Eventually it brought back custom jeans — truly bespoke ones made by a master tailor at $750 or more a pop. For the masses? Maybe not.
I wonder if customized instruction can be accomplished at scale at an affordable price. To really help students who are behind, tutors will need to diagnose each student’s learning gaps, and then develop a customized learning plan for each student. That’s pricey, and maybe impossible to do for millions of students all over the country.
*Correction: An earlier version of this story incorrectly stated how many students were assigned to receive tutoring in the Chicago and Atlanta experiment. Only 784 students were to be tutored out of 1,540 students in the study. About three-quarters of those 784 students received tutoring. The sentence was also revised to clarify which students’ math outcomes drove the results.
