As the field of ed-tech has grown, research around the efficacy of technology has been hard to come by. Part of the difficulty is finding accurate comparisons because schools, administrations, districts and student populations across the country have their own individual sets of criteria and challenges.
A recent report by the RAND Corporation, in partnership with the Department of Education, tries to provide an objective overview of blended learning. RAND conducted a national two-year randomized trial to determine whether a blended learning curriculum developed by Carnegie Learning, Inc. had a positive effect on middle and high school algebra students. The report found that the curriculum, which included both instruction time on computers and in-person, improved high school performance by 8 percentile points.
For the study, RAND reached out to schools in rural, urban and suburban areas and included students living in high-poverty areas, places with high populations of ethnic minorities and every other kind of school. By choosing a diverse population of more than 18,000 students in 147 schools in seven states, the aim of the study was to ensure that the results could be applied to schools in any environment.
“What happens a disappointing amount of time is you don’t find significant effects,” said John Pane, senior scientist at RAND and lead researcher for the study. “So to have identified something that is educationally meaningful is a success. Everything we did was with an eye to not interfering with how schools normally operate.”
For the study, participating schools were given access to the software and teachers were trained without any additional supports, just as they would be if a school purchased the software on its own. Most restricted studies using certain products, Pane said, provide extra professional development or support, skewing the results.
The DOE chose to study Carnegie Learning’s curriculum because of its track record of success in 46 other non-randomized studies. “We've done a large amount of field research coming into this that suggested that this approach would work better than others,” said Steve Ritter, chief product architect at Carnegie Learning. The company is an offshoot of cognitive psychology research at Carnegie Mellon dating back to the late 1980s. This curriculum has been in use since 1998. Its long track record is what prompted this expansive and expensive study. It cost $6 million.
Like other blended learning curriculum the Carnegie Learning approach relies on software to supplement teacher instruction. “We ask students to do a lot of writing about math, solving real world problems,” Ritter said. The company recommends using software two days a week and in-class instruction three days a week because things like classroom interaction, group discussion, collaborating on problem-solving helps cement the learning.
PROGRAM DETAILS
The program itself asks students to solve long multi-step problems. The software tracks the student’s problem-solving strategies and gives feedback on the process along the way. “In our system we ask the student to show their work to the computer,” Ritter said. The curriculum also recommends in-class instruction focused on small group work, articulating the problem solving process and even writing in the textbook.
Algebra is the focus of study, Ritter said, because it’s a moment in math education where students often fall behind. “Algebra is the place where math goes from being concrete to abstract,” Ritter said. He pointed out that it’s an essential skill to making quantitative predictions about the world, a skill required for statistics, for example.
But although students in the treatment group did improve, RAND could not definitively link that improvement to the software used. “Teaching was different in Carnegie Learning classrooms than in the control group classrooms; we can say that,” Pane said. “We can’t tell if it’s the software or something happening in the classroom.”
In a different study, RAND hopes to drill down on data from the software itself to find out its direct impact. When Carnegie Learning applied for the DOE grant seven years ago, the research question was whether the curriculum was effective, not whether the success was directly attributable to software – a more pressing question today as technology in the classroom proliferates.
The study did find that students performed better in the second year of implementation, no matter the experience of the teacher. In other words, even students taught by a teacher who had never used the blended curriculum before performed better than the control group. Both Ritter and Pane speculate that teachers adapted the curriculum in the second year and helped first year teachers along, thus improving its implementation.
It’s worth noting that RAND gave all students in the study a standardized readiness exam at the beginning of the year and a proficiency exam at the end of the course to measure learning. However, the Carnegie Learning curriculum focuses more on performance assessments.
“The real goal is for students to learn math well enough to solve math problems in the real world they live in, not on tests,” Ritter said. Still, he’s happy the RAND results on multiple-choice tests were positive. “We do believe it’s a valuable test of our system because we know that kids are mostly taking these standardized tests,” he said.
