What determines how well kids do in school?

By applying his expertise in statistical physics to analyze school test scores, Mike Marder discovered two times at which the performances of economically disadvantaged kids take a dive: the transitions to middle school and high school. And, he says, his findings question the commonly held idea that teacher effectiveness is the most important variable in student performance. “Poverty may be more important.”

Marder, a physicist in the University of Texas at Austin’s Center for Nonlinear Dynamics, and graduate student Dhruv Bansal analyzed five years of Texas standardized math test results, encompassing around 17 million scores of 4.6 million distinct third to eleventh graders from about 8000 schools. They used coarse graining approaches, the Fokker–Planck equation, and, for predictive purposes, Markov models. “All of these concepts have immediate applications for making sense of the patterns of scores,” says Marder. For example, he says, “only by putting results in terms of the Fokker-Planck equation do I know how to specify convective and diffusive contributions specifically and uniquely.” A convective variable would be “genuine learning, all students get better. The diffusive effect is when [students] guess, or random variations from teacher to teacher. Every phenomenon has some deterministic and some random component. You can capture that with a second order Taylor expansion.”

Sample results for 2006 and 2007 are shown in these plots of score versus grade level. The lower plot shows students who were eligible for free and reduced-price school meals, and the upper plot shows those who were not. The tail of each arrow is the average test score for a given 10% interval in 2006, with the arrow’s point made up of the same students’ scores in 2007. Arrow area is proportional to the number of students. The shading in the plots shows the cutoffs for failing (blue), passing (orange), and commendable (green) performance. The results are similar from year to year. (Figures courtesy of Mike Marder and Dhruv Bansal.)

The flow patterns for all students start out similar, “until the transition to middle school between fifth and seventh grade, when students from economically disadvantaged backgrounds flow downwards at a higher pace than their less-disadvantaged counterparts and never recover,” Marder and Bansal write in a recent paper (Proceedings of the National Academy of Sciences USA, volume 106, page 17267, 2009). Another downward flow is seen at the transition to high school.

“A common claim is that the dominant factor in student performance is the teacher,” says Marder, who helped design and run UTeach—the university’s program to prepare secondary math and science teachers that was held up as an example by President Obama in a 24 July 2009 speech. “I’m committed to improving teacher quality,” says Marder, “but the important question is, Is it enough? I believe our data say otherwise.” Income level plays a large role, he says. More specifically, “it’s not your income that’s so important—the fraction of other low-income families around you is more relevant.”

Bolstering Marder’s belief that improving teacher quality is not itself enough to boost student performance is the poorer performance in schools that have an increasing percentage of kids eligible for a free lunch. Individual and school-averaged scores on annual tests, the changes in those scores from year to year, and scores on the SAT all follow that pattern, says Marder. “Once the fraction of low-income kids in a high school passes 80%, there is not a single school where more than 20% of graduates get the college-ready SAT score of 1110. Not one,” says Marder. “If good teachers alone were actually the most important single factor for student learning, surely at least one school serving disadvantaged kids would have done better than this.”

Marder hopes his results reach people who work to improve schools and measure the effectiveness of teachers. “The coming availability of $5 billion in stimulus [American Recovery and Reinvestment Act] funding for school reform makes this a good time for new ideas,” he says. “I am hoping that by having orthogonal mathematical tools to look at this problem, we may open space for discussion.”