Leonie Haimson does exactly what one would expect a class size advocate to do – make a forceful argument in favor of the benefits of smaller classes, emphasizing the studies that support her position, and minimizing those that do not. I’m not interested in a point-by-point sparring match with her; suffice it to say that I think that her broad claims for the benefits of class size reduction are not well-supported by the relevant social science evidence.
Part of what is at issue is what kinds of evidence are persuasive. There are two main considerations: (a) a study’s internal validity, and (b) a study’s external validity.
Internal validity matters. Internal validity refers to the credibility of cause-and-effect claims within a given study. The STAR study was a randomized experiment, and the design gives us a lot of confidence that the more favorable outcomes observed in the small class treatment was due to being in a class of 13 to 17 students, rather than a class of 22 to 25 students. Lots of other studies are not experiments. Some of them show more positive outcomes in small classes and others do not; but the challenge is justifying the claim that the association between class size and student outcomes is caused by being in a smaller class, rather than by other forces. In some studies, these alternative forces are known as selection bias—the nonrandom sorting of students into smaller and larger classes on the basis of student characteristics that might be associated with student outcomes. For example, we would not want to conclude that the reason that academic performance is higher in small honors classes than larger regular classes is because of class size. eduwonkette constantly reminds us of the dangers of selection bias. In others, such as the AIR/RAND study of California, other policy and assessment changes were occurring at the same time as class size reduction, and the researchers were unable to rule out the possibility that changes in student performance during the class size reduction period were due to these other changes. Like many social scientists, I give more weight to studies that have high internal validity. Economist Alan Krueger agrees: “[class size] studies are of varying quality and often examine very different outcomes for different populations. Should the amalgamation of such studies be trusted? Personally, I think one learns more about the effect of class size from understanding the specifications, data, methods and sensitivity of results in the few best studies than from summarising the entire literature.”
External validity matters. External validity refers to the extent to which claims that are derived from a particular study apply more broadly to other settings. Studies are conducted in a particular time and place, with a particular target population. I’ve suggested that a study of class size reduction in kindergarten and first grade might not tell us much about class size reduction in high school, because the settings are so different. I’ve also suggested that how class size reduction is implemented matters. Are the teachers qualified? Is there adequate space? The issue of external validity is also at the heart of economist Edward Lazear’s model that eduwonkette summarized last week: reducing class size in classes of well-behaved students may not tell us much about reducing class size in classes of poorly-behaved students, and vice versa. Economist Steve Barnett makes this point in discussing the three best-known early childhood education studies: Perry Preschool, Abecedarian, and Chicago Child-Parent Center. “No one should expect any public program to produce the same results as any one of the studies,” he writes. “To borrow a phrase from the US Environmental Protection Agency, for any particular public [early childhood education] program ‘your mileage may vary’. In general, variations in the population served, program design, and the neighborhood and broader social context can be expected to affect costs and benefits.” The same is surely true for class size reduction as well.
I thought that my posts last week were a balanced but positive endorsement for class size reduction, based on a reasonable reading of the research evidence. Leonie Haimson disagrees. My main crime seems to be that I don’t agree with what “actually” happened in STAR, in California, etc., which, miraculously, always favors smaller classes. Always! But don’t take my word for the “conventional wisdom” on class size reduction. Here are some quotes from “Class Size: Counting Students Can Count," a publication of the American Educational Research Association researched by the lead authors of the STAR study, Jeremy Finn and Charles Achilles, and reviewed by David Berliner, Eric Hanushek, and Larry Hedges. Read these quotes and judge for yourself if I’m distorting what leading researchers think about class size.
“The most dramatic impact seems to be achieved by reaching students early. Ideally, students should experience small classes of 13 to 17 students when entering school, in either kindergarten or first grade. While there is strong evidence of academic improvement during the first two years spent in a small class, there is more ambiguity about the value of additional years. It is not certain that there are added gains during second- and third-grade small classes.”
“There is no experimental research suggesting that any benefits are realized by subtracting only a few children from a larger class — for example, transitioning from 28 to 25 students. Even a class of 20 students may be too large.”
“In California, a lightning-quick ramp-up of statewide class-size reduction policy created many complications. Many new classrooms had to be found or built, and thousands of new teachers were hired within several months of the 1996 launch. Teaching quality suffered. While test scores have gone up in California since the small-class initiative started, researchers have been unable to determine how much, if any, of the improvement resulted from class-size reductions, as opposed to several other initiatives that were launched at around the same time.”
“[Small classes] are not a cure-all for low academic achievement, and they may not always be the best use of scarce resources. In weighing the pros and cons of a class-size reduction plan, policymakers will want to measure the costs of class-size reduction against other possible uses of the same funds.”
Let me close with a couple of suggestions to proponents of smaller classes. First, the hearts and minds of policymakers will not be won over by hitting them over the head with an increasingly thick sheaf of studies touting the effects of class size reduction. This is not a topic on which research is the most important determinant of policy outcomes. (In fact, there are very few policy issues that hinge most directly on what the research has to say.) One need only look at the literature on high-quality preschool programs to see that research hasn’t carried the day. The cost-benefit analyses of the Perry Preschool Project conducted by economist Clive Belfield and his colleagues have shown substantial benefits to the general public through age 40, to the tune of almost $13 for every $1 invested, across categories such as tax contributions, criminal activity, and welfare receipt. (There are benefits to individual participants as well.) But we’re still awaiting adequate funding for high-quality early childhood education programs for all children who can benefit from them. Why should it be different for class size reduction?
Second, I think a more successful strategy would be to focus on the development of a small, well-designed class size reduction initiative that can generate some “small wins” that can then be used to leverage expansion. Look, I have no love for the weasels that populate the senior management of the NYC Department of Education. (They’re not all weasels, of course, but let’s just say that the NYC Department of Health would shut down Tweed Courthouse faster than a KFC/Taco Bell in Greenwich Village.) But successful policy reform involves finding ways to work with managers and policymakers to craft actual class size reduction initiatives. All of the research in the world is not an actual class size reduction initiative. That’s what’s going to make a difference, and that’s where I’d direct my energies if I were a class size advocate.