NCLB I: Do accountability systems improve test scores, and for whom?

Part 1 of 3 of "NCLB glass half full (....with poison?)" Here are my thoughts on three questions related to the title of this post:

1) Do accountability systems improve test scores (i.e. percentage of students passing state tests)?

Despite both supporters' and opponents' eagerness to tell us how well NCLB is or isn't working based on limited data, the cleanest evidence of accountability systems' effects pre-dates NCLB. This earlier generation of studies examined data from Texas, North Carolina, and Chicago, and found, unsurprisingly, that accountability systems do substantially increase achievement on the state tests themselves. Other studies have looked at the effects of accountability systems on non-state tests. Two studies in particular, comparing NAEP progress in accountability and non-accountability states (again, all pre-NCLB), found that states with public monitoring (in Hanushek and Raymond's 2004 study) or those with stronger accountability policies (in Carnoy and Loeb's 2002 study (available here, #9 under articles) exhibited greater gains. But the gains on state tests consistently outpace those on NAEP, which some have called evidence of test-score inflation. See a recent paper by Brian Jacob here that addresses this issue in Texas.

So it's fair to say that accountability systems do increase test scores, which is different than saying they improve academic achievement. Remember, the test is not an end in itself, but a proxy for the real thing we care about - students' academic skills. The real question, then, is whether these increases translate to other venues in ways that improve students' life chances over the long haul.

2) Do increases in state tests scores translate into improvements in children's academic skills that generate meaningful improvements in children's life chances?

This question is the crux of the accountability issue, and we unfortunately don't know much about this question - though again, lots of people on both sides claim to. (If I am wrong, please email me references!) Put differently, are kids graduating from high school in Texas and North Carolina, who have gone through the better part of their K-12 educations under NCLB-like accountability systems, more likely to graduate from high school, go to college, and graduate from college than they would been otherwise? Are they more productive workers as a result? Better citizens?

3) For whom do accountability systems improve test scores, and what are implications of maintaining a "proficiency for all" target?

Proficiency-based accountability systems can lead educators to focus on kids close to the cut score for proficiency. These kids have been assigned a variety of titles, including marginal kids, cusp kids, or bubble kids (see my post here about Derek Neal's Chicago study). Since sanctions are doled out based on passing rates, slightly increasing the scores of a small number of students can help a school make AYP. Such a focus could affect the achievement of both high and low-scoring students. The irony is that in states where the bar for passing is higher, it is more likely that very low-performing kids will lose out under accountability systems, as they are likely to be well below passing.

What about the argument that with 100% proficiency, schools can't pursue this strategy forever? When people face impossible tasks, they focus too heavily on the short-term incentives and not at all on the long-term incentives. The short-term incentive here is to push one more kid over the passing mark. Yes, I am arguing, unapologetically, that 100% proficiency is impossible, that most educators would second that assertion, and that almost all researchers who study school effects would argue this is impossible as well, absent a dramatic change in how American schools, or American society for that matter, do business. The rate of improvement required for 100% proficiency would require schools to improve performance at a rate unlike anything even the top 5% of American schools have ever done. This is the jogger's equivalent of asking an eight minute miler to rev it up to a five and a half minute mile; that someone else can do it is not evidence that everyone can.

Teachers and principals are doing what they can to push forward, but if we stay at a 100% proficiency target, even with growth models, I fear that teachers are going to continue to attend to those close to passing first. This is all a way of answering the "for whom" question: kids at the tails of the distribution - low and high achievers - aren't served well by NCLB's current focus on proficiency. Another response to this is, "Well, it's not NCLB's fault that those bad news bears teachers are doing that." If you believe this is true, you may be surprised to learn that you also agree with the argument that guns don't kill people, people kill people. It's no different.

To sum up - and there's something here for both sides, I think - accountability systems do increase test scores, but they do so most for the kids closer to passing.