Responding to NAEP: Blind Faith in New York City

Yesterday, Joel Klein lamented the news coverage of NYC's NAEP results in an email to 100,000 teachers, principals, and staff. He wrote:

Since the NAEP results were released last week, there has been a good deal of what, at best, can be called incomplete information in newspapers and elsewhere. I want to give you the facts. The facts are not uniformly positive, but, generally speaking, this is a story of good progress that is consistent with the overall picture.

Here is an overview of the facts:

1) NYC's progress on state tests has significantly outpaced progress on the NAEP. In addition, the percentage of NYC students proficient as measured by NAEP is dramatically smaller than the percentage measured as proficient by state tests.

The hope of accountability systems is that increases in state tests scores translate into improvements in children's academic skills that generate meaningful improvements in children's life chances. If the improvements in state test scores don't even translate into improvements on other tests, what's the chance that they're going to translate into better jobs or the ability to succeed in college?

The NY Times article has a clear graphic displaying the state test versus NAEP discrepancy. (This is a larger problem that Fordham has dubbed the "proficiency illusion.")

2) The Bloomberg/Klein Children First reforms were not implemented until the fall of 2003. The 2002 scores are not relevant in evaluating the effects of these reforms.

The NYC press release leads by attributing the math gains to the Children's First reforms ("This performance represents a nearly 12 percentage point gain since 2003, when Mayor Michael R. Bloomberg and Joel I. Klein introduced the Children First reforms" - as a sidenote, in this parallel universe, a difference of 11.45 rounds up to 12, not down to 11.)

Suddenly, when it's time to talk about reading, all comparisons are made with respect to 2002. I want to give you the facts: this is because there are no gains when you compare the 2007 scores with the 2003 scores.

3) Whether black-white and Hispanic-white achievement gaps are closing depends on how you look at it.

In his email, Joel Klein wrote, "Even as we're making gains, we're helping to close the achievement gap that separates our African-American and Latino students from our white students." He then displays a figure claiming the achievement gap is closing in every grade level and subject.

Let's take 4th grade math as an example. Viewed through a certain lens, Klein is correct to say that the achievement gap is closing. If you look at the percentage of white students at or above basic in 4th grade math and compare this with the percentage of black students at or above basic, the white/black gap has shrunk 10.4 points (from a difference of 29.5 to 19.1) since 2003.

But if you compare the percent of students at or above proficient, the black/white gap has actually grown from a gap of 30% to a gap of 33%. Hispanic kids have gained slightly on white students (by 2%). On the other hand, the white-Asian, black-Asian, and Hispanic-Asian gaps have all grown significantly. The black-Asian gap in the percentage of students at or above proficient was 35% in 2003; by 2007, it is 45%.

Fourth Grade Math: Percent At or Above Proficient By Race/Ethnicity, 2003-2007

To be clear, all is not lost in NYC. There has been progress in 4th grade math compared to 2003, and we should give credit where credit is due. But to say that NYC is setting the pace for urban school reform based on this performance is a canard. Even with its best results, NYC doesn't look better than many urban districts. For example, the percentage of NYC 4th graders at or above basic in math increased 11% from 2003 to 2007; in Boston it increased 18%; in DC, 13%; in Atlanta, 11%, and in Houston, 10%. And with its 8th grade results, NYC is not making progress. Other urban districts are.

With the exception of the 4th grade math scores, Joel Klein is asking us to take a vow of blind faith. Look at the overall picture, he says, and don't focus so heavily on the details of any one indicator (unless it's the one that shows we're making progress). But when 1.1 million kids' lives are in question, should we be willing to operate on blind faith? When Bloomberg sold us on a candidate who would be accountable for student achievement in the city, did he mean that we should look at the "whole picture?" When Joel Klein said that he was moving the system from a culture of excuses to a culture of results, was there a Barry Bondsian asterisk at the end of his sentence?

Klein, not unlike Bush, is a true believer; he's sure he has the policy toolbox to fix NYC's schools. And I do believe that his commitment to educational equity is genuine. Ironically, it is the strength of this commitment, coupled with his belief that his policies represent the one "true way," that makes him either unwilling or unable to reckon with the hard facts. But at the end of the day, empirical evidence should trump one's fanatical faith in particular policy solutions.

The New York City Dept of Ed has demanded "data-driven decision making" from its educators, but is now asking us to deny the data. NYC leaders see what they want to see, and they expect us to follow suit. Selective cognition, though, is dangerous stuff in policymaking. For the sake of the kids involved, let us hope that those running the Department of Ed will begin to look at all of the evidence and evaluate their policies accordingly.