Wednesday, November 21, 2007

Responding to NAEP: Blind Faith in New York City

Yesterday, Joel Klein lamented the news coverage of NYC's NAEP results in an email to 100,000 teachers, principals, and staff. He wrote:

Since the NAEP results were released last week, there has been a good deal of what, at best, can be called incomplete information in newspapers and elsewhere. I want to give you the facts. The facts are not uniformly positive, but, generally speaking, this is a story of good progress that is consistent with the overall picture.

Here is an overview of the facts:

1) NYC's progress on state tests has significantly outpaced progress on the NAEP. In addition, the percentage of NYC students proficient as measured by NAEP is dramatically smaller than the percentage measured as proficient by state tests.

The hope of accountability systems is that increases in state tests scores translate into improvements in children's academic skills that generate meaningful improvements in children's life chances. If the improvements in state test scores don't even translate into improvements on other tests, what's the chance that they're going to translate into better jobs or the ability to succeed in college?

The NY Times article has a clear graphic displaying the state test versus NAEP discrepancy. (This is a larger problem that Fordham has dubbed the "proficiency illusion.")

2) The Bloomberg/Klein Children First reforms were not implemented until the fall of 2003. The 2002 scores are not relevant in evaluating the effects of these reforms.

The NYC press release leads by attributing the math gains to the Children's First reforms ("This performance represents a nearly 12 percentage point gain since 2003, when Mayor Michael R. Bloomberg and Joel I. Klein introduced the Children First reforms" - as a sidenote, in this parallel universe, a difference of 11.45 rounds up to 12, not down to 11.)

Suddenly, when it's time to talk about reading, all comparisons are made with respect to 2002. I want to give you the facts: this is because there are no gains when you compare the 2007 scores with the 2003 scores.

3) Whether black-white and Hispanic-white achievement gaps are closing depends on how you look at it.

In his email, Joel Klein wrote, "Even as we're making gains, we're helping to close the achievement gap that separates our African-American and Latino students from our white students." He then displays a figure claiming the achievement gap is closing in every grade level and subject.

Let's take 4th grade math as an example. Viewed through a certain lens, Klein is correct to say that the achievement gap is closing. If you look at the percentage of white students at or above basic in 4th grade math and compare this with the percentage of black students at or above basic, the white/black gap has shrunk 10.4 points (from a difference of 29.5 to 19.1) since 2003.

But if you compare the percent of students at or above proficient, the black/white gap has actually grown from a gap of 30% to a gap of 33%. Hispanic kids have gained slightly on white students (by 2%). On the other hand, the white-Asian, black-Asian, and Hispanic-Asian gaps have all grown significantly. The black-Asian gap in the percentage of students at or above proficient was 35% in 2003; by 2007, it is 45%.

Fourth Grade Math: Percent At or Above Proficient By Race/Ethnicity, 2003-2007

To be clear, all is not lost in NYC. There has been progress in 4th grade math compared to 2003, and we should give credit where credit is due. But to say that NYC is setting the pace for urban school reform based on this performance is a canard. Even with its best results, NYC doesn't look better than many urban districts. For example, the percentage of NYC 4th graders at or above basic in math increased 11% from 2003 to 2007; in Boston it increased 18%; in DC, 13%; in Atlanta, 11%, and in Houston, 10%. And with its 8th grade results, NYC is not making progress. Other urban districts are.

With the exception of the 4th grade math scores, Joel Klein is asking us to take a vow of blind faith. Look at the overall picture, he says, and don't focus so heavily on the details of any one indicator (unless it's the one that shows we're making progress). But when 1.1 million kids' lives are in question, should we be willing to operate on blind faith? When Bloomberg sold us on a candidate who would be accountable for student achievement in the city, did he mean that we should look at the "whole picture?" When Joel Klein said that he was moving the system from a culture of excuses to a culture of results, was there a Barry Bondsian asterisk at the end of his sentence?

Klein, not unlike Bush, is a true believer; he's sure he has the policy toolbox to fix NYC's schools. And I do believe that his commitment to educational equity is genuine. Ironically, it is the strength of this commitment, coupled with his belief that his policies represent the one "true way," that makes him either unwilling or unable to reckon with the hard facts. But at the end of the day, empirical evidence should trump one's fanatical faith in particular policy solutions.

The New York City Dept of Ed has demanded "data-driven decision making" from its educators, but is now asking us to deny the data. NYC leaders see what they want to see, and they expect us to follow suit. Selective cognition, though, is dangerous stuff in policymaking. For the sake of the kids involved, let us hope that those running the Department of Ed will begin to look at all of the evidence and evaluate their policies accordingly.

Tuesday, November 20, 2007

Overheard in New York

After almost five and a half years as chancellor, I know you can’t point to a single number, be it a test score or graduation rate, to prove success or failure. The whole picture is important.

--Joel Klein

Unless, of course, you're talking about grading schools - where you can just point to a single letter.

In a memo, Klein goes on to say:

We need to look at all the indicators we have—promotion rates, graduation rates, State English and math test results, Regents pass rates, national test results, and even the results of things like Advanced Placement and PSAT exams—to understand how well our students are performing and progressing. Together, these factors paint a complete picture. In that context, I am encouraged by the results we got last week on a key national indicator, the National Assessment of Educational Progress results.

Monday, November 19, 2007

Impression Management, or: How I Learned to Stop Hating Press Releases on Test Scores

Once upon a time in a land far, far away, district leaders did not live and die by their students' test scores. When I peered back into the educational debates of the 1980s and even the early 1990s, I found that the release of state and federal test scores was not an occasion for carefully orchestrated media stunts.

Sure, there was spin then, too - but district leaders were just as likely to decry poor test results to build political momentum to infuse additional resources or make needed changes.

Not so last week, where almost every district publicly releasing scores from the Trial Urban District Assessment spun a fabulous yarn around the NAEP results. Indeed, some of the results were promising. But many of the results that district leaders swaggered about were not. Instead, district leaders cherry-picked the results, up-playing positive news while ignoring or discounting negative news.

I pulled the press releases and local news articles from the TUDA districts, and here's a sample of what I found:
    • Chicago and Cleveland - both of which posted lackluster performance - chose the "What TUDA Assessment?" approach. Neither district issued a press release. (However, Cleveland did issue a press release when Cleveland football teams made the championship game.) Barbara Eason-Watkins, Chief Education Officer for the Chicago Public Schools, dismissed the flat scores from 2005 to 2007, noting, "We are encouraged that we have been trending up in all subjects at all grades."

    • Houston leads its press release by saying that "HISD students are beating their peers in most other big-city school districts" - which is true - but buries a key fact and then dismisses it: "4th graders in Houston this year declined in reading, but overall, HISD's scores were up."

    • Perhaps most astounding was New York City's press release, which leads with "New York City students made impressive gains on the 2007 NAEP" - never mind the 4th grade and 8th grade reading results, or the 8th grade math results.
If I sound frustrated, it's because I am. This dynamic is too reminiscent of one of the most potent quotes from Ron Suskind's 2004 NYT Magazine article on the Bush administration, where a senior aide remarked:
We're an empire now, and when we act, we create our own reality. And while you're studying that reality -- judiciously, as you will -- we'll act again, creating other new realities, which you can study too, and that's how things will sort out. We're history's actors . . . and you, all of you, will be left to just study what we do.
To me, it seems that the tone of education policy debates has shifted in the last 25 years. At the center of this shift are the concepts of "impression management" and "reality creation" (see here for one recent example in New York City). Maybe this is a common cultural trend, or perhaps I am re-imagining an idealized past that never existed.

For those who have been in this game longer than I have, please weigh in. On Wednesday, I'll profile one district's response to TUDA in more depth in an attempt to understand the increasing role of impression management in urban education policy.

NYC Report Card Haiku Finale!

Last week, readers wrote 68 haiku, which are published here as the New York City Report Card Haiku Magazine! Read it, print it, share it.

In lieu of holding a vote for the best haiku, I've decided to share my five favorite ones instead:

Amateur Night's spozed
to be at the Apollo
not at Tweed courthouse
-Anonymous 7:50 AM

Step onto the scale
I'm fatter than my neighbor
So I'll get a C
-Anonymous 12:31 AM

My school got an A
I never learned to Haiku
I read passages
-Doug Douglass

The wise little tree
Bends with the strong winds and does
Not break easily
-Anonymous 10:21 PM

The only numbers
that level the playing field
have a dollar sign
-Anonymous 10:54 PM

Thank you to everyone who contributed a haiku!

Monday Morning NYC Report Card Factoid

NCLB uses a very different metric of school success than the NYC Report Cards. What percentage of A-F schools in NYC are schools in need of improvement under NCLB?

  • Of A schools, 17.9% (50 schools) are schools in need of improvement under NCLB.

    Of B schools, 29.8% (139 schools) are schools in need of improvement under NCLB.

    Of C schools, 32% (101 schools) are schools in need of improvement under NCLB.

    Of D schools, 32.3% (32 schools) are schools in need of improvement under NCLB.

    Of F schools, 32% (16 schools) are schools in need of improvement under NCLB.

    Of schools with grades still "under review" as of today, 50% (7 schools) are schools in need of improvement under NCLB.

What's a principal to do? To succeed under NCLB, schools need to lift kids over the proficiency bar. To succeed in NYC, schools need to focus on the growth of the lowest performing kids.

On Thursday, you can be thankful for turkey, your family, and the fact that you are not a principal in NYC.

Sunday, November 18, 2007

This week: The Test Score Spin Doctors

Since NCLB, test scores have become the primary currency in education policy debates. From every corner, we are deluged with claims about which direction test scores are moving.

District, state, and federal policymakers now serve as our educational spin doctors, framing test results for the public in order to support their agendas.

In this short week, I'll consider the packaging and repackaging of results from last week's NAEP Trial Urban District Assessment.

: Impression Management, or: How I Learned to Stop Hating Press Releases about Test Scores

Wednesday: A Case Study of One District's Response to NAEP TUDA Results