WWMD on NAEP Exemptions

A-Rus is perplexed over at This Week in Education, and so am I. Why isn't anyone concerned that some districts exempted 20% of the total student population on the NAEP Trial Urban District Assessment?

Elizabeth Green is. A-Rus is. I am. But where are the groups most in favor of keeping ELL/students with disabilities in the accountability system - Ed Trust, Ed Sector, ELL/students with disabilities advocates, and the Secretary of Education - on this? Or groups like Fordham or the NYT that are concerned with having one national measure that allows for comparability across states?

WWMD means "What Will Maggie Do?"

Lies, Damned Lies, and NAEP Exemptions

It turns out that there are four kinds of lies: lies, damned lies, statistics, and NAEP exemptions.

The purpose of the NAEP Trial Urban District Assessment is to "make it possible to compare the performance of students in participating urban school districts to that of public school students in the nation, in large central cities, and to each other." If comparability is the goal, exemption and accommodation provisions must be roughly similar in all districts.

At present, wide variation in exemption and accommodations makes comparing districts an almost impossible task. On the 4th grade reading assessment, some urban districts exempted up to 20% of the total population, while others only exempted 3%. Comparing scores within the same district over time is also a problem - DC exempted 8% of all students from the 4th grade reading assessment in 2002 but 14% in 2007.

Some key problems:

• Four districts - Austin (20%), Cleveland (17%), Houston (17%), and DC (14%) exempted more than 10% of all students on the 4th grade reading test. (See Table 1 below.)
  
  
• Two of the districts that showed exceptional progress - Atlanta and DC - have exhibited large amounts of growth in their exemption rates on the 4th grade reading test. In 2002, Atlanta only exempted 25% of all students with disabilities/ELL; in 2007, it exempted 58%. In 2002, DC exempted 42% of all students with disabilities/ELL; in 2007, it exempted 64%. (See Table 2.)
  
  
• Accommodation rates also vary widely across districts. New York City gave accommodations to 76% of all students with disabilities/ELL on the 4th grade reading test. Los Angeles only gave accommodations to 13%. (Table 3 below.)
  
  
• These patterns persist on the math test; there is wide variation in the use of exemptions and accommodations. (See Tables 4 and 5.)
  

What happened to the "you must test 95% of every subgroup" philosophy? If we're going to invest in these assessments and use them to inform policy, there need to be uniform testing protocols across all districts.

Until then, make these within (i.e. over time) and between district comparisons with care. Hat tip to the New York Sun's Elizabeth Green for breaking this story.

Table 1. Percent of All Students Exempted, 4th Grade Reading: 2002-2007

Table 2. Percent of Students with Disabilities/English Language Learners Exempted, 4th Grade Reading: 2002-2007

Table 3. Percent of Students with Disabilities/English Language Learners Receiving Accommodations, 4th Grade Reading: 2002-2007

Table 4. Percent of Students with Disabilities/English Language Learners Exempted, 4th Grade Math: 2003-2007

Table 5. Percent of Students with Disabilities/English Language Learners Receiving Accommodations, 4th Grade Math: 2003-2007

Responding to NAEP: Blind Faith in New York City

Yesterday, Joel Klein lamented the news coverage of NYC's NAEP results in an email to 100,000 teachers, principals, and staff. He wrote:

Since the NAEP results were released last week, there has been a good deal of what, at best, can be called incomplete information in newspapers and elsewhere. I want to give you the facts. The facts are not uniformly positive, but, generally speaking, this is a story of good progress that is consistent with the overall picture.

Here is an overview of the facts:

1) NYC's progress on state tests has significantly outpaced progress on the NAEP. In addition, the percentage of NYC students proficient as measured by NAEP is dramatically smaller than the percentage measured as proficient by state tests.

The hope of accountability systems is that increases in state tests scores translate into improvements in children's academic skills that generate meaningful improvements in children's life chances. If the improvements in state test scores don't even translate into improvements on other tests, what's the chance that they're going to translate into better jobs or the ability to succeed in college?

The NY Times article has a clear graphic displaying the state test versus NAEP discrepancy. (This is a larger problem that Fordham has dubbed the "proficiency illusion.")

2) The Bloomberg/Klein Children First reforms were not implemented until the fall of 2003. The 2002 scores are not relevant in evaluating the effects of these reforms.

The NYC press release leads by attributing the math gains to the Children's First reforms ("This performance represents a nearly 12 percentage point gain since 2003, when Mayor Michael R. Bloomberg and Joel I. Klein introduced the Children First reforms" - as a sidenote, in this parallel universe, a difference of 11.45 rounds up to 12, not down to 11.)

Suddenly, when it's time to talk about reading, all comparisons are made with respect to 2002. I want to give you the facts: this is because there are no gains when you compare the 2007 scores with the 2003 scores.

3) Whether black-white and Hispanic-white achievement gaps are closing depends on how you look at it.

In his email, Joel Klein wrote, "Even as we're making gains, we're helping to close the achievement gap that separates our African-American and Latino students from our white students." He then displays a figure claiming the achievement gap is closing in every grade level and subject.

Let's take 4th grade math as an example. Viewed through a certain lens, Klein is correct to say that the achievement gap is closing. If you look at the percentage of white students at or above basic in 4th grade math and compare this with the percentage of black students at or above basic, the white/black gap has shrunk 10.4 points (from a difference of 29.5 to 19.1) since 2003.

But if you compare the percent of students at or above proficient, the black/white gap has actually grown from a gap of 30% to a gap of 33%. Hispanic kids have gained slightly on white students (by 2%). On the other hand, the white-Asian, black-Asian, and Hispanic-Asian gaps have all grown significantly. The black-Asian gap in the percentage of students at or above proficient was 35% in 2003; by 2007, it is 45%.

Fourth Grade Math: Percent At or Above Proficient By Race/Ethnicity, 2003-2007

To be clear, all is not lost in NYC. There has been progress in 4th grade math compared to 2003, and we should give credit where credit is due. But to say that NYC is setting the pace for urban school reform based on this performance is a canard. Even with its best results, NYC doesn't look better than many urban districts. For example, the percentage of NYC 4th graders at or above basic in math increased 11% from 2003 to 2007; in Boston it increased 18%; in DC, 13%; in Atlanta, 11%, and in Houston, 10%. And with its 8th grade results, NYC is not making progress. Other urban districts are.

With the exception of the 4th grade math scores, Joel Klein is asking us to take a vow of blind faith. Look at the overall picture, he says, and don't focus so heavily on the details of any one indicator (unless it's the one that shows we're making progress). But when 1.1 million kids' lives are in question, should we be willing to operate on blind faith? When Bloomberg sold us on a candidate who would be accountable for student achievement in the city, did he mean that we should look at the "whole picture?" When Joel Klein said that he was moving the system from a culture of excuses to a culture of results, was there a Barry Bondsian asterisk at the end of his sentence?

Klein, not unlike Bush, is a true believer; he's sure he has the policy toolbox to fix NYC's schools. And I do believe that his commitment to educational equity is genuine. Ironically, it is the strength of this commitment, coupled with his belief that his policies represent the one "true way," that makes him either unwilling or unable to reckon with the hard facts. But at the end of the day, empirical evidence should trump one's fanatical faith in particular policy solutions.

The New York City Dept of Ed has demanded "data-driven decision making" from its educators, but is now asking us to deny the data. NYC leaders see what they want to see, and they expect us to follow suit. Selective cognition, though, is dangerous stuff in policymaking. For the sake of the kids involved, let us hope that those running the Department of Ed will begin to look at all of the evidence and evaluate their policies accordingly.

Overheard in New York

After almost five and a half years as chancellor, I know you can’t point to a single number, be it a test score or graduation rate, to prove success or failure. The whole picture is important.

--Joel Klein

Unless, of course, you're talking about grading schools - where you can just point to a single letter.

In a memo, Klein goes on to say:

We need to look at all the indicators we have—promotion rates, graduation rates, State English and math test results, Regents pass rates, national test results, and even the results of things like Advanced Placement and PSAT exams—to understand how well our students are performing and progressing. Together, these factors paint a complete picture. In that context, I am encouraged by the results we got last week on a key national indicator, the National Assessment of Educational Progress results.

Impression Management, or: How I Learned to Stop Hating Press Releases on Test Scores

Once upon a time in a land far, far away, district leaders did not live and die by their students' test scores. When I peered back into the educational debates of the 1980s and even the early 1990s, I found that the release of state and federal test scores was not an occasion for carefully orchestrated media stunts.

Sure, there was spin then, too - but district leaders were just as likely to decry poor test results to build political momentum to infuse additional resources or make needed changes.

Not so last week, where almost every district publicly releasing scores from the Trial Urban District Assessment spun a fabulous yarn around the NAEP results. Indeed, some of the results were promising. But many of the results that district leaders swaggered about were not. Instead, district leaders cherry-picked the results, up-playing positive news while ignoring or discounting negative news.

I pulled the press releases and local news articles from the TUDA districts, and here's a sample of what I found:

• Chicago and Cleveland - both of which posted lackluster performance - chose the "What TUDA Assessment?" approach. Neither district issued a press release. (However, Cleveland did issue a press release when Cleveland football teams made the championship game.) Barbara Eason-Watkins, Chief Education Officer for the Chicago Public Schools, dismissed the flat scores from 2005 to 2007, noting, "We are encouraged that we have been trending up in all subjects at all grades."
  
  
• Houston leads its press release by saying that "HISD students are beating their peers in most other big-city school districts" - which is true - but buries a key fact and then dismisses it: "4th graders in Houston this year declined in reading, but overall, HISD's scores were up."
  
  
• Perhaps most astounding was New York City's press release, which leads with "New York City students made impressive gains on the 2007 NAEP" - never mind the 4th grade and 8th grade reading results, or the 8th grade math results.

If I sound frustrated, it's because I am. This dynamic is too reminiscent of one of the most potent quotes from Ron Suskind's 2004 NYT Magazine article on the Bush administration, where a senior aide remarked:

We're an empire now, and when we act, we create our own reality. And while you're studying that reality -- judiciously, as you will -- we'll act again, creating other new realities, which you can study too, and that's how things will sort out. We're history's actors . . . and you, all of you, will be left to just study what we do.

To me, it seems that the tone of education policy debates has shifted in the last 25 years. At the center of this shift are the concepts of "impression management" and "reality creation" (see here for one recent example in New York City). Maybe this is a common cultural trend, or perhaps I am re-imagining an idealized past that never existed.

For those who have been in this game longer than I have, please weigh in. On Wednesday, I'll profile one district's response to TUDA in more depth in an attempt to understand the increasing role of impression management in urban education policy.

This week: The Test Score Spin Doctors

Since NCLB, test scores have become the primary currency in education policy debates. From every corner, we are deluged with claims about which direction test scores are moving.

District, state, and federal policymakers now serve as our educational spin doctors, framing test results for the public in order to support their agendas.

In this short week, I'll consider the packaging and repackaging of results from last week's NAEP Trial Urban District Assessment.

Monday: Impression Management, or: How I Learned to Stop Hating Press Releases about Test Scores

Wednesday: A Case Study of One District's Response to NAEP TUDA Results