WWMD on NAEP Exemptions

A-Rus is perplexed over at This Week in Education, and so am I. Why isn't anyone concerned that some districts exempted 20% of the total student population on the NAEP Trial Urban District Assessment?

Elizabeth Green is. A-Rus is. I am. But where are the groups most in favor of keeping ELL/students with disabilities in the accountability system - Ed Trust, Ed Sector, ELL/students with disabilities advocates, and the Secretary of Education - on this? Or groups like Fordham or the NYT that are concerned with having one national measure that allows for comparability across states?

WWMD means "What Will Maggie Do?"

Lies, Damned Lies, and NAEP Exemptions

It turns out that there are four kinds of lies: lies, damned lies, statistics, and NAEP exemptions.

The purpose of the NAEP Trial Urban District Assessment is to "make it possible to compare the performance of students in participating urban school districts to that of public school students in the nation, in large central cities, and to each other." If comparability is the goal, exemption and accommodation provisions must be roughly similar in all districts.

At present, wide variation in exemption and accommodations makes comparing districts an almost impossible task. On the 4th grade reading assessment, some urban districts exempted up to 20% of the total population, while others only exempted 3%. Comparing scores within the same district over time is also a problem - DC exempted 8% of all students from the 4th grade reading assessment in 2002 but 14% in 2007.

Some key problems:

• Four districts - Austin (20%), Cleveland (17%), Houston (17%), and DC (14%) exempted more than 10% of all students on the 4th grade reading test. (See Table 1 below.)
  
  
• Two of the districts that showed exceptional progress - Atlanta and DC - have exhibited large amounts of growth in their exemption rates on the 4th grade reading test. In 2002, Atlanta only exempted 25% of all students with disabilities/ELL; in 2007, it exempted 58%. In 2002, DC exempted 42% of all students with disabilities/ELL; in 2007, it exempted 64%. (See Table 2.)
  
  
• Accommodation rates also vary widely across districts. New York City gave accommodations to 76% of all students with disabilities/ELL on the 4th grade reading test. Los Angeles only gave accommodations to 13%. (Table 3 below.)
  
  
• These patterns persist on the math test; there is wide variation in the use of exemptions and accommodations. (See Tables 4 and 5.)
  

What happened to the "you must test 95% of every subgroup" philosophy? If we're going to invest in these assessments and use them to inform policy, there need to be uniform testing protocols across all districts.

Until then, make these within (i.e. over time) and between district comparisons with care. Hat tip to the New York Sun's Elizabeth Green for breaking this story.

Table 1. Percent of All Students Exempted, 4th Grade Reading: 2002-2007

Table 2. Percent of Students with Disabilities/English Language Learners Exempted, 4th Grade Reading: 2002-2007

Table 3. Percent of Students with Disabilities/English Language Learners Receiving Accommodations, 4th Grade Reading: 2002-2007

Table 4. Percent of Students with Disabilities/English Language Learners Exempted, 4th Grade Math: 2003-2007

Table 5. Percent of Students with Disabilities/English Language Learners Receiving Accommodations, 4th Grade Math: 2003-2007

NCLB III: Testing Special Education Kids

Part 3 of a series on NCLB. Links to previous posts available here.
One of the biggest complaints about NCLB is that only 2% of students (essentially, up to 20% of all students with disabilities) can take modified assessments. Here's the debate in brief - those concerned with extending modified assessments worry that special ed kids will be overlooked if the standards are different, while others worry that these kids - who are classified as special education in many cases precisely because they aren't working on grade level - are being asked to do something that's educationally and developmentally inappropriate.

What do we know from states' experience with special education exemptions? David Figlio and Lawrence Getzler, looking at data from Florida, found that such loopholes led educators to hide low-scoring kids in special education. Julie Cullen and Randy Reback, analyzing data from Texas, found similar processes in play. Neither of these states had strong participation rules, so reclassification is not as big an issue under NCLB as it has been in the past. However, rational choice theory would predict that if teachers face high pressure to push some students but not others forward, these kids may very well be neglected. I can't stress enough how ridiculous it is blame educators for responding to the incentives presented; it's no different than arguing that guns don't kill people - people kill people. As I discussed on Monday, a system that incentivizes teaching some kids (those close to passing) over others will inevitably produce this result, and the same goes for special ed incentives.

On the other hand, we can imagine that there are potentially negative social effects of the current approach (i.e. all kids on grade level, no matter what) on kids facing tasks they're simply not prepared for. A new book instructive in thinking through this issue is The Short Bus: A Journey Beyond Normal by Jonathan Mooney. He's a young and fresh voice in the literature on learning disabilities, and is an entertaining read. Check out the video clip at his website above, and here's the blurb:

When his teachers decided Jonathan Mooney needed special ed because he couldn’t follow directions, sit still, or read well, he feared he’d lost his chance to be a regular kid. Suddenly he was “not normal.” Suddenly he was a short-bus rider destined to travel a harder road, a distinction that screamed out his “difference” to a hostile world. Along with other kids facing similar challenges, he was denigrated daily....ultimately, Jon shocked the skeptics, graduating from Brown University (with honors). But he could never shake the voice that insisted he would always be “less than.” So he hit the road. To free himself, he dreamed up an epic journey across the U.S. on a broken-down short bus. This inspiring record of his odyssey documents Mooney’s search to help himself by learning from others once labeled abnormal who had learned to live in beautifully original ways.

My take-home point: special ed kids are going to get a raw deal in terms of academic attention if they aren't incorporated into the accountability system. I worry about the social consequences, too - but think that teachers' social and emotional support can in many ways alleviate this problem.

That said, these comments should be taken in the context of my earlier comments about measurement. As I've written before, it doesn't make sense to say that schools are failing when they can't get all kids to proficiency, special ed or otherwise, because a) it's an unreasonable goal, and b) they are not all working with the same inputs. All of the points that I made about measuring teacher effectiveness fairly also apply to schools, too.

Enjoy the weekend, everyone!