Eduwonkette: Lies, Damned Lies, and NAEP Exemptions

It turns out that there are four kinds of lies: lies, damned lies, statistics, and NAEP exemptions.

The purpose of the NAEP Trial Urban District Assessment is to "make it possible to compare the performance of students in participating urban school districts to that of public school students in the nation, in large central cities, and to each other." If comparability is the goal, exemption and accommodation provisions must be roughly similar in all districts.

At present, wide variation in exemption and accommodations makes comparing districts an almost impossible task. On the 4th grade reading assessment, some urban districts exempted up to 20% of the total population, while others only exempted 3%. Comparing scores within the same district over time is also a problem - DC exempted 8% of all students from the 4th grade reading assessment in 2002 but 14% in 2007.

Some key problems:

Four districts - Austin (20%), Cleveland (17%), Houston (17%), and DC (14%) exempted more than 10% of all students on the 4th grade reading test. (See Table 1 below.)
Two of the districts that showed exceptional progress - Atlanta and DC - have exhibited large amounts of growth in their exemption rates on the 4th grade reading test. In 2002, Atlanta only exempted 25% of all students with disabilities/ELL; in 2007, it exempted 58%. In 2002, DC exempted 42% of all students with disabilities/ELL; in 2007, it exempted 64%. (See Table 2.)
Accommodation rates also vary widely across districts. New York City gave accommodations to 76% of all students with disabilities/ELL on the 4th grade reading test. Los Angeles only gave accommodations to 13%. (Table 3 below.)
These patterns persist on the math test; there is wide variation in the use of exemptions and accommodations. (See Tables 4 and 5.)

What happened to the "you must test 95% of every subgroup" philosophy? If we're going to invest in these assessments and use them to inform policy, there need to be uniform testing protocols across all districts.

Until then, make these within (i.e. over time) and between district comparisons with care. Hat tip to the New York Sun's Elizabeth Green for breaking this story.

Table 1. Percent of All Students Exempted, 4th Grade Reading: 2002-2007