A-Rus is perplexed over at This Week in Education, and so am I. Why isn't anyone concerned that some districts exempted 20% of the total student population on the NAEP Trial Urban District Assessment?Elizabeth Green is. A-Rus is. I am. But where are the groups most in favor of keeping ELL/students with disabilities in the accountability system - Ed Trust, Ed Sector, ELL/students with disabilities advocates, and the Secretary of Education - on this? Or groups like Fordham or the NYT that are concerned with having one national measure that allows for comparability across states?
WWMD means "What Will Maggie Do?"
9 comments:
Have you looked carefully at the NAEP inclusion policy? http://nces.ed.gov/nationsreportcard/about/inclusion.asp
The intention is to include as many students with disabilities or ELL in the assessments as possible, and to give local school staff discretion over the accommodations that might be appropriate. Agreed that the % excluded rises above 5% in some cities in 2007, especially for 4th- & 8th-grade reading, which is inconsistent with the goal of maximizing participation of students with disabilities or ELL. But participation in NAEP is *voluntary*, and the feds have had increasing difficulty in getting schools to agree to participate -- for lots of reasons. Robbing school personnel of the discretion to determine who should be excluded, and what kinds of accommodations are appropriate, would only lower the response rates. The TUDA documentation addresses the exclusion issue:
"Some students identified as SD or ELL who are sampled for NAEP participation may be excluded from
the assessment if NAEP does not offer the accommodations given on the student’s state assessment. School personnel, guided by the student’s Individualized Education Program (IEP) as well as by Section 504 eligibility, make decisions regarding inclusion in the assessment of students with disabilities. Based on NAEP’s guidelines, they also make the decision whether to exclude students identified as ELL. The percentages of students excluded from NAEP may vary considerably across districts and over time.
Comparisons of achievement results across districts should be interpreted with caution if the exclusion rates vary widely. See appendix tables A-2 and A-3 for
the exclusion rates in the urban districts."
hi anonymous 3:39,
i agree that districts are acting within the boundaries of the exclusion/accommodation policy, and acknowledge that the voluntary nature of NAEP makes this difficult. this is my point, though:
Comparisons of achievement results across districts should be interpreted with caution if the exclusion rates vary widely.
If we can't compare across districts or compare the performance of the same district over time, why are we administering this assessment at all?
Anon 3:39 again ...
The NAEP Trial Urban District Assessment is, after all, a trial, funded through NCLB funds. At some point, there should be an evaluation of it, and the issue of exclusions ought to be considered. But you are right to worry about why it is we're doing this at all, because as early as 2003, there was evidence of differential exclusion of students with disabilities and English Language Learners in the TUDA. Ed Haertel wrote a paper showing that differential inclusion rates had big consequences for the relative rankings of districts participating in TUDA. (See his paper, and others on the same topic, on the NAGB website.) So if this was already identified as a problem as early as 2003, why hasn't there been an effort to address it? Seems like you ought to direct this question to NAGB and NCES.
Anon 3:39 yet again ...
You might note that this problem is especially acute with the Trial Urban District Assessment, and not nearly as problematic for the overall NAEP and state-to-state comparisons. In 2007, the rates of exclusion for Grade 4 math at the state level ranged from 1% to 6%, and for Grade 8 math 2% to 9%. I'd like to see these all under 5%, but they're not too bad. The situation is worse for reading, where the state-level rates of exclusion range from 2% to 12% for Grade 4 reading, and 2% to 9% for Grade 8 reading. At the national level, the percentages of students with disabilities and ELL students excluded in NAEP reading and math have been very consistent across time since the onset of reporting.
Hi Anon 3:39,
Thanks so much for the Haertel ref. And that's good news about the regular NAEP - I'll check out those tables.
Do you thoughts on why the TUDA exclusion rates are so much higher? One idea is that the lower the unit of analysis (i.e. district versus state reporting) the more gaming one would expect. I doubt individual district officials or schools feel an obligation to pump up the state numbers, but they might feel differently about the district numbers.
Hmm. Not sure that gaming is the main dynamic here. It's clear that the rates of identified students with disabilities and ELLs are much higher in the urban districts than in the states in which the urban districts are located. Might it simply be that large urban districts are more likely to have specialized administrative units dedicated to identification and accommodation of students with disabilities and ELLs? IEPs are socially constructed, and they may look different (i.e., call for more exclusions and accommodations) in settings with elaborate bureaucracies to manage them than in districts where there are relatively few students with disabilities or ELLs.
I don't much about the sampling of TUDA schools, but it's clear that the TUDA sample is an augmentation of the state sample of schools and students in the regular NAEP. Although we know that some districts are actively engaged in attempting to "game the system," I don't know what social process would result in "pumping up the numbers" in this case.
No one is going to want to discuss accommodations / exemptions / participation rates on NAEP tests. That would only lead to greater scrutiny. Given your adept analysis of statistics, go get the 2007 NAEP Grade 4 and 8 results in Reading and Math. Compare the average NAEP scores, pass rates, and accommodation / exemption rates and ask yourself if the state by state comparisons are valid.
Would you be surprised to know that a state like Massachusetts that posted some of the highest scaled scores also had some of the highest exemption / accommodation rates? See NCES documents 2007-494 for Math and 2007-496 for Reading for the data. Both reports list the same tables. Tables 6 and 11 show the average scaled scores by states, table A-3 shows the exclusions / accommodations, and tables A-12 and A-19 show the state-by-state differences in scaled scores by students with and without disabilities.
From these data, you will see that Massachusetts posted some of the very highest average NAEP scores in 2007, but excluded at least as many students as the national average exemption and, in some cases, such as Grade 8 Reading, Massachusetts had the highest exemption rate of any state in the nation. Look at Tables A-12 and A-19 and you will see that the students without disabilities scored about 20-40 points more than those without, so the difference is quite large. You can imagine a few percentage points difference in participation rates could really skew the results.
Is it possible that Massachusetts just does a better job of identifying students with these challenges? Perhaps, but from Tables A-12 and A-19, you can see that students with disabilities in Massachusetts have an average scaled score higher than students without disabilities in many other states. Does that say something about those states or Massachusetts? One last point of reference is worth noting. In Grade 4 Math in 2007, students with disabilities in Massachusetts earned an average scaled score of 238. The national average was 239. Are we really to believe that the average student with disabilities in Massachusetts is just as smart as the average student in the nation? Perhaps. I guess. Possibly.
Think of how many people rely on NAEP numbers to make a political point. How many of them are prepared to discuss this?
CORRECTION!
...Massachusetts posted some of the very highest average NAEP scores in 2007, but excluded at least as many students as the national average exemption and, in some cases, such as GRADE 8 MATH (NOT READING), Massachusetts had the highest exemption rate of any state in the nation....
Massachusetts' exemptions are at or above the national average in all four grade/subject combinations. In Grade 8 Math, the excluded rate is 9%, the highest of any state, and more than twice the national average of 4%
Apologies to eduwonkette.
hi anon 6:02,
now i'm convinced i should look at the regular NAEP exemptions - i'll let you know what i find.
Post a Comment