NCLB III: Testing Special Education Kids

Part 3 of a series on NCLB. Links to previous posts available here.
One of the biggest complaints about NCLB is that only 2% of students (essentially, up to 20% of all students with disabilities) can take modified assessments. Here's the debate in brief - those concerned with extending modified assessments worry that special ed kids will be overlooked if the standards are different, while others worry that these kids - who are classified as special education in many cases precisely because they aren't working on grade level - are being asked to do something that's educationally and developmentally inappropriate.

What do we know from states' experience with special education exemptions? David Figlio and Lawrence Getzler, looking at data from Florida, found that such loopholes led educators to hide low-scoring kids in special education. Julie Cullen and Randy Reback, analyzing data from Texas, found similar processes in play. Neither of these states had strong participation rules, so reclassification is not as big an issue under NCLB as it has been in the past. However, rational choice theory would predict that if teachers face high pressure to push some students but not others forward, these kids may very well be neglected. I can't stress enough how ridiculous it is blame educators for responding to the incentives presented; it's no different than arguing that guns don't kill people - people kill people. As I discussed on Monday, a system that incentivizes teaching some kids (those close to passing) over others will inevitably produce this result, and the same goes for special ed incentives.

On the other hand, we can imagine that there are potentially negative social effects of the current approach (i.e. all kids on grade level, no matter what) on kids facing tasks they're simply not prepared for. A new book instructive in thinking through this issue is The Short Bus: A Journey Beyond Normal by Jonathan Mooney. He's a young and fresh voice in the literature on learning disabilities, and is an entertaining read. Check out the video clip at his website above, and here's the blurb:

When his teachers decided Jonathan Mooney needed special ed because he couldn’t follow directions, sit still, or read well, he feared he’d lost his chance to be a regular kid. Suddenly he was “not normal.” Suddenly he was a short-bus rider destined to travel a harder road, a distinction that screamed out his “difference” to a hostile world. Along with other kids facing similar challenges, he was denigrated daily....ultimately, Jon shocked the skeptics, graduating from Brown University (with honors). But he could never shake the voice that insisted he would always be “less than.” So he hit the road. To free himself, he dreamed up an epic journey across the U.S. on a broken-down short bus. This inspiring record of his odyssey documents Mooney’s search to help himself by learning from others once labeled abnormal who had learned to live in beautifully original ways.

My take-home point: special ed kids are going to get a raw deal in terms of academic attention if they aren't incorporated into the accountability system. I worry about the social consequences, too - but think that teachers' social and emotional support can in many ways alleviate this problem.

That said, these comments should be taken in the context of my earlier comments about measurement. As I've written before, it doesn't make sense to say that schools are failing when they can't get all kids to proficiency, special ed or otherwise, because a) it's an unreasonable goal, and b) they are not all working with the same inputs. All of the points that I made about measuring teacher effectiveness fairly also apply to schools, too.

Enjoy the weekend, everyone!

Thoughts on Teach for America

Lots of edubloggers have weighed in on the Sunday Times article on Teach for America – see Eduwonk, Ms. Frizzle, and Teaching in the 408 - and I will throw in my two cents as well. There are at least five constituencies affected by TFA, and we need to think about TFA’s effects on each of them. Apologies in advance for the crazy long post.

While reading, keep in mind what the counterfactual is; in other words, if TFA didn’t exist, what might the world look like? Spoiler alert: my conclusion is that we are better off in a world where TFA exists. However, contrary to the sweeping arguments made by TFA and its supporters, TFA is in no way “closing the achievement gap” or changing the face of urban education. Part 2, on Monday, will focus on the dangers of these capacious claims and focus - from my own perspective - on what you know, and what you don't know, after teaching for two years.

1) The students corps members teach: First and second year teachers are not at the top of their game, and we all know it. But I have seen little convincing evidence that TFA teachers are much worse at promoting kids’ academic achievement than other new teachers; see, for example, this EdNext summary of the Kane, Staiger, and Rockoff study on New York City, or the Mathematica study. When compared to other new teachers, principals are happy with TFAers as well; according to the Times article, 74% considered TFA teachers to be better than beginning teachers. Non-TFA beginning teachers seem to be the right comparison group here, as in the world where TFA never existed, these kids (absent some substantial policy interventions) would be taught by inexperienced teachers.

That said, the notion that the two-year classroom component of TFA is even part of the answer to closing the achievement gap - a claim that is tossed around loosely at campus recruiting - is not supported by the existing evidence. Corps members do no harm to the students they serve, but at least in their first two years in the classroom, they’re not revolutionizing education for disadvantaged kids, either.

2) The schools in which corps members teach: One argument is that TFA destabilizes schools by providing a permanent revolving army of inexperienced teachers. In the Times article, a principal relates that she has to constantly retrain teachers. Again, it’s my sense that if TFA had never been founded, these schools would be staffed by inexperienced teachers who teach, on average, for a couple of years before moving on.

3) The corps members themselves: The Times article reviewed the “second half of the movement,” i.e. to staff multiple fields with former teachers who understand the realities of the classroom. Kopp made the following comment in the article: “In order to have a real impact…you will have to influence the priorities of a generation. We are completely redirecting students’ paths.” Supporters point to the accomplishments of core members such as Michelle Rhee and the founders of KIPP as evidence of TFA’s broader impact on education reform.

For this to be true, we need to believe that core members who stayed in education and garnered a lot of attention (i.e. Michelle Rhee, the founders of KIPP) would not be involved in urban education reform had they not been corps members. But if you’ve talked to a corps member recently, you know that these are young people with strong social justice commitments; a lot of the people who stayed would still be working in urban, education, and social policy in one way or another. Again, we have a comparison group problem; we need to compare people who were selected for TFA and joined with those who were selected and declined. That some former corps members hold key education policy roles is not evidence of TFA’s success; it’s very possible that they would have been there anyway. On the other hand, we might expect big effects on those who were unlikely candidates for TFA - i.e. not kids who spent all four years of college tutoring in urban schools - but ended up there for one reason or another.

To my knowledge, only one study has attempted to make this comparison. Doug McAdam, a professor of sociology at Stanford University, recently surveyed TFA applicants who became corps members, those who became corps members and did not serve the full two years, and those who were selected but did not accept the offer. McAdam is the right person to do the study – he wrote a phenomenal book on the long-term impact of Freedom Summer involvement on its participants, comparing those who applied and went with those who applied but didn’t go – and found long-term effects on the life paths of those involved. McAdam’s study is ongoing, so we’ll have to wait a while for the answer to this question.

Nice op-ed on the second part of the movement in Ed Week by Michael J. Salmonowicz, a former corps member.

4) The profession of teaching: An expressed goal of TFA was to elevate the status of teaching – to make teaching with TFA the equivalent of winning a Rhodes. It’s clear that the status of the profession of teaching affects who goes into teaching and stays in the future. So framing teaching as a self-effacing, heroic act of service, as opposed to a profession, may actually work against that goal. There was a terrific NYT op-ed by Tom Moore, a Bronx teacher, a while back about the hero dilemma, which pointed out the problem with this mythology:

Films like "Freedom Writers" portray teachers more as missionaries than professionals, eager to give up their lives and comfort for the benefit of others, without need of compensation. Ms. Gruwell sacrifices money, time and even her marriage for her job. Her behavior is not represented as obsessive or self-destructive, but driven — necessary, even....The film applauds Ms. Gruwell's dedication, but also implies that she has no other choice. In order to be a good teacher, she has to be a hero. "Freedom Writers"...is presented as a celebration of teaching, but its message is that poor students need only love, idealism and martyrdom.

Every day teachers are blamed for what the system they're just a part of doesn't provide: safe, adequately staffed schools with the highest expectations for all students. But that's not something one maverick teacher, no matter how idealistic, perky or self-sacrificing, can accomplish.

On the other hand, professions do gain status when they are selective. Whether you can elevate the status of teaching, which has an enduring status problem, by making it selective is an empirical question, and one we won’t know the answer to for some time. I have very mixed feelings about this. On one hand, we know that since women’s opportunities opened up, the quality of the teaching force has declined. For example, Sean Corcoran, Bill Evans, and Robert Schwab found that in the 1960s, 40% of women in the top 10% of their high school classes went into teaching. By 1996, less than 10% of women in the top 10% went into teaching. So the problem is real. (See paper here.)

But, do we want people who see selectivity as a plus teaching? (See commentary by a college senior on making teaching a status symbol here.) Do we end up encouraging young people to “do well by doing good?” Is this good for kids? I don't know, but these are important questions.

5) The public: TFA gets a lot of media attention, and what gets said affects what the average citizen thinks about the problems of urban education. Does TFA give the message that we just need “smart” people in front of the classroom? That urban kids are behind only because what happens in school? That heroes alone can close the achievement gap?

To sum up, I don’t see evidence of harm done to the kids and schools themselves, but I continue to wonder about TFA’s long-term effects on the teaching profession. Stay tuned for more on this issue on Monday.

No Child Left Behind II: Collateral Damage

Part 2 of 3 on NCLB. On Monday, I focused on what we gain through NCLB. What's clear is that the accountability systems most studied - Texas, North Carolina, and Chicago - have increased test scores. Whether these increases translate into meaningful improvements in the life chances of children is unknown. Nonetheless, for those who equate achievement with test scores, accountability systems are very successful.

But my theme this week is policy trade-offs; that is, all policies have costs and benefits. What's clear is that accountability systems in the past, as well as NCLB, have changed the business of school on a day-to-day basis, often in ways that belie the stated purposes of these policies - i.e., to improve the overall quality of education that students receive, with a focus on increasing opportunities for disadvantaged kids. Many have made these points more eloquently than I can, so see also Jim Horn's comprehensive posting about what is left behind, as well as Debbie Meier and Diane Ravitch's on-going exchange at "Bridging Differences." But to summarize, here's what we're losing:

1) "Hey Mom! What's a tundra?": What about non-tested subjects?

The Center for Education Policy reports that 44 percent of school districts in the country have made substantial cutbacks in social studies, science, art, and music lessons in elementary school.

2) "Hey Mom! What do you mean there's no Halloween parade?": The value of "pointless" kid stuff

Maybe I'm sentimental, but is it so bad to have Halloween parades and Valentine's Day parties and trips to watch penguins at the zoo? As an inducement, I've offered Madame Secretary a peek into this alternate future, where she can ride a My Little Pony for Halloween.

For just a couple of days a year, can we have no objectives on the board, no SWBAT (students will be able to), no Grant Wiggins backwards design, no authentic assessment, and no Do-Nows? I'm not suggesting we let the childrens run wild. And of course I understand why principals feel like schools don't have time to just muck around, even once in a while. But maybe a tiny bit would keep everyone sane, students and teachers alike.

3) What about non-tested competencies we care about?

Teaching kids to do well on a test is not the same thing as teaching them to be critical thinkers, good scientists, and creative mathematicians. Lest I be maligned as an ed school hugging-hippie-commie who should just go back to her drum circle, let me give a very concrete example. Teaching kids how to do science - how to design an experiment to test a hypothesis - is not the same thing as teaching them the details of photosynthesis for a multiple choice exam. Surely kids need basic skills, but I am not convinced by the temporal ordering demanded by NCLB - i.e. "first things first." Diane Ravitch says it better:

If youngsters, in large numbers, have not learned and cannot use the basic skills, they are not likely to be prepared to be thinking citizens of our democracy. Thinking citizens need the tools and the power of reading and math, and they need the skills and knowledge of science and history so as to contribute to our common project as a democracy.

4) What happened to the idea of deliberative discussion about the goals of education?

It's increasingly difficult to move forward a serious discussion of what schools are for. It's a foregone conclusion in many circles that schools are for increasing test scores, period. Again, Diane Ravitch provides a succinct explanation of this problem:

It is educators who are being pushed aside, as businessmen, lawyers, MBAs, and other organization men and women move in to rationalize education and run it like a business....The business leaders think that the problems of education are all managerial; they belittle the importance of curriculum and instruction. They don’t understand anything about the civic purpose of education. And right now, they have the upper hand.

Reader comments: NCLB led to the Dean scream

Last week, to make the point that attributing NAEP scores to NCLB (in either direction) has little validity, I posted a tongue in cheek list of things that NCLB caused, including the iPhone, the rise of crystal meth, and Hannah Montana (reader Seth berated me for alarming his officemates by linking to a Disney page that blared Queen Montana - sorry, Seth!)

I invited readers to chime in, and the list below is the result of your ideas. For serious news and analysis, here are some comments on NAEP from the entire spectrum of observers, including Ed Week, Diane Ravitch, Donna Garner on NAEP exclusions by state (for 4th grade, CA excluded only 4% of students, while Texas excluded 10% - I had no idea there was that much variation), the Quick and the Ed, EdTrust, and FairTest.

My favorite reader comment was from Patrick - NCLB caused the Dean scream, as well as:

• the falling price of plasma TVs
• the closing down of Oldsmobile
• Hurricane Katrina and the Iraq war (my contribution: Bush is the latent variable - i.e. he caused both as well as NCLB, which then led to the iPhone)
• opposition to the Iraq war
• Jennifer Aniston & Brad Pitt's divorce
• Britney Spears & Kevin Federline's marriage
• Lindsay Lohan's downfall from Parent Trap to rehab
  
• Justin Timberlake's "bringing sexy back"
  
• increased access to internet via broadband
• the dominance of the New England Patriots (via cheating methods similar to those reported under NCLB)
• increased verbal content in entertainment
  
• ability to catch perpetrators who sent anthrax through the mail in the U.S. (reader "lyndon johnson" writes: "Oh, wait - what? Really? Still? It's been five years... Oh, never mind. Moving on....")
  
• Hilary Swank's performance in Million Dollar Baby
• children becoming increasingly bored and fat
• reader Helen in California's weight gain
• Dance Dance Revolution
• Grom (anonymous reader says this is, "yummy gelato in New York")
• "Dancing with the Stars"
• a rocking chair to fall asleep in during class
  
• dirty rugs full of fleas to keep the children awake (reader proofoflife - next time I lecture, I am going to use this idea to keep Gen Friendster itching for more)
  

Cool people you should know: Tom Corcoran

Cool people you should know returns.

Tom Corcoran is Co-Director of the Center for Policy Research in Education. I once heard him described as "every researcher's favorite practitioner, and every practitioner's favorite researcher," and that pretty much sums it up. He understands the daily rhythms of teachers' lives and gets the ways that schools work.

Fun facts: Corcoran served as the policy advisor for education for Governor Florio of New Jersey (and survived the politics of the Quality Education Act that pre-dated the Abbott v. Burke decision) and director of evaluation and later chief-of-staff of the New Jersey Department of Education.

Here is a link to some of his recent publications. My top pick is an older brief on professional development, "Helping Teachers Teach Well" (1995). His "best practices for professional development" are telling about the ways that our treatment of teachers has changed over the last 12 years:

"A number of experts and organizations have suggested that the most promising professional development programs or policies are those that:

• support teacher initiatives as well as school or district initiatives. These initiatives could promote the professionalization of teaching and may be cost-effective ways to engage more teachers in serious professional development activities.
  
• are grounded in knowledge about teaching. Good professional development should encompass expectations educators hold for students, child development theory, curriculum content and design...
• demonstrate respect for teachers as professionals and as adult learners. Professional development should draw on the expertise of teachers and take differing degrees of teacher experience into account."
  

A Bushism for education policy: Attack the idea, not the idea haver

My proposal for education policy wonks and wonkettes: let's attack the ideas, not the people who had them. Why? For one, it's intellectually lazy. If you have beef with someone's claims or proposals, it should be easy to identify why they're wrong. Second, it makes for sloppy conversation. Using people's names as cognitive shortcuts for a broad set of ideas results in our not knowing to what exactly the speaker objects.

A prime example of this problem is the reaction to Jonathan Kozol's partial fast, which is Kozol's attempt to draw attention to what he sees as the injustices of NCLB. Rather than engaging with Kozol on what is wrong with the ideas he's putting forth, some outlets have taken this opportunity to vitriolically attack him, using his name as a proxy for the big, bad opponents of "real reform." (Image above courtesy of Education Next.)

I can understand that some may disagree with Kozol about, for example, the effects of NCLB on disadvantaged children, the utility of a hunger strike to address these issues, or any number of the other arguments he's advanced since 1964. It's not my opinion, but I get where his opponents could disagree. But rather than initiating an intelligent conversation about what's wrong with his argument, we've seen a series of posts that go after him. (Sidenote: this is ironic coming from those who support TFA because of its "effect on the conscience of a generation." No other writer has drawn more attention to poor and minority kids than Kozol.)

To be fair, I am as guilty of this cerebral sloppiness as anyone else. When I first heard about the Fordham Fellows and its associated blog, my vision was of wonks and wonkettes, hunched over Hpnotic martinis at Old Ebbitt Grill, scheming to marketize public education. So I was pleasantly surprised when I started reading the posts of the Fordham Fellows, each of whom has something thoughtful and unique to say about education reform.

From Gabrielle Capone's painfully honest essay about how hard school reform is, to Zach Blattner's thoughts on how "reading sometimes sucks" because we make kids read boring passages all the time; Maya Wallace's grasp for a creative policy solution to give kids more time with adults without killing the teachers themselves to Cait Ferrell's thoughts on race in America; Liam Honigsberg's proposal on how to measure teacher effectiveness to Kate Sullivan's insight on whether separate can ever truly be equal, and Cecilia Le's reflections on bipartisanship, these are folks who are really thinking. You should check out their blog here.

Don't forget to read these books!

I've nominated four of this fall's best academic books on education for the first (quasi-recurring) eduwonkette gold star award. Read more about these books here. Award to be announced on October 15th.

• Mitchell Stevens, Creating a Class: College Admissions and the Education of Elites
• Jack Buckley and Mark Schneider, Charter Schools - Hope or Hype?
• Bruce Fuller, Standaridzed Childhoods: The Political and Cultural Struggle Over Early Education
• David Kirp, The Sandbox Investment: The Preschool Movement and Kids-First Politics
  

NCLB I: Do accountability systems improve test scores, and for whom?

Part 1 of 3 of "NCLB glass half full (....with poison?)" Here are my thoughts on three questions related to the title of this post:

1) Do accountability systems improve test scores (i.e. percentage of students passing state tests)?

Despite both supporters' and opponents' eagerness to tell us how well NCLB is or isn't working based on limited data, the cleanest evidence of accountability systems' effects pre-dates NCLB. This earlier generation of studies examined data from Texas, North Carolina, and Chicago, and found, unsurprisingly, that accountability systems do substantially increase achievement on the state tests themselves. Other studies have looked at the effects of accountability systems on non-state tests. Two studies in particular, comparing NAEP progress in accountability and non-accountability states (again, all pre-NCLB), found that states with public monitoring (in Hanushek and Raymond's 2004 study) or those with stronger accountability policies (in Carnoy and Loeb's 2002 study (available here, #9 under articles) exhibited greater gains. But the gains on state tests consistently outpace those on NAEP, which some have called evidence of test-score inflation. See a recent paper by Brian Jacob here that addresses this issue in Texas.

So it's fair to say that accountability systems do increase test scores, which is different than saying they improve academic achievement. Remember, the test is not an end in itself, but a proxy for the real thing we care about - students' academic skills. The real question, then, is whether these increases translate to other venues in ways that improve students' life chances over the long haul.

2) Do increases in state tests scores translate into improvements in children's academic skills that generate meaningful improvements in children's life chances?

This question is the crux of the accountability issue, and we unfortunately don't know much about this question - though again, lots of people on both sides claim to. (If I am wrong, please email me references!) Put differently, are kids graduating from high school in Texas and North Carolina, who have gone through the better part of their K-12 educations under NCLB-like accountability systems, more likely to graduate from high school, go to college, and graduate from college than they would been otherwise? Are they more productive workers as a result? Better citizens?

3) For whom do accountability systems improve test scores, and what are implications of maintaining a "proficiency for all" target?

Proficiency-based accountability systems can lead educators to focus on kids close to the cut score for proficiency. These kids have been assigned a variety of titles, including marginal kids, cusp kids, or bubble kids (see my post here about Derek Neal's Chicago study). Since sanctions are doled out based on passing rates, slightly increasing the scores of a small number of students can help a school make AYP. Such a focus could affect the achievement of both high and low-scoring students. The irony is that in states where the bar for passing is higher, it is more likely that very low-performing kids will lose out under accountability systems, as they are likely to be well below passing.

What about the argument that with 100% proficiency, schools can't pursue this strategy forever? When people face impossible tasks, they focus too heavily on the short-term incentives and not at all on the long-term incentives. The short-term incentive here is to push one more kid over the passing mark. Yes, I am arguing, unapologetically, that 100% proficiency is impossible, that most educators would second that assertion, and that almost all researchers who study school effects would argue this is impossible as well, absent a dramatic change in how American schools, or American society for that matter, do business. The rate of improvement required for 100% proficiency would require schools to improve performance at a rate unlike anything even the top 5% of American schools have ever done. This is the jogger's equivalent of asking an eight minute miler to rev it up to a five and a half minute mile; that someone else can do it is not evidence that everyone can.

Teachers and principals are doing what they can to push forward, but if we stay at a 100% proficiency target, even with growth models, I fear that teachers are going to continue to attend to those close to passing first. This is all a way of answering the "for whom" question: kids at the tails of the distribution - low and high achievers - aren't served well by NCLB's current focus on proficiency. Another response to this is, "Well, it's not NCLB's fault that those bad news bears teachers are doing that." If you believe this is true, you may be surprised to learn that you also agree with the argument that guns don't kill people, people kill people. It's no different.

To sum up - and there's something here for both sides, I think - accountability systems do increase test scores, but they do so most for the kids closer to passing.

Policy tradeoffs: NCLB glass half full (....with poison?)

This week I will talk about policy tradeoffs. While an obvious point - that is, all policies have costs and benefits - it is too often ignored in the NCLB reauthorization debate. Supporters of NCLB espouse kumbaya arguments, as if NCLB has only positive consequences for American schools. On the other hand, opponents' arguments suggest that NCLB is killing our schools and eating our children. I want to argue that the NCLB glass may be both half full and half empty, depending on the outcome in question and for whom the outcome is considered. In part, this disagreement is not about the effects of the law, but about what the goals of public schools should be, and the weight these competing goals should receive. Here's the overview:

Monday, Part I: Do accountability systems increase reading and math test scores, and if so, for whom?

Wednesday, Part II: What is the collateral damage of a "laser-like focus" on math and reading scores? As Jim Horn nicely put it, what gets left behind by No Child Left Behind?

Friday, Part III: How should we handle populations for whom grade-level English-only tests may not be appropriate, i.e. English language learners and special education students? What are the implications of testing, or not testing, these children?