Who's Afraid of Educational Triage?

Last week, the blogs were abuzz with word that the bubble kids are dead! See Swift and Changeable and Eduwonk for a taste. At the same time, NYC has explicitly designed an accountability system that is intended to eliminate triage (at least on the lowest performing kids) by focusing on growth and giving more weight to the lowest scoring kids' progress. What should we make of these competing claims?

Here's the basic idea behind triage: proficiency-based systems create short-run incentives to get some kids to pass this year, so the educational triage hypothesis is that schools focus on "bubble kids" and not high and low performers.

But there are two general scenarios in which we would not expect to see triage - 1) when resources are flush or 2) when there are few kids failing - so I thought the Education Next study by Matthew Springer of Peabody might have stumbled on these two conditions. I've never believed universal triage claims; we should expect to see triage in some schools and not others. The problem, of course, is that the schools NCLB is most concerned about - those with the most disadvantaged kids - are likely to have large numbers of students well below failing if the standards are high enough.

There are two take-home lessons for educational research that apply beyond this EdNext article:

1) Education Next is a magazine published by the Hoover Institution, not a scholarly journal.

This comment is not intended to sound cranky, but - Education Next is not a scholarly journal because scholarly journals a) provide enough information about the study to enable one to fully evaluate the methods used by the author , and b) pick their editorial boards based on the scholarly expertise, not adherence to a particular reform agenda. To be clear, a number of exceptional scholars are involved as editors and editorial board members of EdNext - but those who don't fully adhere to the particular angle provided by EdNext are glaringly absent; to name a few that study topics covered in EdNext, Alan Krueger, James Heckman, David Card, Charlie Clotfelter, Sunny Ladd, Ceci Rouse, Brian Jacob, Julie Cullen, Jesse Rothstein, Derek Neal, Bill Evans, Susanna Loeb, etc. To say "Education Next partakes of no program, campaign, or ideology. It goes where the evidence points" is just not true, and is no different than saying Fox News is fair and balanced. That said, I'm glad they're around for debate's sake and I give their articles to my students occasionally.

2) This is a "duh" point, but....that research has been published does not mean that it's true.

From reading the Economics of Education Review paper, I concluded that the study lacks a design that would enable one to identify the presence or absence of educational triage. Why?

1) The paper lacks a pre-NCLB measure: Derek Neal, who studied this issue in Chicago, identified a focus on marginal kids by examining the distribution of achievement before and after an accountability system was put into place; it is through this strategy that one learns that the distribution of achievement changes in response to incentives. See his paper here.

2) The paper doesn't identify meaningful variation in incentives to act strategically: If one doesn't have a pre-NCLB measure, some fancy statistical gymnastics are required. An exemplar is Randy Reback's elegant paper using data from Texas. Rather than simply examining the "didn't make AYP last year/did may AYP" dichotomy that Springer uses, he explicitly calculates schools' short-run incentives to improve the performance of various students at the school. What does he find?

• Schools respond to math performance incentives both by targeting math resources towards specific students and by making broad changes which also help very low achieving students. These responses tend to sacrifice the targeted students’ reading performance and to sacrifice relatively high achieving students’ performance in both math and reading.


• Schools respond to reading performance incentives by targeting resources towards the reading performance of particular students, sacrificing these students’ math performance and sacrificing all other students’ performance in reading.
  
• Finally, schools devote fewer resources towards students in the terminal grades during years when short-run incentives are low than during years when incentives are high.

In contrast to Springer, Reback concluded:

If one of the primary goals is to create a sort of educational triage, in which students below minimum grade-level skills are pushed up, then the No Child Left Behind type of accountability system appears to be fairly effective. If accountability systems are not intended to induce schools to shift resources disproportionately towards certain types of students, then these systems should use test results to formulate school ratings that do not simply reflect the fraction of students achieving minimum competency.

Similarly, Derek Neal concluded:

Based on our results, it is reasonable to conjecture that hundreds of thousands of academically disadvantaged students in large cities are currently being left behind because the use of proficiency counts in NCLB does not provide strong incentives for schools to direct more attention toward them.

Since we have two large-scale studies in hand that find educational triage in full swing, at least in some schools, it's something we need to take seriously. For all of the NYC's report card flaws (and there are many), the Department of Education deserves credit for focusing on growth rather than proficiency, and for giving additional weight to the lowest performing kids' progress.