Thoughts on NYC's Performance Pay Plan

Now that we've gone over performance pay assumptions and how performance pay works in other sectors, what about the NYC plan?

The refrain we've heard this week from the UFT and other observers is that this is a collective plan that, as Leo Casey wrote, "transformed a negative into a positive, and 'shut the door' on individual merit pay programs." Echoing this sentiment in the NY Daily News, Diane Ravitch wrote, "Score one for this union."

I've already written about the likely impacts of financial incentives on collaboration (see Tuesday and Thursday's posts). To recapitulate, the argument that school-based incentives address collaboration problems fundamentally misunderstands the reasons why collaboration doesn't happen.

However, if we did believe that school-based incentives spur collaboration, existing accountability systems, both state and federal, should have already taken care of this. That your school will lose students, control over how funds are allocated (i.e. NCLB SES provisions), and potentially control over the school itself are strong incentives to collaborate. I just haven't seen that happening.

So let me talk about a different question. Namely, is NYC's plan a collective or an individual plan? I see this as an individual plan with an additional step - first, the school must meet benchmarks, and second, teachers can be differentially rewarded. Unless the committee of four announces upfront that bonuses will be distributed equally (can someone weigh in here about whether a distribution plan will be ratified upfront?), teachers are going to operate under the assumption that there will be unequal shares based on their students' test scores. Even if we see equal bonuses this year, the door is wide open and I see Mike and Joel on the horizon in a performance pay Mack truck.

Given the limited supervisory resources, the availability of individual teacher reports via the new data system (ARIS), and widespread concerns about favoritism and cronyism, I predict scores will weigh heavily if unequal shares are allocated. Despite all the hype about data-driven decision making, most administrators do not have the quantitative expertise to analyze test score data to differentiate teachers. Nor should they. Administrators should be instructional leaders, and data expertise and instructional leadership don't come in packages.

But to drive this point home: I've sat next to many principals as they review test scores, and they are not well-equipped to think about non-random assignment (in the same breath that they acknowledge that they intentionally placed more difficult kids with teacher A than B) and measurement error (I've had conversations that go like this: "Look, eduwonkette. Teacher A had 72% passing and Teacher B had 70%. I wouldn't have guessed it, but it looks like Teacher A is a better teacher." Principals really struggle with the idea of randomness.)

Before you cry eduwonkette elitism, let me also note that 95% of education researchers lack the training to make these distinctions. As I've said before, we would be much better off with a holistic evaluation system, but the resources just aren't there.

So I worry about the ramshackle way of comparing teachers that is likely to come out of this plan, and I'm afraid that Bloomberg/Klein gave the UFT just enough rope. If the UFT wanted to experiment with performance pay, I wish they would have locked in an equal share bonus plan or none at all. Why? Because when committees can't agree, or they do agree and some faculty members cry foul, there is going to be a tsunami of momentum for creating a centralized, individualized, and entirely test score-based system of measuring teacher effectiveness.

Did the UFT score one for NYC kids? I don't think so. But I hope for their sake that I'm wrong.

** Special thanks go to the UFT's Leo Casey and Steve Perez for fielding my questions about the plan, despite knowing that I probably wouldn't write a glowing endorsement. I'll look forward to your comments about this post.

Bonus Babies

Beautiful op-ed in the NYT today by Barry Schwartz asks the big questions about performance pay. His conclusion:

The more society embraces the idea that nobody will do anything right unless it pays, the more true it will become that nobody does anything right unless it pays. And this is no way to run a ballclub, a school system, or a country.

How Does Performance Pay Work in Other Sectors?

There's been lots of banter since last week's NYC deal about how compensation works in other sectors. Consider Joel Klein's musings in yesterday's New York Sun:

In business...successful firms pay more for what's worth more: they create incentives that motivate employees to work harder and more effectively.

Or Bloomberg's comments last week:

In the private sector, cash incentives are a proven motivator for producing results....[later in the article, Bloomberg goes on to say] I'm a capitalist.

But do other sectors reward workers based on the quantifiable outcomes of their clients? Richard Rothstein wrote on this issue seven years ago, and here's what he concluded:

It is hard to find private sector examples for [merit pay] proposals….When quality of work is important, corporations do not generally evaluate college-educated employees by quantifiable goals.

Consider, [Professor Brian Hall of Harvard Business School] said, management consulting. You might think the pay of consultants could be tied to growth in the equity value of companies they had assisted. But, he said, ''if you were evaluating employees who advised Cisco Systems and tried to measure their effectiveness by Cisco's stock price, every consultant in the last five years would look great, yet some may have given bad advice.''

So for professionals, Professor Hall said, decisions on merit pay are mostly subjective. Firms typically do a ''360-degree evaluation -- they go all around an individual, asking advice from supervisors above, co-workers to the left and right, and subordinates below.''

And how much time do supervisors spend on these subjective evaluations?

Bain managers devote about 100 hours a year to evaluating five employees. ''When I try to imagine a school principal doing 30 reviews, I have trouble,'' [a Bain director] said.

How about in private schools? Rothstein explains:

Pay for performance in private schools also does not resemble ideas being floated for public education. Certainly, elite academies do not use test data to calculate teacher pay. Nor, typically, do private schools for the less privileged.

In health care, performance pay is allocated based on clinical hours worked, procedures performed, and the revenues brought in, not the outcomes of the patients. Physicians I talked to in preparing this post laughed at me when I asked if their performance bonuses were based on patient outcomes. The most common response was that those outcomes were largely out of their control, so their hospitals rewarded them based on their inputs - i.e. hours, procedures, and revenues.

In law, contingency contracts - i.e. the firm or lawyer gets a certain amount if they win the case - are partially based on client outcomes, and certainly this gets reflected in bonuses. But the lawyers I talked to in preparing this post explained that hours billed are more important in determining their bonuses - again, there is a focus on inputs rather than outputs.

To be clear, I have no great affection for the current compensation system. For example, there's no reason to give educators a premium for earning an education masters degree (a degree in their content area makes sense, but most teachers aren't getting these degrees). There's no reason that we can't create incentives for teachers to teach in low-performing schools, though the best incentive is good working conditions. And as I wrote yesterday, I am all for rewarding expert teachers financially for providing support to struggling colleagues.

But in attending exclusively to annual test scores, a short-term and narrow outcome measure that is easy to manipulate, we are trotting down a dangerous path paved with unintended consequences. This is a bad idea in the private sector - i.e. Enron - and it's a bad idea for education, too. First, there are more goals of education than producing this year's test scores, and these goals have short and long-term components, i.e. are we rewarding teachers for what sticks with their students in subsequent years? Second, measuring teacher and school effectiveness based on a single year of data entirely ignores the issue of measurement error and what we know from the best research about the need to observe teachers for multiple years in order to accurately differentiate teachers. (See explanation here.) Third, performance pay systems rarely take into account the differences in the student populations served by different schools and teachers, and measuring growth does not account for imporant differences in student populations. (Again, see link above.) And they do not control for differences in structures over which the school has little control - i.e. class size and large intradistrict differences in per-pupil expenditures - which are relevant for plans that plan to compare performance across schools, not only within a given school.

So performance pay proponents' claims that other sectors reward workers based on client outcomes are largely untrue. While there is pay differentiation, it is based on a comprehensive and holistic evaluation of each person's work - something that will be difficult for schools to emulate given current levels of supervision.

Performance Pay Assumptions

Part 1 of 4 on performance pay. (Why Joe Torre? The Yankees blew it big time by offering a contract heavily based on performance pay.)

How is performance pay supposed to “work?” Performance pay advocates make three assumptions about how these plans will improve our schools, none of which are compelling to me:

1) Performance pay will cause teachers to try harder.

Last week, NYC Mayor Michael Bloomberg related, "In the private sector, cash incentives are a proven motivator for producing results. The most successful employees work harder, and everyone else tries to figure out how they can improve as well." Are teachers just slackers who need a juicy carrot to get them moving?

Given that most people go into teaching because of a commitment to kids, it strikes me that 1) teachers give kids their best as a matter of principle and conviction, and 2) teachers may be less likely to increase effort because of financial incentives than workers in the private sector. In econo-speak, point 2 suggests that teachers have different "objective functions" than private sector workers.

However, teachers do vary widely in their effectiveness. (See postings on measuring teacher effectiveness here.) Some performance pay supporters attribute a large proportion of this variation to something that can be changed by an incentive. But incentive plans are “capacity sorters” (i.e. they distinguish schools and teachers that already have organizational and instructional capacity from those that don't) as opposed to “capacity builders” (props to the Chicago Consortium’s Melissa Roderick for this insight). In this view, performance pay doesn’t help teachers who are struggling become better teachers. It only highlights what these teachers already know – that things aren’t going well in their classrooms.

Why not measure teacher effects, identify teachers who need more support, and fund expert teachers to provide instructional support to their colleagues – to plan with them, observe their classes, and provide feedback for improvement? This would be a higher yield investment than performance bonuses.

Of course, we can all muster examples of teachers who were dreadfully inadequate. My favorite example of late is a teacher who spent class time managing her inventory of Beanie Babies (noted in Karin Chenoweth’s Post op-ed). Ask yourself whether someone bizarre enough to spend class time organizing Beanie Babies is likely to stop doing so for $3000. I seriously doubt it.

2) In the case of school-based performance pay, it will encourage schoolwide collaboration.

If teachers in a school don’t collaborate, will a few thousand bucks melt years of bad feelings and internal rivalries? Ms. Frizzle made a great comment on this point, writing:

My gut tells me that in schools with high levels of trust and collaboration, this will work well, but that in schools with a divisive culture, it will make no difference or serve to divide the faculty even more….Once upon a time I worked in a much larger school with a fair amount of internal conflict. I don’t think dangling cash over our heads would have changed that very much…. and when I worked there, I would never have voted for a rewards system like this because I did not feel like a part of something larger than myself and did not trust my colleagues to work together towards a common goal.

For more on the role of trust in schools, check out this book by Tony Bryk and Barbara Schneider.

3) Performance pay will attract and keep better people in the teaching profession.

Will performance pay attract teachers who would not otherwise go into teaching? This argument assumes that there is a group of people out there who would teach but for their concern than their good efforts will not be rewarded financially. But relative salaries are central to career choices, and I suspect that people choose careers on the base salary as opposed to the salary plus bonus potential. In short, if you want to attract better people into the profession, you would be better off increasing base salaries.

Would performance pay keep better teachers in teaching? I don’t think so, because educators don’t leave teaching primarily because of the pay (though, as noted above, many don’t go into teaching in the first place because relative salaries are low – see this EPI study for comparisons with similar fields). By the time people have entered the profession, they already have accepted that teaching is not a bling-bling job. Many more teachers leave because they feel like they aren’t given the support to do their job well: i.e. their schools are chaotic, their classes are too big, their books are never delivered on time, there isn’t support to meet kids non-academic needs that affect learning, and there is little support to improve their own instruction. If we want teachers to stay in the profession – and we want them to teach in the most disadvantaged schools – improving working conditions would be a better place to invest than performance bonuses. (Hat tip to Leonie Haimson's analysis of the Public Agenda and North Carolina teacher surveys.)

To sum up, if performance pay advocates want to push these policies forward, they need to develop more satisfying rationales than the three reviewed above. I would love to see these folks lay out a detailed theory of change. (And please point me towards these explanations if they already exist.)

This week: Performance Pay

New York City just instituted a landmark performance pay plan, so this week I'll focus on the merits and the pitfalls of performance pay.

Tuesday: What is the theory of action underlying performance pay? That is, what do supporters of performance pay think will happen as a result of these plans?

Wednesday: How does performance pay work in other sectors?

Thursday: Reader comments roundup

Friday: How might individual performance pay plans work differently than collective ones? What are the potential benefits and costs of the NYC plan?