Thoughts on NYC's Performance Pay Plan

Now that we've gone over performance pay assumptions and how performance pay works in other sectors, what about the NYC plan?

The refrain we've heard this week from the UFT and other observers is that this is a collective plan that, as Leo Casey wrote, "transformed a negative into a positive, and 'shut the door' on individual merit pay programs." Echoing this sentiment in the NY Daily News, Diane Ravitch wrote, "Score one for this union."

I've already written about the likely impacts of financial incentives on collaboration (see Tuesday and Thursday's posts). To recapitulate, the argument that school-based incentives address collaboration problems fundamentally misunderstands the reasons why collaboration doesn't happen.

However, if we did believe that school-based incentives spur collaboration, existing accountability systems, both state and federal, should have already taken care of this. That your school will lose students, control over how funds are allocated (i.e. NCLB SES provisions), and potentially control over the school itself are strong incentives to collaborate. I just haven't seen that happening.

So let me talk about a different question. Namely, is NYC's plan a collective or an individual plan? I see this as an individual plan with an additional step - first, the school must meet benchmarks, and second, teachers can be differentially rewarded. Unless the committee of four announces upfront that bonuses will be distributed equally (can someone weigh in here about whether a distribution plan will be ratified upfront?), teachers are going to operate under the assumption that there will be unequal shares based on their students' test scores. Even if we see equal bonuses this year, the door is wide open and I see Mike and Joel on the horizon in a performance pay Mack truck.

Given the limited supervisory resources, the availability of individual teacher reports via the new data system (ARIS), and widespread concerns about favoritism and cronyism, I predict scores will weigh heavily if unequal shares are allocated. Despite all the hype about data-driven decision making, most administrators do not have the quantitative expertise to analyze test score data to differentiate teachers. Nor should they. Administrators should be instructional leaders, and data expertise and instructional leadership don't come in packages.

But to drive this point home: I've sat next to many principals as they review test scores, and they are not well-equipped to think about non-random assignment (in the same breath that they acknowledge that they intentionally placed more difficult kids with teacher A than B) and measurement error (I've had conversations that go like this: "Look, eduwonkette. Teacher A had 72% passing and Teacher B had 70%. I wouldn't have guessed it, but it looks like Teacher A is a better teacher." Principals really struggle with the idea of randomness.)

Before you cry eduwonkette elitism, let me also note that 95% of education researchers lack the training to make these distinctions. As I've said before, we would be much better off with a holistic evaluation system, but the resources just aren't there.

So I worry about the ramshackle way of comparing teachers that is likely to come out of this plan, and I'm afraid that Bloomberg/Klein gave the UFT just enough rope. If the UFT wanted to experiment with performance pay, I wish they would have locked in an equal share bonus plan or none at all. Why? Because when committees can't agree, or they do agree and some faculty members cry foul, there is going to be a tsunami of momentum for creating a centralized, individualized, and entirely test score-based system of measuring teacher effectiveness.

Did the UFT score one for NYC kids? I don't think so. But I hope for their sake that I'm wrong.

** Special thanks go to the UFT's Leo Casey and Steve Perez for fielding my questions about the plan, despite knowing that I probably wouldn't write a glowing endorsement. I'll look forward to your comments about this post.

Halloween Edu-Parade Sneak Preview

Meet Lord VoldeBush, harbinger of things to come in next Wednesday's Halloween edu-parade.

And if you're still looking for a costume, check out the 2007 Halloween Consumer Intentions and Actions Survey. The princess dominates the top ten children's costumes:

1) Princess (4,168,472)
2) Spider-Man (1,882,536)
3) Pirate
4) Witch
5) Fairy
6) Disney Princess
7) Star Wars Character
8) Pumpkin
9) Ghost
10) Power Ranger



And the top ten adult costumes include:

1) Witch
2) Pirate
3) Vampire
4) Cat
5) Princess
6) Fairy
7) Wench
8) Clown
9) Angel
10) Nurse

Reader comments roundup: Performance Pay

Dear jack phelps, andromeda, roger sweeny, anonymous (x2) and world,

I like you. I hope you keep writing comments, because you said some important things that forced me to think through my posts this week.

First things first. Anonymous 7:14AM – I can’t even stand to read this blog at 7:14AM. So hats off to you. You wrote:

The change theory seems to be "only if the best teachers help their colleagues identify challenges and improve will entire schools succeed and will teachers then become eligible for more pay. This will encourage individual teachers to do their personal best. It also will encourage collaboration and teamwork, as teachers share both responsibilities and rewards." Looks like capacity-sorting to me; how does this incentive build teachers' capacity to collaborate effectively?

Agreed - collaboration in a school is something that needs to be nurtured – it takes time, organizational structures, and clear attention to building a process of feedback and continuous conversation into everyday school life. In some of the best schools I’ve visited, the schedule builds in additional prep periods that have specified purposes – i.e. planning together or going over this week’s lessons to forge interdisciplinary connections. One of the best collaborative structures I’ve seen in junior and high schools is “kidtalk,” where a team of teachers gets together to raise issues about kids, have conferences with individual kids as a team, and share strategies for helping each kid learn. It’s an amazing process to watch and it produces good results in addition to fostering rapport among the teachers.

The trouble is that organizational structures cost money. If we cost it out on an hourly basis (5 additional hours a week at $30/hour, which is a low-ball number), we’re talking about $5400 per faculty member. Even if we assume that $3000 is delayed payment for this work, we’re not getting anywhere close to compensating teachers for this additional time. And the truth is that struggling teachers don’t just need common planning, but someone in their classroom on a regular basis to observe and coach them – which also requires money. So anonymous 7:14AM, I agree that this incentive plan is largely a capacity sorter.

jack phelps was not convinced about capacity sorting. (great post, btw – I’m happy we have you on the education team now instead of the banking one. everyone, read his whole comment.)

I'd say that sure, your average 1st year teacher is working his or her ass off just to get the kids focused. Among 1st years, pay might be a capacity sorter. But you think that as they get married, start saving for retirement, have their own kids and family to look after--basically their own, outside set of issues--most teachers still "give their best" not just while they're required to be in class, but in planning activities, developing content, reaching out to students or parents, and even analyzing quiz results to see which questions are weak and should be thrown out? There are a host of marginal efforts teachers can make outside their 47 minutes of class time (where, I agree, they probably give their best), and additional performance pay may encourage them to do so. And it's ridiculous to say that a teacher trying to retire won't work a little harder his last few years to make sure he's saved as much as possible!

So you’ve convinced me that performance pay could produce marginal efforts in teachers at different life course stages, but I want to add a “for whom” statement here – for teachers for whom the aggregate of small additional efforts put their schools in the running for getting performance pay. What proportion of teachers this involves depends on how high the standards are for getting performance pay – we’ll know more about this as details of the NYC plan become available.

andromeda, a teacher in a private school in Massachusetts, is thinking about leaving the profession because of the lack of merit pay. She writes:

It's not that I want to be paid a lot of money -- if I did, I wouldn't have gone into teaching (and private school at that). What rankles is that one of the major ways respect is expressed in this society is money, and I want to be respected if I'm good at what I do. I want to be rewarded if I work hard and do a good job. The way the system's set up, I can't….I'd like my employers to say, in a meaningful way, that they appreciate my contribution….And I'm really starting to think, maybe I'd be happier in a corporate culture that values me.

I definitely hear you - you do a hard job, and you deserve to be recognized when you do it well. And schools absolutely need to develop structures to do this - and I'm okay with these rewards being financial. The question is about how we decide who gets these rewards; I'm partial to the kind of holistic evaluation that happens in other sectors (anonymous #2, check out yesterday's post on this question), rather than a test score-based system, but am skeptical about the capacity of schools to perform these kinds of evaluations given limited supervisory resources.

Finally, Roger Sweeny wrote:

Just about every school system pays its teachers more if they take additional education courses. Most teachers complain that these courses do little or nothing to make them better teachers. If teachers didn't respond to financial incentives, very few of them would take any of the courses. But many, many do.

Good point, Roger - I didn't mean that teachers were unresponsive to financial incentives, but perhaps less responsive than private sector workers. I like the idea of looking at increases in masters degree enrollment in the past (i.e. when financial incentives were tied to degree attainment) as providing insight into just how responsive teachers are.

Keep the comments coming, everyone. I'll try to respond once a week.

Cool people you should know: Jesse Rothstein

Jesse Rothstein is a Princeton economist who has written on a formidable range of topics: racial segregation, affirmative action in law schools, school choice, the value of SATs in predicting college performance, and the effects of students loans on career choices. You can find his papers here.

Cool findings:

* The SAT’s role as a predictor of freshman grades is quite sensitive to students’ background characteristics. Characteristics of students’ schools are particularly important, and account for a large share of the SAT’s predictive power.

* For example, SAT scores are much less predictive of collegiate grade point averages (GPAs) among same-race students from demographically-similar schools than they are in the population at large. For example, racial minorities and students from schools with high concentrations of Blacks, Hispanics, or low-education parents earned lower freshman grades than did White students from high income schools, even when SAT scores were similar.

* High school grades are equally predictive across and within demographic categories, suggesting that grade inflation does not erode the value of GPA comparisons across high schools.

(There is a good policy brief on his site that says more about this study - the bullets above are from this brief.)

Gold Star Book Award! Mitchell Stevens, Creating a Class

A month ago, I nominated four of this fall's best academic books for the first eduwonkette "Gold Star Award." All four were a treat - I would honestly recommend reading all of them - but I fell in love with one.

That book was Mitchell Stevens' Creating a Class: College Admissions and the Education of Elites. Whether you are a teacher, a guidance counselor, a parent, a high school student, or a policy wonk, this book will introduce you to the inner workings of an elite liberal arts admissions office. Don't be fooled - this is not your garden variety expose book by a former admissions officer - there are already enough of these that they deserve their own genre.

In this book, Mitchell Stevens does something different - he spends a year and a half inside an admissions office, and shows how the increasingly cutthroat competition for institutional standing affects every aspect of the admissions process. He reveals how the need to increase the school's relative standing collides with ideals of educational opportunity, as officers reject applications because the parents can't pay, take applications because the college wants to "get a thing going" with various private schools, and recruit at more advantaged high schools simply because they have the organizational machinery to meet the needs of admissions officers. There are a lot of parallels to K-12 accountability policies.

Certainly, a lot of these behaviors have been reported elsewhere - but Stevens helps the reader to understand how the structure of the higher education market causes these behaviors, and how this has changed over time. And he never falls into the common journalistic trap of calling these officers rotten apples when they do and say things that raise eyebrows. Instead, he helps the reader understand why she would have done the job exactly the same way.

Higher education is not my area of study, and I think that the frenzy over elite college admissions is overplayed. (See an earlier post about income effects of attending an elite school.) But Stevens makes an important point about the role of elite colleges as an elaborate insurance policy for the wealthy. In an increasingly uncertain world, middle class futures aren't guaranteed for the kids of affluent parents. While a college degree is more necessary than ever, half of the kids who enter college don't finish. And all else equal, kids are much more likely to graduate at elite schools.

Perhaps best of all - and what makes Creating a Class stand apart from your average academic book - it was a pleasure to read and an important contribution. In addition to lending new insights about the structure of opportunity in higher education, Stevens is wickedly witty.

So go pick up Creating a Class, everyone. And kudos to Stevens for turning in this important piece of scholarship.

Bonus Babies

Beautiful op-ed in the NYT today by Barry Schwartz asks the big questions about performance pay. His conclusion:

The more society embraces the idea that nobody will do anything right unless it pays, the more true it will become that nobody does anything right unless it pays. And this is no way to run a ballclub, a school system, or a country.

How Does Performance Pay Work in Other Sectors?

There's been lots of banter since last week's NYC deal about how compensation works in other sectors. Consider Joel Klein's musings in yesterday's New York Sun:

In business...successful firms pay more for what's worth more: they create incentives that motivate employees to work harder and more effectively.

Or Bloomberg's comments last week:

In the private sector, cash incentives are a proven motivator for producing results....[later in the article, Bloomberg goes on to say] I'm a capitalist.

But do other sectors reward workers based on the quantifiable outcomes of their clients? Richard Rothstein wrote on this issue seven years ago, and here's what he concluded:

It is hard to find private sector examples for [merit pay] proposals….When quality of work is important, corporations do not generally evaluate college-educated employees by quantifiable goals.

Consider, [Professor Brian Hall of Harvard Business School] said, management consulting. You might think the pay of consultants could be tied to growth in the equity value of companies they had assisted. But, he said, ''if you were evaluating employees who advised Cisco Systems and tried to measure their effectiveness by Cisco's stock price, every consultant in the last five years would look great, yet some may have given bad advice.''

So for professionals, Professor Hall said, decisions on merit pay are mostly subjective. Firms typically do a ''360-degree evaluation -- they go all around an individual, asking advice from supervisors above, co-workers to the left and right, and subordinates below.''

And how much time do supervisors spend on these subjective evaluations?

Bain managers devote about 100 hours a year to evaluating five employees. ''When I try to imagine a school principal doing 30 reviews, I have trouble,'' [a Bain director] said.

How about in private schools? Rothstein explains:

Pay for performance in private schools also does not resemble ideas being floated for public education. Certainly, elite academies do not use test data to calculate teacher pay. Nor, typically, do private schools for the less privileged.

In health care, performance pay is allocated based on clinical hours worked, procedures performed, and the revenues brought in, not the outcomes of the patients. Physicians I talked to in preparing this post laughed at me when I asked if their performance bonuses were based on patient outcomes. The most common response was that those outcomes were largely out of their control, so their hospitals rewarded them based on their inputs - i.e. hours, procedures, and revenues.

In law, contingency contracts - i.e. the firm or lawyer gets a certain amount if they win the case - are partially based on client outcomes, and certainly this gets reflected in bonuses. But the lawyers I talked to in preparing this post explained that hours billed are more important in determining their bonuses - again, there is a focus on inputs rather than outputs.

To be clear, I have no great affection for the current compensation system. For example, there's no reason to give educators a premium for earning an education masters degree (a degree in their content area makes sense, but most teachers aren't getting these degrees). There's no reason that we can't create incentives for teachers to teach in low-performing schools, though the best incentive is good working conditions. And as I wrote yesterday, I am all for rewarding expert teachers financially for providing support to struggling colleagues.

But in attending exclusively to annual test scores, a short-term and narrow outcome measure that is easy to manipulate, we are trotting down a dangerous path paved with unintended consequences. This is a bad idea in the private sector - i.e. Enron - and it's a bad idea for education, too. First, there are more goals of education than producing this year's test scores, and these goals have short and long-term components, i.e. are we rewarding teachers for what sticks with their students in subsequent years? Second, measuring teacher and school effectiveness based on a single year of data entirely ignores the issue of measurement error and what we know from the best research about the need to observe teachers for multiple years in order to accurately differentiate teachers. (See explanation here.) Third, performance pay systems rarely take into account the differences in the student populations served by different schools and teachers, and measuring growth does not account for imporant differences in student populations. (Again, see link above.) And they do not control for differences in structures over which the school has little control - i.e. class size and large intradistrict differences in per-pupil expenditures - which are relevant for plans that plan to compare performance across schools, not only within a given school.

So performance pay proponents' claims that other sectors reward workers based on client outcomes are largely untrue. While there is pay differentiation, it is based on a comprehensive and holistic evaluation of each person's work - something that will be difficult for schools to emulate given current levels of supervision.

Performance Pay Assumptions

Part 1 of 4 on performance pay. (Why Joe Torre? The Yankees blew it big time by offering a contract heavily based on performance pay.)

How is performance pay supposed to “work?” Performance pay advocates make three assumptions about how these plans will improve our schools, none of which are compelling to me:

1) Performance pay will cause teachers to try harder.

Last week, NYC Mayor Michael Bloomberg related, "In the private sector, cash incentives are a proven motivator for producing results. The most successful employees work harder, and everyone else tries to figure out how they can improve as well." Are teachers just slackers who need a juicy carrot to get them moving?

Given that most people go into teaching because of a commitment to kids, it strikes me that 1) teachers give kids their best as a matter of principle and conviction, and 2) teachers may be less likely to increase effort because of financial incentives than workers in the private sector. In econo-speak, point 2 suggests that teachers have different "objective functions" than private sector workers.

However, teachers do vary widely in their effectiveness. (See postings on measuring teacher effectiveness here.) Some performance pay supporters attribute a large proportion of this variation to something that can be changed by an incentive. But incentive plans are “capacity sorters” (i.e. they distinguish schools and teachers that already have organizational and instructional capacity from those that don't) as opposed to “capacity builders” (props to the Chicago Consortium’s Melissa Roderick for this insight). In this view, performance pay doesn’t help teachers who are struggling become better teachers. It only highlights what these teachers already know – that things aren’t going well in their classrooms.

Why not measure teacher effects, identify teachers who need more support, and fund expert teachers to provide instructional support to their colleagues – to plan with them, observe their classes, and provide feedback for improvement? This would be a higher yield investment than performance bonuses.

Of course, we can all muster examples of teachers who were dreadfully inadequate. My favorite example of late is a teacher who spent class time managing her inventory of Beanie Babies (noted in Karin Chenoweth’s Post op-ed). Ask yourself whether someone bizarre enough to spend class time organizing Beanie Babies is likely to stop doing so for $3000. I seriously doubt it.

2) In the case of school-based performance pay, it will encourage schoolwide collaboration.

If teachers in a school don’t collaborate, will a few thousand bucks melt years of bad feelings and internal rivalries? Ms. Frizzle made a great comment on this point, writing:

My gut tells me that in schools with high levels of trust and collaboration, this will work well, but that in schools with a divisive culture, it will make no difference or serve to divide the faculty even more….Once upon a time I worked in a much larger school with a fair amount of internal conflict. I don’t think dangling cash over our heads would have changed that very much…. and when I worked there, I would never have voted for a rewards system like this because I did not feel like a part of something larger than myself and did not trust my colleagues to work together towards a common goal.

For more on the role of trust in schools, check out this book by Tony Bryk and Barbara Schneider.

3) Performance pay will attract and keep better people in the teaching profession.

Will performance pay attract teachers who would not otherwise go into teaching? This argument assumes that there is a group of people out there who would teach but for their concern than their good efforts will not be rewarded financially. But relative salaries are central to career choices, and I suspect that people choose careers on the base salary as opposed to the salary plus bonus potential. In short, if you want to attract better people into the profession, you would be better off increasing base salaries.

Would performance pay keep better teachers in teaching? I don’t think so, because educators don’t leave teaching primarily because of the pay (though, as noted above, many don’t go into teaching in the first place because relative salaries are low – see this EPI study for comparisons with similar fields). By the time people have entered the profession, they already have accepted that teaching is not a bling-bling job. Many more teachers leave because they feel like they aren’t given the support to do their job well: i.e. their schools are chaotic, their classes are too big, their books are never delivered on time, there isn’t support to meet kids non-academic needs that affect learning, and there is little support to improve their own instruction. If we want teachers to stay in the profession – and we want them to teach in the most disadvantaged schools – improving working conditions would be a better place to invest than performance bonuses. (Hat tip to Leonie Haimson's analysis of the Public Agenda and North Carolina teacher surveys.)

To sum up, if performance pay advocates want to push these policies forward, they need to develop more satisfying rationales than the three reviewed above. I would love to see these folks lay out a detailed theory of change. (And please point me towards these explanations if they already exist.)

Halloween Edu-Parade Costume-a-Thon: Last Call!

Your school may not be able to have a parade, but my blog can! Costume nominations end Monday, October 22 at midnight.

My new favorites are Eli Broad as Mr. Burns from the Simpsons, Bloomberg as the Emperor, Joel Klein as Darth Vader, Broad as Jar Jar Binks, and Maggie S. as a giant bar of Ivory soap; old favorites include Randi Weingarten as a teacher, Roland Fryer as Willie Wonka, George Bush as Voldemort, Steve Barr walking a pet pig (remarkable explanation available here), and Joel, Mike, Eli, Bill, and Andy as a boy band. Also, in honor of the pending NCLB reauthorization, I've decided to add Rod Paige - nominate away.

1) Randi Weingarten
2) Joel Klein
3) Michael Bloomberg
4) Paul Vallas
5) Bob Herbert
6) Roland Fryer
7) George Bush
8) Andy Rotherham
9) Michelle Rhee
10) Bill Gates
11) Eli Broad
12) Wendy Kopp
13) Steve Barr
14) Alexander Russo
15) Margaret Spellings
16) Rod Paige

This week: Performance Pay

New York City just instituted a landmark performance pay plan, so this week I'll focus on the merits and the pitfalls of performance pay.

Tuesday: What is the theory of action underlying performance pay? That is, what do supporters of performance pay think will happen as a result of these plans?

Wednesday: How does performance pay work in other sectors?

Thursday: Reader comments roundup

Friday: How might individual performance pay plans work differently than collective ones? What are the potential benefits and costs of the NYC plan?