Do KIPP schools have a positive effect on their students' achievement?

KIPP has been held up as a savior by its supporters and used as a perennial punching bag by its critics. In my view, these exchanges have not been particularly illuminating. And in most battles, both sides have neglected some important questions. So each day this week, I'll write about some of the issues raised by the KIPP case.

The Knowledge is Power Program (KIPP) is a network of 57 schools in 17 states. KIPP has received a significant amount of media attention for the results their schools have posted with urban students. A quick tour through the KIPP Annual Report reveals some impressive gains. What do we need to know before confirming that KIPP has a positive effect on their students’ academic achievement?

One way of outlining the challenge with evaluating KIPP is to think about how we would design a study to estimate its effects on the students KIPP serves. Recall that I am not asking whether KIPP provides a solution for students beyond those currently attending KIPP schools – this is a question I will address on Wednesday. But the first question we need to ask is whether a student attending KIPP is better off than she would have been had she attended a non-KIPP school.

Many journalists have attempted to answer this question by comparing the performance of students in a KIPP school with those at the closest neighborhood schools. For example, Paul Tough wrote in last year’s New York Times magazine:

When the scores [at KIPP Bronx Academy] are compared with the scores of the specific high-poverty cities or neighborhoods where they are located…it isn’t even close: 86 percent of eight-grade students at KIPP Academy scored at grade level in math this year, compared with 16 percent of students in the South Bronx.

Tough observes that KIPP students are doing much better, and concludes that KIPP is effective. What’s wrong with this argument?

First, that students selected into a KIPP lottery makes them different from than those who did not. It may be that their parents are more involved in their education, that they are having a particularly bad experience at their neighborhood school, or that their parents can no longer pay for private school. Whatever the reason, families selecting in, even if they are all poor and minority kids, are different by virtue of choosing a non-neighborhood school.

Lots of choice advocates will spar on this point, and argue that everyone wants a better choice for their children, so there is no selection problem. While rhetorically effective, anyone arguing that families that choose into a charter school are the same as those who don’t is simply wrong. Random assignment is the gold standard of causal inference in the natural and social sciences, and kids are not randomly assigned to KIPP lotteries. Saying that 80% of the kids are poor and 90% are African-American and Hispanic doesn't solve this problem. Even if KIPP kids had test scores identical to their neighborhood school peers, we still couldn't compare KIPP and neighborhood school kids who didn't opt in with any confidence because there is selection on "unobservables" - things like motivation and aspirations that are not measured by administrative datasets used to make these comparisons.

To get around this problem, KIPP has compared students' baseline performance with their performance after three years in a KIPP school. Jay Mathews summarizes the results of these analyses in an article earlier this year:

A KIPP analysis of the scores of about 1,400 students in 22 cities who have completed three years at KIPP show they went from the 34th percentile at the beginning of fifth grade to the 58th percentile at the end of the seventh grade in reading and from the 44th percentile to the 83rd percentile in math....Gains like that for that many disadvantaged children in one program have never happened before.

The argument is that students’ growth demonstrates the effect of the KIPP school. The problem with this approach is we have no way of knowing that these students wouldn’t have made similar gains anyway.

The best approach, and one that Mathematica has been contracted to carry out, is to compare students who entered the lottery and won with those who entered the lottery and lost. To keep this example as straightforward as possible, let’s assume that all lottery winners enroll and stay, and all lottery losers go to their neighborhood school, and there is no attrition in either case. We can now compare the achievement of these two groups and call the average difference the “treatment effect on the treated” – the effect of receiving a KIPP education on the students who received it. If the KIPP students are better off, we can say that KIPP “worked” for them.

The effectiveness of the research design I just proposed rests on two assumptions. First, we need to be sure that the lottery is actually a lottery, i.e. that we are not dealing with a “broken experiment.” (For a good explanation of this problem, see the Cullen and Jacob paper I wrote about last week.) A second way in which the design I described falls apart is when there is “selective attrition.” Selective attrition is the idea that people who choose to leave an experiment are different in one way or another. I’ll discuss these two issues later in the week.

For now, I'll conclude that we know that 1) KIPP kids perform better, on average, than their neighborhood school peers, 2) KIPP kids exhibit very large value-added gains on standardized tests. But we actually don’t know if KIPP kids are better off academically by virtue of attending KIPP than they would have been if KIPP didn't exist. There are certainly good reasons to believe that they are - i.e. they are in school substantially more - but the size of the "KIPP effect" is probably much smaller than we currently believe it to be.

It perplexes me that journalists continue to downplay these concerns. For example, in an article from a few years back, Jay Mathews wrote:

Whatever the academic or family characteristics of incoming KIPP students, they are clearly disadvantaged -- 82 percent of all KIPP students qualify for federal lunch subsidies -- and at KIPP have achieved gains in reading and mathematics far above those of other programs trying to help such children.

And Paul Tough wrote:

In some ways, the debate seems a trivial one — KIPP is clearly doing a great job of educating its students; do the incoming scores at a single school really matter?

I hope this post convinces you that these evaluation concerns are non-trivial. If we really want to know if these schools are working and how large their effects are (and for whom), we need to take these issues seriously. Perhaps Jay Mathews himself said it best when he wrote:

I understand why we education reporters try to make KIPP sound like more than it is. We are starved for good news about low-income schools. KIPP is an encouraging story, so we are tempted to gush rather than report. We don't ask all the questions we should.