Monday, October 15, 2007

Do KIPP schools have a positive effect on their students' achievement?

KIPP has been held up as a savior by its supporters and used as a perennial punching bag by its critics. In my view, these exchanges have not been particularly illuminating. And in most battles, both sides have neglected some important questions. So each day this week, I'll write about some of the issues raised by the KIPP case.

The Knowledge is Power Program (KIPP) is a network of 57 schools in 17 states. KIPP has received a significant amount of media attention for the results their schools have posted with urban students. A quick tour through the KIPP Annual Report reveals some impressive gains. What do we need to know before confirming that KIPP has a positive effect on their students’ academic achievement?

One way of outlining the challenge with evaluating KIPP is to think about how we would design a study to estimate its effects on the students KIPP serves. Recall that I am not asking whether KIPP provides a solution for students beyond those currently attending KIPP schools – this is a question I will address on Wednesday. But the first question we need to ask is whether a student attending KIPP is better off than she would have been had she attended a non-KIPP school.

Many journalists have attempted to answer this question by comparing the performance of students in a KIPP school with those at the closest neighborhood schools. For example, Paul Tough wrote in last year’s New York Times magazine:

When the scores [at KIPP Bronx Academy] are compared with the scores of the specific high-poverty cities or neighborhoods where they are located…it isn’t even close: 86 percent of eight-grade students at KIPP Academy scored at grade level in math this year, compared with 16 percent of students in the South Bronx.

Tough observes that KIPP students are doing much better, and concludes that KIPP is effective. What’s wrong with this argument?

First, that students selected into a KIPP lottery makes them different from than those who did not. It may be that their parents are more involved in their education, that they are having a particularly bad experience at their neighborhood school, or that their parents can no longer pay for private school. Whatever the reason, families selecting in, even if they are all poor and minority kids, are different by virtue of choosing a non-neighborhood school.

Lots of choice advocates will spar on this point, and argue that everyone wants a better choice for their children, so there is no selection problem. While rhetorically effective, anyone arguing that families that choose into a charter school are the same as those who don’t is simply wrong. Random assignment is the gold standard of causal inference in the natural and social sciences, and kids are not randomly assigned to KIPP lotteries. Saying that 80% of the kids are poor and 90% are African-American and Hispanic doesn't solve this problem. Even if KIPP kids had test scores identical to their neighborhood school peers, we still couldn't compare KIPP and neighborhood school kids who didn't opt in with any confidence because there is selection on "unobservables" - things like motivation and aspirations that are not measured by administrative datasets used to make these comparisons.

To get around this problem, KIPP has compared students' baseline performance with their performance after three years in a KIPP school. Jay Mathews summarizes the results of these analyses in an article earlier this year:

A KIPP analysis of the scores of about 1,400 students in 22 cities who have completed three years at KIPP show they went from the 34th percentile at the beginning of fifth grade to the 58th percentile at the end of the seventh grade in reading and from the 44th percentile to the 83rd percentile in math....Gains like that for that many disadvantaged children in one program have never happened before.

The argument is that students’ growth demonstrates the effect of the KIPP school. The problem with this approach is we have no way of knowing that these students wouldn’t have made similar gains anyway.

The best approach, and one that Mathematica has been contracted to carry out, is to compare students who entered the lottery and won with those who entered the lottery and lost. To keep this example as straightforward as possible, let’s assume that all lottery winners enroll and stay, and all lottery losers go to their neighborhood school, and there is no attrition in either case. We can now compare the achievement of these two groups and call the average difference the “treatment effect on the treated” – the effect of receiving a KIPP education on the students who received it. If the KIPP students are better off, we can say that KIPP “worked” for them.

The effectiveness of the research design I just proposed rests on two assumptions. First, we need to be sure that the lottery is actually a lottery, i.e. that we are not dealing with a “broken experiment.” (For a good explanation of this problem, see the Cullen and Jacob paper I wrote about last week.) A second way in which the design I described falls apart is when there is “selective attrition.” Selective attrition is the idea that people who choose to leave an experiment are different in one way or another. I’ll discuss these two issues later in the week.

For now, I'll conclude that we know that 1) KIPP kids perform better, on average, than their neighborhood school peers, 2) KIPP kids exhibit very large value-added gains on standardized tests. But we actually don’t know if KIPP kids are better off academically by virtue of attending KIPP than they would have been if KIPP didn't exist. There are certainly good reasons to believe that they are - i.e. they are in school substantially more - but the size of the "KIPP effect" is probably much smaller than we currently believe it to be.

It perplexes me that journalists continue to downplay these concerns. For example, in an article from a few years back, Jay Mathews wrote:

Whatever the academic or family characteristics of incoming KIPP students, they are clearly disadvantaged -- 82 percent of all KIPP students qualify for federal lunch subsidies -- and at KIPP have achieved gains in reading and mathematics far above those of other programs trying to help such children.
And Paul Tough wrote:
In some ways, the debate seems a trivial one — KIPP is clearly doing a great job of educating its students; do the incoming scores at a single school really matter?
I hope this post convinces you that these evaluation concerns are non-trivial. If we really want to know if these schools are working and how large their effects are (and for whom), we need to take these issues seriously. Perhaps Jay Mathews himself said it best when he wrote:
I understand why we education reporters try to make KIPP sound like more than it is. We are starved for good news about low-income schools. KIPP is an encouraging story, so we are tempted to gush rather than report. We don't ask all the questions we should.


Anonymous said...

Why is it you think that kids who spent 5 years in school and only got to the 34th percentile nationally would not logically stay at that percentile when they hit the middle school years in the same system? It is statistically extremely likely that they would see very little movement, so the KIPP increase to the 58th percentile is, one can be quite sure (even without your methodology), a dramatic improvement over the district school system.

Anonymous said...

I understand some of your concerns with the "KIPP hype", but...

1. I have to agree with the other poster who said it was extremely unlikely that students who had consistently been in the 34th percentile would make such dramatic improvements in the SAME school system they had been in.

2. I've had a lot of conversations with teachers at a KIPP school in my city, Washington, DC, and (while there is a CHANCE this may not be the norm for other KIPP schools) now know that this school (a) is one of the best--if not THE best--public school in the inner city for kids with special needs (it actually has students MOVING OUT of SpEd) (b) has students coming in at the 17th percentile in reading ON AVERAGE. that means many of the kids CANNOT READ WORDS. they are in 5th grade and are reading at a kindergarten (at BEST!) reading level. many of these students were actually the LOWER performing students in the public school, not the brightest students that you suggest KIPP skims off. Let me just repeat that if in 5 YEARS in the DC PS system, these kids CANNOT READ or ADD NUMBERS, they almost definitely would NOT be scoring so high without KIPP. KIPP will spend HOURS AND HOURS AND HOURS a day on reading and math, and yeah, some people will say "what about art and music and science"? but dude, who the hell cares if little Michael can play twinkle-twinkle on the recorder if he CAN'T READ WORDS!? And as I said, KIPP has to WORK and WORK for these numbers! Their students do not come in knowing how to read or do math. Their students have just as many problems at home as the kids in the normal PS. Your assertion that of course THESE students at KIPP are special and smarter and have better families is just wrong--KIPP basically has to go door to door in bad neighborhoods and beg parents to sign up. Sure, you might get some parents who are very supportive and motivated, but you can always find a few parents like that anywhere. The majority of KIPP students, however, are just like their counterparts at the DC PS, and often even LOWER PERFORMING.

3. The numbers are not an illusion. While we may be starved for good news, we cannot ignore the fact that the numbers are RIDICULOUSLY low in the public schools and fabulously high at the KIPP schools. If KIPP students graduate knowing how to read books, write essays, do pre-algebra, and golly-gee even have good study skills, then I'm not just happy--I'm IMPRESSED. Why don't you let yourself be a little impressed? What have you got against KIPP's success?

Anonymous said...

Well said and beautifully written

Anonymous said...

selection bias is a cruel mistress indeed - excellent points.

SLM said...

If you are saying that KIPP's results are due only to having a better crop of students than the public schools', and not to having a better educational system, then why do you care who gets into KIPPs or not? It's not as though either set of students, by your reckoning, would have different outcomes if their positions were switched.

It seems more likely that even if KIPP manages to snag the better students, these students at least have a chance at a better education. Why not provide an option for parents and children who are motivated to work hard?

Clix said...

I was actually thinking about the same thing myself!

I think KIPP does do a fantastic job educating the students who go there. Can that success be replicated for all students? And if not, what CAN we do to help students who do not have family and community support?

I also wonder how the teacher retention rate at KIPP schools compares to that at typical public schools?

Anonymous said...

I also agree with my fellow posters,I believe that students that attend to a KIPP school will have a high chance in getting into a good high school.The lessons that we learn at my Kipp School will not only help me suceed in school, but in life aswell.