Tuesday, October 16, 2007

When a Lottery is Not a Lottery

Even KIPP’s critics acknowledge that KIPP students are chosen by lottery. What it means to be chosen by lottery is that there is no intermediate step between picking your name out of a hat and deciding whether you get to show up on the first day of school. From media depictions of KIPP, I was under the impression that a certain number of names are chosen and, give or take a handful of kids that represent random attrition, these are the kids who matriculate at KIPP.

It turns out that’s not exactly true. For example, consider the four KIPP schools that are located in New York City. From their websites, I found that students are chosen, and then are asked to come to the school for a meeting or have a phone conversation to discuss the expectations for the school; after these interactions, parents – or potentially the schools - decide if they will enroll their students at KIPP.

Contrary to many of KIPP’s critics, I don’t see this as strategic action (i.e. an attempt to cream the best students). KIPP wants to run schools where kids are committed to the culture, and this makes a lot of sense. Meeting kids individually before school starts is a necessary part of this process. KIPP schools ask kids to do things that are not required by regular public schools, and it’s impossible to run such a school without student and parent buy-in.

That said, here is the text from the KIPP A.M.P. application form, which is available at their website:

KIPP: A.M.P. Academy will extend offers of admission to the first 77 children whose names are picked in the lottery. If my child’s name is selected in the lottery, KIPP: A.M.P. Academy will contact me at the above-listed address and phone number to schedule a meeting. After my meeting, I will have to decide whether to accept the offer of admission and register my child at KIPP: A.M.P Academy for the 2007-2008 school year. If KIPP: A.M.P. makes a good faith effort to reach me but is unable to schedule a meeting, my child’s place at KIPP: A.M.P. Academy will automatically be surrendered to the first child on the waiting list.

I thought this might be specific to New York, so I checked out KIPP Indianapolis, where there is not just a phone call or a meeting, but a home visit before formal enrollment:

KIPP Indianapolis College Preparatory will extend offers of admission to the first 85 children whose names are picked in the lottery…If my child’s name is selected in the lottery, KIPP Indianapolis College Preparatory will contact me at the above-listed address and phone number to schedule a home visit.

It’s certainly possible that there is not significant attrition as a result of this process. From KIPP’s end, it would be helpful to know how much lottery to first day of school attrition we’re talking about. In other words, if a KIPP school draws 100 names, I would like to know what proportion of the original 100 attend KIPP, and also know the reasons why lottery winners chose not to attend. (This is a different kind of attrition than the post-matriculation attrition covered in Ed Week’s recent article; I’ll discuss this on Thursday.)

What’s the big deal? From the researcher's and public policymakers' perspective, the primary benefit of lottery selection is that it results in balancing the treatment and control groups on observable characteristics – i.e. students’ test scores, free lunch status, etc – and their “unobservable” characteristics – i.e. their motivation, aspirations, and commitment to KIPP’s principles. On any dimension we can think of, the treatment and control groups should be indistinguishable. We can then compare these groups to figure out how effective KIPP is, as I wrote in yesterday's post.

But if we draw a lottery of 100 people and after learning more about KIPP’s approach, 30 kids who were less motivated select out or are counseled out because they are a bad fit, we end up comparing very different treatment and control groups. The treatment group is now much more motivated/committed than the control group – many of whom might have opted out as well if they had these meetings before entering the lottery.

Again, I'm not sure that school leaders are aware that these post-lottery screening processes effectively result in non-lottery admission. For example, an old friend of mine recently started a KIPP-style charter school. This is a remarkable and brilliant guy who is deeply committed to improving educational options for disadvantaged kids; he’s not trying to play the system. His school drew 150 kids in the lottery for 100 seats at his school. His rationale for over-drawing was that many kids would realize they wouldn’t like the extra time, their parents wouldn’t accept the culture of the school, or the school would feel the family wasn’t sufficiently committed, and about 50 kids would withdraw even before school began. He still believed his students represented random draws from the lottery pool – after all, he argued, these are all poor and minority kids. I am not sure I was successful in convincing him otherwise.

The take-home lesson from this post is that we need to beware of the term “lottery” – it doesn’t always mean what we think it does.


Joanne Jacobs said...

I don't think KIPP loses many students in the sign-up process, though it would be good to have those numbers. I suspect it's more common to lose students during the school year when those long hours kick in.

I observed the sign-up process when I did a story on a KIPP principal recruiting students for a new school for the Christian Science Monitor. It's standard practice to get parents to commit to getting the child to school on time and, if necessary, on Saturdays. Students have to make a commitment too. By the time parents agreed to the sign-up interview, they seemed to have made up their minds to go with KIPP.

In this case, there was no lottery: The principal was signing up every kid she could find, literally walking the streets to find prospects. I'm sure the least committed parents don't consider KIPP or any other choice, but most who did in this case were low-income, poorly educated Mexican immigrants. The girl in the story was behind academically and had two older siblings who were doing poorly in school. It seems unlikely to me this girl would have turned into a good student at her old school.

By the start of its second year, KIPP Heartwood was doing so well that people now swear it creams the best students. I'm sure that its very, very high scores are attracting education-minded parents. But KIPP Heartwood started out with kids like Delia.

Robert Pondiscio said...

Love the blog, but frankly this is controversy looking for a place to affix itself. Of course KIPP kids are different. They become so the second their parents pick up the phone to enter the lottery. I taught in the South Bronx for several years, and in that time had exactly ONE parent follow through after I encouraged her to make that call; that kid goes to KIPP. Parental support matters.

Why shouldn't KIPP screen parents to make sure that the families are down with the program? The Gods may strike me dead for this unfashionable opinion, but families NOT being down with the program is precisely what turns the average low-performing school into a house of horrors.

The gold standard is high achievement among a random assignment of students? That's great. I'm glad there are well intentioned people out there who will not rest until EVERY child is doing well. But I'm even happier there are the KIPPs of the world, who are ready to help the kids and families who want to take us up on our offer of a great education. Today. Right now.

People of means send their children to private school. I've long held that what you're really paying for is not a first-rate education (at least at the primary level), but a first-rate environment. As a parent, I want my child to be in school with other kids from families who value education. I'm not prepared to demand that poor families have their child's educational time taken up by those who have to be coerced into acting in their own best interests.

KIPP offers an education -- and more to the point, a learning environment -- that is light years ahead of what those same children would get in their district schools. I fail to see how quibbling about their selection process alters that unassailable fact, or diminishes their accomplishment.