The Trouble with NYC Report Card Peer Groups

Central to the NYC Report Cards is the idea of a "peer group." The concept is a good one - it's not fair to compare schools serving vastly different populations of kids, as does NCLB, because kids may be doing worse in school because of out-of-school issues the school can't control.

The NYC DOE deserves credit for moving away from these apples-to-oranges comparisons. But NYC constructed elementary school and K-8 peer groups using a weighted formula of only four school characteristics - percent black and Hispanic combined, percent free lunch, percent special ed, and percent ELL. The junior high and high school peer groups were constructed using 4th and 8th grade scores, respectively. Each school is then assigned a "peer index" number, and the 20 schools above and below a school in the peer index serve as their "peer group."

How did this work out? Consider elementary school PS 196 in Queens, which received a B, as an example. PS 196 serves students who are:

• 40% Asian, but is being compared to schools that have between 1% and 69% Asian students (average=17.8%).
  
• 44.8% white, but is being compared to schools that have between 22.1% and 90% white students (average=66.6%)
  
• 2.1% African-American students, but is being compared to schools that have between 0% and 16.7% African-American students (average=3.96%).
  
• 13.1% Hispanic students, but is being compared to schools that have between 2.9% and 17.1% Hispanic students (average=10.4%)
  
• 14.6% free lunch students, but is being compared to schools that have between 2.4% and 26.8% free lunch students (average=15.5%).

Key inputs that are not in control of the school vary widely as well:

• PS 196 is at 106.4% of building capacity, but is being compared to schools that are between 51.6% and 136.5% of capacity (average=92.1%)
  
• 54.3% teachers at PS 196 have more than 5 years experience, but is being compared to schools that have between 35.7 and 81.3% teachers with more than 5 years experience (average=59.5)
  
• PS 196 has 658 students, but is being compared to schools that have between 180 and 1263 students (average=601).

The comparison groups were constructed differently at the high school level, where middle school test scores, not demographics, were used to create comparison groups. But do middle school test scores provide enough information to net out all factors that influence students' graduation?

Consider the Manhattan Center for Science and Mathematics, a selective school located in East Harlem that screens students not only on their test scores, but on their middle school grades and attendance. However, the Manhattan Center is not only compared with other schools that select students on their test scores, grades, and attendance; in fact, a handful of high schools with zoned programs are included in its peer group.

These schools have students with widely varing characteristics as incoming 9th graders. The Manhattan Center serves students who have the following characteristics as entering 9th graders:

• 61.2% proficient in reading in 9th grade, but is being compared to schools that have between 37.5 % and 100% proficient as entering 9th graders (average=70%) .
  
• 79.5% proficient in math in 9th grade, but is being compared with schools that have between 45.2% and 100% proficient in 9th grade (average=77%)
  
• 65% free lunch students, but is being compared to schools that have between 12.2% and 93.9% free lunch students (average=39.8%)
  
• 16.8% of students who are overage for grade (i.e. they have been held back), but is being compared to schools that have between 3.3% and 61% overage for grade (average=12.5%).
  
• 12.4% ELL, but is being compared to schools that have between 0 and 84.1% ELL (average=5.01%)
  
• 7.1% full-time special education, but is being compared to schools that have between 0 and 4.7% full-time special education (average=.8%)

There are two problems with this approach - one statistical and one practical:

1) As the comparisons above demonstrate, the peer groups falsely provide the illusion of fair comparison.

2) Perhaps more important for responses to these report cards - any educator who sits down with numbers will likely conclude that these comparisons lack face validity. And if these peer groups are not believeable to educators - that is, they don't feel that they have an equal chance of winning or losing this game - we are going to see enormous amounts of playing the system as schools attempt to succeed in a system that is perceived as fundamentally unfair.

Last Chance for NYC School Report Card Haiku Contest

our writers' workshop
activism by haiku
start a file on her

Yesterday, I set the goal of at least 6o haiku by the end of the day on Friday, and now we have 58! Push us over the edge. As promised, I'll format them as a lit mag so we all have a souvenir.

The haiku contest runs through the end of the day today (Friday) - I'll pick three I like and we'll vote next week. The only rules are the 5-7-5 syllable one, and to keep it playful and G rated.

Here are some submissions from yesterday:

The only numbers
that level the playing field
have a dollar sign
-Anonymous 10:54PM

Who would have thunk it?
A system that is worse than
No Child Left Behind
-Anonymous 10:38PM

It's only Kool-Aid
A little sip won't hurt you
said the Chancellor
-Anonymous 11:16PM

Cool teachers you should know: Lori Murphy, South Newton High School

Lori Murphy is a 15th year high school art teacher at South Newton High School in Kentland, Indiana. Ms. Murphy grew up in Rensselaer, Indiana and then studied art education at Ball State University. She has been teaching at South Newton ever since.

Her nominator, colleague Karen Molter, reports that Ms. Murphy has a classroom full of energy, and so much demand for her courses that she has to turn students away! Her courses are diverse, ranging from Photoshop and website production to 3-D art. Ms. Murphy is well-known for using e-learning in her classes and for supporting her colleagues in integrating technology (you can find her syllabi and a really neat virtual art gallery of her students' work here).

In her spare time, Ms. Murphy is an active artist, selling ceramics and photographs and working on professional website creation - check out some of her own work here (great holiday gift ideas). Her colleague sums up by saying, "Most importantly, however, Ms. Murphy does all these things with a smile and genuine concern for others."

Keep up the good work, Ms. Murphy!

Give us all the weekly dose of educational anti-depressants we need by nominating a great teacher today - send me an email at eduwonkette (at) gmail (dot) com including their name, school, subject/grade, and a few sentences explaining why you were moved to nominate them.

Cool people you should know: Jennifer Russell

There have been a number of pop books claiming that American childhood has fundamentally changed and is not just about "kid stuff" anymore - but what are academics saying about this question?

Enter Jennifer Russell, a new professor at the University of Pittsburgh School of Education. As a doctoral student at Berkeley's School of Education, Russell's dissertation research examined the extent to which kindergartens have shifted from a developmental model emphasizing social skills and play to a year of formal academic instruction. Part of her dissertation focused on how state and local policies, like testing and accountability, influence kindergarten teachers’ instructional practices. Great topic, and really important questions - we'll all stay tuned for her forthcoming papers.

Comment on "Lies My KIPP Teacher Told Me"

I've written on KIPP before (see posts archived here), so I wanted to respond to Jim Horn's post on KIPP. In summary, he contends that KIPP teachers force kids to participate in their own subjugation by promising opportunities that don't exist, no matter how hard they work or how nice they are.

Horn is tapping into a long lineage of writing criticizing teachers' endorsement of the "achievement ideology" - the idea that hard work and effort yield success irrespective of one's racial or class background. (The canonical works in this tradition are Learning to Labour: How Working Class Kids Get Working Class Jobs or Ain't No Makin' It: Aspirations and Attainment in a Low-Income Neighborhood.)

As a critique of the structure of opportunity and meritocracy, these works are convincing and important. But as texts to guide the action of educators on a day-to-day basis, they are not. It was at this theory/practice divide that I had a hard time with Horn's post. On KIPP, Horn writes:

For black parents, the KIPP appeal is the promise that for those who "work hard, be nice," there is a new world of opportunity waiting to embrace their efforts. Hope, however groundless, remains the only alternative to despair.

Horn implies that because of the obstacles that black kids are likely to bump up against, KIPP teachers should tell their kids to slack off and be mean. As a teacher, no matter how aware you are of broader structures of inequality, you want the best for your kids. And if you're a betting woman or man, you know that their odds of making a decent living are much better if they "work hard and act nice," even if the odds are still pretty bad.

Perhaps this post hit home because I got an unexpected call from a former student who I'll call Jannell yesterday. Jannell is easily one of the brightest students I ever taught, and should have had a college degree in hand years ago. But she doesn't, in part because of big picture inequality factors. Jannell was working her way through college since her parents couldn't pay and having a hard time balancing school and work. Then she fell in love with Troy, got married, and dropped out of school to support her husband's higher education. Now her husband has left her and she's stuck in a dead-end job answering phones.

In Jannell's story, there's a story about racial, socioeconomic, and gender inequality. And the truth is that if Jannell re-enrolls and "works hard and acts nice," she's still going to have a hard time making it in an increasingly bleak economy. While in the past Jannell and I have talked about race, class, and opportunity in America, yesterday I advised her to get back to college and work her butt off to finish - essentially, to work hard and be nice. Given the existing structure of opportunity, this will be the best thing for her. And ultimately, when you're on the other end of the line with a kid you care about, what you're concerned about is helping this kid make it, now.

It's easy for academics and bloggers (both of which I am) to stand outside the ring and hand down theory. It's harder to figure out how to integrate these big theories into the work of educating kids on a day-to-day basis. And KIPP is trying to find a way to make this system work, even if it's for a small subset of urban kids. We can criticize the pundits who latch on to KIPP for many of the reasons Horn notes.

But it's important to separate what KIPP supporters say about KIPP from what the educators at KIPP do. As I've written before, KIPP is not a viable solution for the vast majority of urban kids. Nonetheless, KIPP teachers are undoubtedly doing important work with the kids they're serving.

The KIPP teachers don't deserve the derision and disrespect that I saw in Horn's post. They can't change the existing structure of social inequality today, so they're trying to work within the system to get a few more kids a good education. And for that, they deserve our utmost respect.

Help Me Make a Children's Book Gift List!

The holidays are coming up fast, and I confess that I love to give children's books to adults and kids alike.

Anyway, can you help me by recommending your favorite new children's books (that would also make good presents)? I have an Olivia thing, so Olivia Helps With Christmas is one of my top picks for little people. Some others that I'm considering include:

Incredible Book-Eating Boy

The Chronicles of Narnia Pop-Up Book

Thank You Bear

Trail: Paper Poetry Pop-Up

Edwardo: The Horriblest Boy in the Whole Wide World

The Higher Power of Lucky

Alice's Adventures in Wonderland: A Pop-Up Adaptation (not new, but wonderful.)

Or for the more adult-oriented edu-giftee, try the New Yorker Book of Teacher Cartoons.

For a great blog on kiddie lit, check out "The Miss Rumphius Effect."

NYC Parent and Teacher Surveys: Who Responded?

Last spring, the NYC Dept of Education spent $2 million administering a survey to all parents, teachers, and students in the school system. What did the response rates look like by school, and did these response rates vary in a patterned way? Here’s what I found:

1) Parents and teachers at A schools were more likely to respond to surveys than parents and teachers at F schools. “A” schools had an average parent response rate of 31.75% versus 22.59% at F schools. The differences are relatively small for teachers: 48.23% versus 45.03%.

There’s something in this finding for everyone. One could argue that A schools are actually higher quality schools and more parents are involved as a result, which is why a higher proportion of parents responded. Alternatively, it may be that schools with lower levels of parent support are struggling for this very reason. The graph below, which plots the overall score for high schools against the parent response rate, provides some support for this point (correlation=.54).

2) Schools with higher proportions of Hispanic, African-American, and free lunch kids had much lower parental survey response rates, while those with higher proportions of White and Asian kids had much higher response rates.

To make the graph above, I divided schools into quintiles – five equally sized groups that each represent 20% of the schools. Schools in Quintile 1 have the lowest proportions of a given group, while schools in Quintile 5 have the highest proportions of that group. For example, Quintile 1 schools for the free lunch population have 48.7% free lunch or fewer, while Quintile 5 schools have 85.25% free lunch kids or more. For the Hispanic population, schools with 14.2% Hispanic kids or fewer are Quintile 1 schools, while those with 65.35% Hispanic kids or more are Quintile 5 schools.

3) This relationship holds for the teacher survey response rates. Schools with higher proportions of African-American and free lunch kids had significantly lower teacher survey response rates, while those with higher proportions of White and Asian kids had higher response rates.

Overall, these results raise the question of the validity of the parent surveys for any given school. 25% of schools had parent response rates of 18% or less (one school had a parent response rate of 1.97%!).

In addition, given how much these results vary by demographics, it is also not clear that we can validly compare actual survey responses across all schools. For example, is it meaningful to look at the safety and respect scores of a school with a parent response rate of 67% and compare those with a school with a response rate of 9%?

Lots of food for thought here – as always, email me if you'd like to see the full tables.

NYC School Report Card Groundhog Week

Last week, I wrote about NYC School Report Cards. (All posts archived here.) Though I was planning on looking at abstinence only sex education this week, there's a lot left to say about report cards. So here we are again. This week:

Monday: New York City Parent and Teacher Surveys: Who Responded?

Friday: Peer Comparison Schools - Now that the peer indices have been released, we can take a closer look.

In Progress: Extra Credit - Who Got It?