NYC School Report Cards II: A Closer Look

One of the most striking patterns from my post yesterday was that schools with higher report card grades, on average, had higher proportions of Asian kids than those with lower grades. This raised serious questions for me about the validity of the comparison groups, which were generated using only four school characteristics: the combined African-American and Hispanic population, percent free/reduced lunch, percent special ed, and percent ELL.

From national datasets like the Early Childhood Longitudinal Study, it's clear that Asian kids have different growth trajectories throughout elementary school. It's also clear that Hispanic and African-American kids have very different growth trajectories, with African-American kids falling behind at a much faster rate. If a large proportion of the report card is based on growth, but the growth measures don't account for varying trajectories that are, in part, the result of non-school factors, schools serving higher proportions of Asian kids will look like they're producing more growth. Similarly, schools serving higher proportions of Hispanic kids will look like they are producing more growth if they are compared to schools serving similar proportions of African-American kids. [We can debate how much of these differences in trajectories are explained by school versus outside-of-school factors.]

Here's what I found (I'll interpret a whole line of the table so I'm clear on what these measures are):

• At the elementary school level, A schools have an average of 20.99% Asian students. By contrast, the median A school has 9.55% Asian students. This tells us that the average is pulled up by schools with very high proportions of Asian kids. The standard deviation is there for stats junkies, but most readers will prefer to look at the interquartile range (the 25% and 75% columns) to get a sense of how much variation there is. If we read across the A row, the 25% column tells us that 25% of A schools have 2.3% Asian or fewer. Similarly, the 75% column says that 25% of A schools have 36.8% Asian students or more. The range column represents the lowest and highest values for a given grade; A schools have between .2 and 92.6% Asian students.


• If we compare medians instead of averages, we still see that A elementary schools have more than 3 times as many Asian students than F schools (9.55% versus 2.90%). The 25% of A schools with the highest Asian population have between 36.8% and 92.6% Asian students, while the 25% of F schools with the highest Asian populations have 5.3% Asian students and 28.5% Asian students.


• The general pattern is similar across all levels of schooling; A schools have substantially higher proportions of Asian students.

The tables below show the distribution of Asian students by school grade and school level (i.e. elementary, middle, K-8, and HS):

* 3 elementary schools had missing data in 2005 and are thus not represented in the table above.

* Because many 6-12 schools opened recently and thus do not have 2005 data available, the middle school data are missing 32 schools, including 10 A schools, 14 B schools, 5 C schools, 1 D school , and 2 F schools, and thus should be interpreted with this in mind.

* 1 K-8 school had missing data in 2005 and is not represented in the table above.
* 1 high school had missing data in 2005 and is not represented in the table above.

Unless we believe that schools with high concentrations of Asian students are higher quality schools, these results raise a lot of questions about the validity of these comparison groups. In the multivariate analyses that I describe below, I also find that schools with higher proportions of Hispanic students are more likely to receive A or B grades - which raises the question of the accuracy of using an aggregate black/Hispanic number to compare schools.

Overall, these results suggest that the Dept of Ed's method of establishing comparisons groups ultimately results in apple-to-orange comparisons.

For geekier analyses, read on:

For those who are interested, I also ran a series of descriptive logistic regressions for the purpose of examining the association between school racial composition and schools' odds of earning an A or B grade, net of many other factors that could explain this association. In these models, I controlled for percent free lunch, percent female, percent immigrant, percent stability (i.e. the opposite of mobility), percent full and part-time special education, percent ELL, school size, percent capacity (how crowded the school is), and teacher characteristics (percent with more than 5 years teaching and percent with a masters degree). I didn't impute missing values, so these analyses include 970 schools of the 1187 that had data available in 2005. Remember, regressions like these are just descriptive, not causal [that is, they describe patterns observed in the data and do not necessarily explain *why* a school received the grade it did]. Nonetheless, unless we believe that schools with the highest concentrations of Hispanic and Asian students are much higher quality than those with lower concentrations of these students, these results suggest that the peer comparison groups are not entirely fair:

• First, I divided schools into four equal groups - quartiles - based on their percent Asian. (This is a typical approach to modeling non-linearities.) The first quartile included the 25% of schools that have the lowest proportions of Asian students, and the fourth quartile included the 25% of schools that have the highest proportions of Asian students - in these analyses, quartile 4 includes >15.3% Asian. I did the same thing with the African-American and Hispanic populations, i.e. divided schools into four quartiles.
  

• When we just examine the association between racial composition and a schools' odds of getting an A or a B, quartile 4 schools (those with the most Asian students) are almost 2.5 times more likely to get an A or B (odds ratio=2.44, p=.001) compared to those with the lowest proportions of Asian students (those with 1.5 percent Asian or less) . However, schools in the 2nd and 3rd quartiles of the Asian population have no advantage over those in the first. When we just look at the relationship between getting and A/B and African-American and Hispanic composition, we see no stastically significant relationship.

• Once we control for all variables listed above, quartile 4 Asian schools have a smaller advantage (they are slightly less than twice as likely to get an A or B (odds ratio=1.87, p=.041). Again, there is no advantage of being in the 2nd or 3rd quartile of the Asian population over the first. But schools in quartile 4 of the Hispanic population have an even larger advantage - they are slightly more than two times as likely to get an A or B (odds ratio=2.12, p=.036).

• Nonetheless, this full set of predictors only explains a tiny proportion of the variance (pseudo R2=.06).

If anyone is interested in checking out the full results, email me and I'll send you the output.