(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
Josh Miller writes:
Did you see this splashy NYT headline, “Surprising New Evidence Shows Bias in Police Use of Force but Not in Shootings”?
It’s actually looks like a cool study overall, with granular data, and a ton of leg work, and rich set of results that extend beyond the attention grabbing headline that is getting bandied about (sometimes with ill-intent). While I do not work on issues of race and crime, I doubt I am alone in thinking that this counter-intuitive result is unlikely to be true. The result: whites are as likely as blacks to be shot at in encounters in which lethal force may have been justified? Further, in their taser data, blacks are actually less likely than whites to subsequently be shot by a firearm after being tasered! While its true that we are talking about odds ratios for small probabilities, dare I say that the ratios are implausible enough to cue us that something funny is going on? (blacks are 28-35% less likely to be shot in the taser data, table 5 col 2, PDF p. 54). Further, are we to believe that suddenly, when an encounter escalates, the fears and other biases of officers suddenly melt away and they become race-neutral? This seems to be inconsistent with the findings in other disciplines when it comes to fear, and other immediate emotional responses to race (think implicit association tests, fMRI imaging of the amygdala, etc.).
This is not to say we can’t cook up a plausible sounding story to support this result. For example, officers may let their guard down against white suspects, and then, whoops, too late! Now the gun is the only option.
But do we believe this? That depends on how close we are to the experimental ideal of taking equally dangerous suspects, and randomly assigning their race (and culture?), and then seeing if police end up shooting them.
Looking at the paper, it seems like we are far from that ideal. In fact, it appears likely that the white suspects in their sample were actually more dangerous than the black suspects, and therefore more likely to get shot at.
Potential For Bias:
How could this selection bias happen? Well, this headline result comes solely from the Houston data, and for that data, their definition of a “shoot or don’t shoot” situation (my words) is defined as an arrest report that describes an encounter in which lethal force was likely justified. What is the criteria for lethal force to be likely justified? Among other things, for this data, it includes “resisting arrest, evading arrest, and interfering in arrest” (PDF pp.16-17, actual p. 14-15—they sample 5% of 16,000 qualifying reports) They also have a separate data set in which the criteria is that a taser was deployed (~5000 incidents). Remember, just to emphasize, these are reports involving encounters that don’t necessarily lead to an officer involved shootings (OIS). Given the presences of exaggerated fears, cultural misunderstandings, and other more nefarious forms of bias, wouldn’t we expect an arrest report to over-apply these descriptors to blacks relative to whites? Wouldn’t we also expect the taser to be over-applied to blacks relatively to whites? If so, then won’t this mechanically lower the incidence of shootings of blacks relative to whites in this sample? There are more blacks in the researcher-defined “shoot, or don’t shoot” situation that just shouldn’t be there; they are not as dangerous as the whites, and lethal force was unlikely to be justified (and wasn’t applied in most cases).
Conclusion:
With this potential selection bias, yet no discussion of it (as far as I can tell), the headline conclusion doesn’t appear to be warranted. Maybe the authors can do a calculation and find that the degree of selection you would need to cause this result is itself implausible? Who knows. But I don’t see how it is justified to spread around this result without checking into this (This takes nothing away, of course, from the other important results in the paper).
Notes:
The analysis for this particular result is reported on PDF pp. 23-25 with the associated table 5 on PDF p. 54. Note that when adding controls, there appear to be power issues. There is a partial control for suspect danger, under “encounter characteristics,” which includes, e.g. whether the suspect attacked, or drew a weapon—interestingly, blacks are 10% more likely to be shot with this control (not significant). The table indicates a control is also added for the taser data, but I don’t know how they could do that, because the taser data has no written narrative.
See here for more on the study from Rajiv Sethi.
And Justin Feldman pointed me to this criticism of his. Feldman summarizes:
Roland Fryer, an economics professor at Harvard University, recently published a working paper at NBER on the topic of racial bias in police use of force and police shootings. The paper gained substantial media attention – a write-up of it became the top viewed article on the New York Times website. The most notable part of the study was its finding that there was no evidence of racial bias in police shootings, which Fryer called “the most surprising result of [his] career”. In his analysis of shootings in Houston, Texas, black and Hispanic people were no more likely (and perhaps even less likely) to be shot relative to whites.
I’m not endorsing Feldman’s arguments but I do want to comment on “the most surprising result of my career” thing. We should all have the capacity for being surprised. Science would go nowhere if we did nothing but confirm our pre-existing beliefs. Buuuuut . . . I feel like I see this reasoning a lot in media presentations of social science: “I came into this study expecting X, and then I found not-X, and the fact that I was surprised is an additional reason to trust my result.” The argument isn’t quite stated that way, but I think it’s implicit, that the surprise factor represents some sort of additional evidence. In general I’m with Miller that when a finding is surprising, we should look at it carefully as this could be an indication that something is missing in the analysis.
P.S. Some people also pointed out this paper by Cody Ross from last year, “A Multi-Level Bayesian Analysis of Racial Bias in Police Shootings at the County-Level in the United States, 2011–2014,” which uses Stan! Ross’s paper begins:
A geographically-resolved, multi-level Bayesian model is used to analyze the data presented in the U.S. Police-Shooting Database (USPSD) in order to investigate the extent of racial bias in the shooting of American civilians by police officers in recent years. In contrast to previous work that relied on the FBI’s Supplemental Homicide Reports that were constructed from self-reported cases of police-involved homicide, this data set is less likely to be biased by police reporting practices. . . .
The results provide evidence of a significant bias in the killing of unarmed black Americans relative to unarmed white Americans, in that the probability of being {black, unarmed, and shot by police} is about 3.49 times the probability of being {white, unarmed, and shot by police} on average. Furthermore, the results of multi-level modeling show that there exists significant heterogeneity across counties in the extent of racial bias in police shootings, with some counties showing relative risk ratios of 20 to 1 or more. Finally, analysis of police shooting data as a function of county-level predictors suggests that racial bias in police shootings is most likely to emerge in police departments in larger metropolitan counties with low median incomes and a sizable portion of black residents, especially when there is high financial inequality in that county. . . .
I’m a bit concerned by maps of county-level estimates because of the problems that Phil and I discussed in our “All maps of parameter estimates are misleading” paper.
I don’t have the energy to look at this paper in detail, but in any case its existence is useful in that it suggests a natural research project of reconciling it with the findings of the other paper discussed at the top of this post. When two papers on the same topic come to such different conclusions, it should be possible to track down where in the data and model the differences are coming from.
P.P.S. Miller points me to this post by Uri Simonsohn that makes the same point (as Miller at the top of the above post).
In their reactions, Miller and Simonsohn do something very important, which is to operate simultaneously on the level of theory and data, not just saying why something could be a problem but also connecting this to specific numbers in the article under discussion.
The post About that claim that police are less likely to shoot blacks than whites appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
