(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
Macartan Humphreys sent me a Shiny app demonstrating you can get statistical significance from just about any pattern of random numbers. I posted it, and, in response, commenter Rahul wrote:
It sure is a cute demo but it’s a bit like insinuating a doctor’s stethoscope is useless by demonstrating ten ways in which it can be mis-used.
And, indeed, if the doctor’s stethescope were being used to routinely pubish spurious findings in the leading research journal in psychology; if leading figures in psychology such as Ted-talk star (and respected researcher) Daniel Gilbert were to vociferously defend ridiculous claims of fecundity and clothing which are based on nothing but a highly malleable theory and that doctor’s stethethscope; if the New York Times and various other news outlets reports a claim about ESP which is based, again, on nothing but that stethescope; if Steven Levitt, the leading voice in academic social science, lends his platform to endorse an innumerate claim about beauty and sex ratios, a claim that, you guessed it, is based on no evidence beyond what came from that stethoscope; if well-respected political scientist Larry Bartels posts on the leading political science research blog to promote a study on the effects of subliminal smiley-faces as “punching a big hole in democratic theory,” based on that stethoscope; if world-famous psychology researcher Daniel Kahneman uses the stethoscope to insist that “you have no choice but to accept that the major conclusions of these studies [on social priming] are true”; if brilliant economist James Heckman points to the stethoscope as evidence of large effects of early-childhood intervention programs (ironically, in doing so ignoring selection, the problem that made him famous); if all that is happening, then, yes, I’ll continue to explore what’s going wrong here.
You could of course take the quite reasonable position that Macartan Humphreys, Brian Nosek, etc., and I are wrong and that Daryl Bem, Daniel Gilbert, Steven Levitt, Satoshi Kanazawa, etc. are right. Fair enough—ultimately you have to make your own judgment. The point is, this is a live issue. It’s not just that the “stethoscope” could be misused; it’s that (in the judgment of myself and many others whom I respect) the stethoscope is being misused, all the time.
Fundamentally, the problem’s not (just) with p-values or with any particular technique. I see the problem as being with the entire hypothesis testing framework, with the idea that we learn by rejecting straw-man (or, as Dave Krantz charmingly said once, “straw-person”) null hypotheses, and with the binary true/false attitude which leads people to believe that, once a result is judged statistically significant (by any standard) and published in a good journal, that it deserves the presumption of belief.
The post Stethoscope as weapon of mass distraction appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
