(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
I got a little bit of pushback on my recent post, “The difference between ‘significant’ and ‘not significant’ is not itself statistically significant: Education edition”—some commenters felt I was being too hard on the research paper I was discussing, because the research wasn’t all that bad, and the conclusions weren’t clearly wrong, and the authors didn’t hype up their claims.
What I’d like to say is that it is OK to criticize a paper, even it isn’t horrible.
We’ve talked on this blog about some papers that are just terrible (himmicanes) or that are well-intentioned but obviously wrong (air pollution in China) or with analyses that are so bad as to be uninterpretable (air rage) or which have sample sizes too small and data too noisy to possibly support any useful conclusions (beautiful parents, ovulation and voting) or which are hyped out of proportion to whatever they might be finding (gay gene tabloid hype) or which are nothing but a garden of forking paths (Bible Code, ESP). And it’s fine to blow these papers out of the water.
But it’s also fine to present measured criticisms of research papers that have some value but also have some flaws. And that’s what I did in that earlier post. As I wrote at the time:
Just to be clear, I’m not trying to “shoot down” this research article nor am I trying to “debunk” the news report. I think it’s great for people to do this sort of study, and to report on it. It’s because I care about the topic that I’m particularly bothered when they start overinterpreting the data and drawing strong conclusions from noise.
If my goal were to make a series of airtight cases, destroying published paper after published paper, then, yes, it would make sense to concentrate my fire on the worst of the worst. But that’s not what it’s all about. “The difference between ‘significant’ and ‘not significant’ is not itself statistically significant” is a real error, it’s an important error, and it’s a frequent error—and I think it’s valuable to point out this error in the context of a paper that’s not trash, reported on in a newspaper that’s not prone to sensationalism.
So, you commenters who told me I was being too harsh: What you’re really saying is that methods criticisms should be reserved for papers that are terrible. But I disagree. I say it can be helpful to criticize some of the reasoning of a paper on methodological grounds, even while other aspects of the paper are fine.
Criticism and review are all about advancing science, not about making airtight cases against particular papers or particular bodies of work.
The post It’s ok to criticize appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
