(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
We’ve spent a lot of time during the past few years discussing the difficulty of interpreting “p less than .05” results from noisy studies. Standard practice is to just take the point estimate and confidence interval, but this is in general wrong in that it overestimates effect size (type M error) and can get the direction wrong (type S error).
So what about noisy studies where the p-value is more than .05, that is, where the confidence interval includes zero? Standard practice here is to just declare this as a null effect, but of course that’s not right either, as the estimate of 0 is surely a negatively biased estimate of the magnitude of the effect. When the confidence interval includes 0, we can typically say that the data are consistent with no effect. But that doesn’t mean the true effect is zero; it could mean that we should bring more data to bear on the question.
Jeff points us to a recent example, presented in this letter from Elizabeth Hatch, Lauren Wise, and Kenneth Rothman:
I’m not so sure. I agree that the journal should not describe this result as “indicating no effect,” but it’s not quite clear what should be said instead, beyond simply reporting the confidence interval and letting readers go from there.
The journal’s phrase is “the absence of a clear benefit.” It would be more accurate to say “absence of clear evidence of a net benefit,” but that’s a pretty minor difference.
Given that the confidence interval does overlap with zero, I’m not sure why Hatch et al. are so bothered by the journal’s characterization of the results. I’m guessing that they (Hatch et al.) have other evidence, not from that published study, which they feel supports the idea that this treatment should have a large effect. That is, they have prior information. But, as I said, I can’t be sure, because I don’t know this area at all.
Speaking in the abstract, I agree with Hatch et al. that 30% is not zero; on the other hand, I wouldn’t expect to see an effect as large as 30% in a replication.
Given the information above, the best estimate of the effect in the general population is somewhere between 0 and 30%. And the data are also consistent with a zero or even a negative effect (in the parameterization of the letter above, a hazard ratio of 1 or higher).
Looking forward, it seems to me that the next step is to explicitly include more information in the decision process. I agree with the journal that the confidence interval of [0.47, 1.02] indicates enough uncertainty that, given the data in that experiment alone, we shouldn’t act as if we’re sure that the treatment has a net benefit—and we certainly shouldn’t go around acting as if the effect is 30%. But the topic is important, so let’s bring more information to the table. The problem, as I see it, is not that the journal made any mistakes in conveying the evidence; rather, the problem is with the attitude that a single noisy study should be considered as dispositive.
To put it another way: Had the original paper reported an effect size estimate of 30% with p=.04, I’d be skeptical: I’d say that I’d guess the 30% was an overestimate and that we should be aware that treatment effects can vary. With p=.06, I’m just very slightly more skeptical (or maybe less skeptical; see discussion here). I don’t think the experiment should be thrown away, but I see where the journal is coming from, in emphasizing that there’s no clear evidence from these data alone.
P.S. I don’t agree with Hatch et al. when they write, “Given the expected bias toward a null result that comes from non-adherence coupled with an intent-to-treat analysis . . .” Yes, such estimates are biased toward zero, but that makes a true population effect size of 30% even less plausible, and it remains that, with the confidence interval as reported, such a difference could have been explained by chance alone. Having a more biased or noisier estimate does not in any way increase the strength of the results. Eric Loken and I discuss this issue in our post on the “What does not kill my statistical significance makes it stronger” fallacy.
The post How to interpret “p = .06” in situations where you really really want the treatment to work? appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
