(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
The above is the question asked to me by Michael Stutzer, who writes:
I have attached an increasingly influential paper [“Effects of Adolescent Caffeine Consumption on Cocaine Sensitivity,” by Casey O’Neill, Sophia Levis, Drew Schreiner, Jose Amat, Steven Maier, and Ryan Bachtell] purporting to show the effects of caffeine use in adolescents (well, lab rats anyway) on biomarkers of rewards to cocaine use later in life. I’d like to see a Gelmanized analysis of the statistics. Note, for example, Figure 2, panels A and B. Figure 2, Panel A contrasts the later (adult) response to cocaine between 16 rats given caffeine as adolescents, vs. 15 rats who weren’t given caffeine as adolescents. Panel B contrasts the adult response to cocaine between 8 rats given caffeine only as adults, vs. 8 rats who weren’t given caffeine. The authors make much of the statistically significant difference in means in Panel A, and the apparent lack of statistical significance in Panel B, although the sign of the effect appears to still be there. But N=8 likely resulted in a much larger calculated standard error in Panel B than the N=16 did in Panel A. I wonder if the results would have held in a balanced design with N=16 rats used in both experiments, or with larger N in both. In addition, perhaps a Bonferroni correction should be made, because he could have just lumped the 24 caffeine-swilling (16 adolescent + 8 adult) rats together and tested the difference in the mean response between them and the 23 adolescent and adult rats who weren’t given caffeine. The authors may have done that correction when they contrasted the separate Panels differences in means (they purport to do that in the other panels), but the legend doesn’t indicate it.
Because the paper is getting a lot of citations, some lab should try to replicate all this with larger sample sizes, and perturbations of the experimental procedures.
My reply:
I don’t have the equipment to replicate this one myself, so I’ll post your request here.
It’s hard for me to form any judgment about the paper because all these biology details are so technical, I just don’t have the energy to track everything that’s going on.
Just to look at some details, though: It’s funny how hard it is to find the total number of rats in the experiment, just by reading the paper. N is not in the abstract or in the Materials and Methods section. In the Results section I see that one cohort had 32 adolescent and 20 adult rats. So there must be other cohorts in the study?
I also find frustrating the convention that everything is expressed as a hypothesis test. The big big trouble with hypothesis tests is that the p-values are basically uninterpretable if the null hypothesis is false. Just for example:
What’s the point of that sort of thing? If there’s a possibility there is no change, then, sure, I can see the merit of including the p-value. But when the p-value is .0001 . . . c’mon, who cares about the damn F statistic, just give me the numbers: what was the average fluid consumption per day for the different animals? Also I have a horrible feeling their F-test was not appropriate, cos I’m not clear on what those 377 and 429 are.
I’d like to conclude by saying two things that may at first seem contradictory, but which are not:
1. This paper looks like it has lots of statistical errors.
2. I’m not trying to pick on the authors of this paper. And, despite the errors, they may be studying something real.
The lack of contradiction comes because, as I wrote last month, statistics is like basketball, or knitting. It’s hard. There’s no reason we should expect a paper written by some statistical amateurs to not have mistakes, any more than we’d expect the local high school team to play flawless basketball or some recreational knitter to be making flawless sweaters.
It does not give me joy to poke holes in the statistical analysis of a random journal article, any more than I’d want to complain about Aunt Edna’s sweaters or laugh at the antics of whoever is the gawky kid who’s playing center for the Hyenas this year. Everyone’s trying their best, and I respect that. To point out statistical errors in a published paper is not an exercise in “debunking,” it’s just something that I’ll notice, and it’s relevant to the extent that the paper’s conclusions lean on its statistical analysis.
And one reason we sometimes want brute-force preregistered replications is because then we don’t have to worry about so many statistical issues.
The post Will youths who swill Red Bull become adult cocaine addicts? appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
