(This article was originally published at Econometrics Beat: Dave Giles' Blog, and syndicated at StatsBlogs.)
Like a lot of others, I follow Andrew Gelman's blog with great interest, and today I was especially pleased to see this piece relating to a recent study on the extent to which researchers do or do not interpret confidence intervals correctly.
If you've ever taught an introductory curse on statistical inference (from a frequentist, rather than Bayesian perspective), then I don't need to tell you how difficult it can be for students to really understand what a confidence interval is, and (perhaps more importantly) what it isn't!
It's not only students who have this problem. Statisticians acting as "expert witnesses" in court cases have no end of trouble getting judges to understand the correct interpretation of a confidence interval. And I'm sure we've all seen or heard empirical researchers misinterpret confidence results! For a specific example of the latter, involving a subsequent Nobel laureate, see my old post here!
The study that's mentioned by Andrew today was conducted by four psychologists (Hoekstra et al., 2014) and involved a survey of academic psychologists at three European Universities. The participants included 442 Bachelor students, 34 Master students, and 120 researchers (Ph.D. or faculty members).
Yes, the participants in this survey are psychologists, but we won't hold that against them, and my hunch is that if we changed "psychologist" to "economist" the results wouldn't alter that much!
Before summarizing the findings of this study, let's see what the authors have to say about the correct interpretation of a confidence interval (CI) constructed from a particular sample of data:
"Before proceeding, it is important to recall the correct definition of a CI. A CI is a numerical interval constructed around the estimate of a parameter. Such an interval does not, however, directly indicate a property of the parameter; instead, it indicates a property of the procedure, as is typical for a frequentist technique. Specifically, we may find that a particular procedure, when used repeatedly across a series of hypothetical data sets (i.e., the sample space), yields intervals that contain the true parameter value in 95 % of the cases. When such a procedure is applied to a particular data set, the resulting interval is said to be a 95 % CI. The key point is that the CIs do not provide for a statement about the parameter as it relates to the particular sample at hand; instead, they provide for a statement about the performance of the procedure of drawing such intervals in repeated use. Hence, it is incorrect to interpret a CI as the probability that the true value is within the interval (e.g., Berger & Wolpert, 1988). As is the case with p-values, CIs do not allow one to make probability statements about parameters or hypotheses." (Hoekstra et al., 2014, 2nd. page of online pre-print.)
For what it's worth, I agree that this description and interpretation of a CI is correct.
I'm not saying that we should be using CI's. Specifically, when I'm wearing my Bayesian hat, CI's make no sense at all, and the very term is banished from my vocabulary. But I digress.........
So, what are the findings of the study in question? Very briefly (because you should read the paper yourself):
Participants were given 5 6 incorrect statements about a confidence interval, and were asked which ones , if any were correct. 8 undergraduate students (1.8%), 0 Masters students, and 3 (2.5%) Ph.D./faculty correctly said that all five six statements were incorrect. The claimed level of experience of the respondents had a slight positive correlation with the extent to which misinterpretations of CIs were made. Researchers (Ph.D. and faculty) scored about as well as first-year students without any training in statistics.
Very much a case of "read it and weep"!
However,....... check the survey questions in the Appendix of the Hoekstra et al. paper, and see how you score.
Very much a case of "read it and weep"!
However,....... check the survey questions in the Appendix of the Hoekstra et al. paper, and see how you score.
References
Berger, J. O. and R. L. Wolpert, 1988. The Likelihood Principle (2nd. ed.), Institute of Mathematical Statistics, Hayward, CA.
Hoekstra, R., R. D. Morey, J. N. Rouder, and E-J. Wagenmakers, 2014. Robust misinterpretation of confidence intervals. Psychonomic Bulletin Review, in press.
Please comment on the article here: Econometrics Beat: Dave Giles' Blog
