(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
Lee Jussim pointed me to the recent article in Psychological Science by Joseph Simmons and Uri Simonsohn, expanding on their blog post on flaws in the notorious power pose article. Jussim then commented:
I [Jussim] think that Cuddy/Fiske world is slowly shrinking. I think your “What Has Happened Here…” post was:
1. A bit premature ONLY in its claim that Fiske was working in a dead paradigm
BUT
2. Inspirationally aspirational — that is Exactly what most of us are shooting for — to kill that paradigm.
and, therefore
3. Perhaps that claim in your post, though not descriptive, may actually be prescient.
In that spirit, you may enjoy this killer unpubbed paper by Vazire:
And you can find a great turn of phrase in this Freakonomics Podcast:
“We were practicing eminence-based medicine” which I have used to frame as a question I sometimes use in talks that you might enjoy:
“Do we want an eminence-based psychology or an evidence-based psychology?”
I replied that I found Vazire’s article inspirational and I agree with its general flow, but I have some concerns.
First, I could imagine the current problems with reproducibility arising even in a completely eminence-free environment. To me, it just seems like there are basic misunderstandings of statistics, along with a desire for certainty. Indeed, when Vazire writes, “Scientists constantly struggle with the tension between human desires and scientific ideals. Why make it harder for them by giving these human desires an even stronger platform?”, this makes me think of the human desire for certainty, which is amplified by statistical methods of null hypothesis significance testing that inappropriately transmute uncertainty into certainty.
I also wonder about the statement in the paper that heterosexuals are overrepresented at high levels of status. This could be true, I guess; I’ve just never even seen this claimed before.
I also am not so sure about the claim that we can “reliably distinguish rigorous science from shoddy science.” I guess I’d have to know how these are defined.
Also it’s funny to see the link to a Freakonomics podcast given that Freakonomics has a habit of hyping junk science! We have to take the good where we find it, I guess.
I also cc-ed the above to Simine Vazire, who wrote:
I [Vazire] completely agree with you that the human drive for certainty would also lead to shortcut-taking and therefore to reproducibility problems, without adequate checks in the system (which we currently lack – see another paper of mine here: http://collabra.org/articles/10.1525/collabra.74/ )
Regarding whether we can reliably detect rigorous science from shoddy science, I think we can do a decent job if we learn our lessons from the replicability crisis. I think we now know that there are some pretty reliable indicators of rigor, based on meta-science, simulations, and sheer logic (e.g., pre-registration is good for reasons that don’t need to be empirically demonstrated, though the empirical evidence also supports the view that results from pre-registered studies are more accurate – e.g., effect sizes are less inflated).
Regarding whether heterosexuals are overrepresented in high status positions – I admit I was extrapolating here. I think there is pretty good evidence that sexual orientation is associated with discrimination (with heterosexuals receiving better treatment; though admittedly I am not an expert on this literature), and I am extrapolating from that to status. But of course it’s an empirical question and perhaps I should have been more careful. I’d be curious to know the data, and I’d bet money that, compared to proportions at lower levels of status within academia, heterosexuals are overrepresented at high levels of status. Perhaps not compared to proportions in the general population though, since I suspect sexual orientation minorities may be more likely to select into academia in the first place.
Jussim then added:
Here is why I think Simine’s article is killer, regardless of whatever imperfections it may have or disagreements I have with details:
1. She is absolutely right that the field is gaga-eyed over eminence. Fiske’s “bullying” essay only got the play it did because she is famous. But this problem is even worse in peer review. You all know the old Peters & Ceci paper? It is amazing. And it is the best concrete evidence for exactly Simine’s main point (hey, Simine, I just noticed this paper is not cited . . . ?). It shows that the exact same paper which had previously been published by someone at a prestigious institution, when resubmitted as a manuscript that had not been published and was authored by people at low status sounding institutions, was almost always rejected.
Points that I believe to be true based on anecdotal evidence and experience:
2. The old boy/girl network is alive and well, though it has morphed with the times. Connections among eminent authors, editors and reviewers creates a hidden track for publication among the eminent that the rest of us mortals do not usually have.
3. It is even more severe with grants.
None of this is absolute. Yes, NSF in 1998 or 2005 might reject Fiske’s or Bargh’s proposal, but they are far less likely to have done so than if the identical proposal had been submitted by someone less eminent.
4. Andrew’s points and Simine’s are not mutually exclusive. Scientists have many motivations that are either orthogonal to, or even in conflict with, the search for truth. Eminence is one. Certainly, science can go wrong out of ignorance, as well. I suspect, however, that some of that is motivated or willful ignorance.
5. The Identification of Shoddy Science question is an interesting one. As this empirical question, say, “Can most social psychologists distinguish good from bad social psychology?” — IDK. As this empirical question: “Can distinguishing good from bad research be done using known methods?” — I am pretty sure the answer is yes. Maybe not perfectly, but pretty well.
And Vazire then took the discussion in a slightly different direction:
I know the Peters & Ceci paper, and forgot to cite it! The sample size is quite small, but still provides evidence that should at least make us think… I still run into a lot of people who are adamantly against blinding. Here’s some more anecdotal evidence: since I started blinding myself (as editor) to authors’ identities before making desk reject decisions, it’s been a completely different (and somewhat frightening) experience. It just feels so so different. Anyone who’s against it should try it both ways for a few hundred manuscripts – I’d love to see if anyone can do that and then tell me that it makes no difference.
The post Discussion with Lee Jussim and Simine Vazire on eminence, junk science, and blind reviewing appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
