Using statistics to make the world a better place?

(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)

In a recent discussion involving our frustration with crap research, Daniel Lakeland wrote:

I [Lakeland] really do worry about a world in which social and institutional and similar effects keep us plugging away at a certain kind of cargo-cult science that produces lots of publishable papers and makes it easier to get funding for projects that don’t really promise to give us fundamental and predictive models that can drive real improvements in people’s lives.

It’s sort of a “it’s 2014, where’s my flying car?” attitude I know, but I’d be satisfied with a lot of things other than flying cars, such as:

1) real, effective solutions to antibiotic resistant organisms
2) cures for cystic fibrosis
3) reducing the effect of heart disease on people under age 75 by 30%
4) understanding major causes of “the obesity epidemic” in a real detailed way and finding effective ways to reverse it.
5) Being able to regenerate organic replacement joint components instead of titanium hip implants etc
6) Growing replacement kidneys
7) A more significantly more effective and long lasting pertussis vaccine

Is the way we are doing science today going to provide any or all of these things in the next 30 years? What are some similar order of magnitude things that it has provided since 1980 using current “modern” methods and funding priorities, publication priorities, tenure systems, and so forth?

I want to consider this question, difficult as it is. It does seem that slow progress is being made in our conceptual understanding, at least in the sense that new paradigms are developing in different sciences. In sociology there’s an increased interest in network effects, in medicine it seems that a lot of things are being now understood in terms of the microbiome, in large-scale biology there’s a focus on the fetal environment, in microbiology we’ve been hearing a lot about the rapid evolution of viruses, etc. It seems to me, as an outsider in these fields, that the concepts I’ve mentioned have been around for awhile, but it’s only recently that they’ve been moved to the center of discussion. And this, in turn, seems to be the result of millions of research studies on many thousands of topics, which collectively ruled out earlier favored explanations. By revealing areas where the old paradigms failed, all this research made room for new, more difficult, paradigms to be taken seriously.

It’s posterior predictive checking, on a larger scale. A chicken is an egg’s way of constructing another egg, and empirical research is a scientific theory’s way of uncovering the theory’s flaws.

It’s harder to make such clear statements about statistics or engineering or computer science, as these are essentially tools in the service of science rather than being the object of study themselves.

And there are fields where new paradigms don’t seem so apparent. Consider three areas with which I’m somewhat familiar: political science, psychology, and economics. In political science, I see persistent difficulties in integrating different perspectives coming from the studies of public opinion, institutions, and political maneuvering. It really feels like we’re not seeing the whole elephant at once. And I include my own research as an example of this incomplete perspective. Psychology seems to be undergoing a reforming process, in which various unsuccessful paradigms such as embodied cognition are being rejected, with no clear unification of the cognitive and behavioral approaches. Similarly in economics, although there it seems worse in that various incomplete perspectives are taken by their proponents as being all-encompassing.

The point of that previous paragraph is not to pass judgment on these different social science fields; it’s just my impression that they are in various stages of reconstruction and reform. From my outsider’s perspective, biology and medicine seem to be in better shape, philosophically speaking, in that there seems to be more of a recognition that traditionally dominant paradigms miss a lot.

How does statistics fit into all this? Statistics can (potentially) do a lot:
– Guidance in data collection and the assessment of measurements. And recall that “data collection” is not just about how to collect a random sample or assign treatments in an experiment; it also includes considerations of what to measure and how to measure it. For example, if you are interested in measuring the most fecund days of a woman’s monthly cycle, it makes sense to check out the medical literature on the topic.
– Methods for calibrating variation by comparing to models of randomness. This is where I think that statistical significance and p-values fit in: not as a way to make scientific discoveries (“p less than .05 so we get published in the tabloids!”) but as a measuring stick when interpreting observed comparisons and variation.
– Tools for combining information. That to me is the most general way to think of “inference,” and it encompasses all sorts of things, from classical “iid” models to more complicated approaches including all sorts of time series and spatial analyses, multilevel models for partial pooling, regularization methods that allow users to ramp up the amount of data they can include in their models, and Bayesian inference for balancing uncertainties.
– Methods for checking fit, for revealing the aspects of data that are not well explained by our models. To me this includes all of exploratory data analysis, which is about learning the unexpected. Or, to step back slightly, exploratory data analysis is about putting us in a situation in which we are able to learn the unexpected. And, remember, “unexpected” = “not expected” = “something not predicted by our model,” where this “model” may be implicit.

Again, statistics is in the service of science, and I see statistics as a way of organizing science rather than as a way of making scientific discovery.

Sometimes we can perform a statistical analysis that seems to take us all the way from data to model to scientific conclusion—for example, this cool age-period-cohort model with Yair—but, even there, I think most of the real scientific heavy lifting is coming from existing substantive theories; the statistics is more of a way of rearranging the data or, as Dan Kahan would put it, of adjudicating between competing hypotheses or underlying models of reality.

The post Using statistics to make the world a better place? appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

Tags: Decision Theory, Miscellaneous Statistics, Political Science, Public Health