(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
Tom Daula points to this news article by Heidi Vogt, “EPA Wants New Rules to Rely Solely on Public Data,” with subtitle, “Agency says proposal means transparency; scientists see public-health risk.” Vogt writes:
The Environmental Protection Agency plans to restrict research used in developing regulations, the agency said Tuesday . . . The new proposal would exclude the many research studies that don’t make their raw data public and limit the use of findings that can’t be reproduced by others. The EPA said this would boost transparency. . . .
The move prompted an uproar from scientists who say it would exclude so much research that the resulting rules would endanger Americans’ health. Ahead of the announcement, a coalition of 985 scientists issued a statement decrying the plan.
“This proposal would greatly weaken EPA’s ability to comprehensively consider the scientific evidence,” they said in a letter issued Monday. The group said the EPA has long been very transparent in explaining the scientific basis for decisions and that requiring public data would exclude essential studies that involve proprietary information or confidential personal data. . . .
The administrator made his announcement flanked by two lawmakers who introduced that legislation: Sen. Mike Rounds (R., S.D.) and Rep. Lamar Smith (R., Texas).
Mr. Smith has argued that confidential data such as patient records could be redacted or given only to those who agree to keep it confidential.
Scientists have said this sort of process would still exclude many studies and make others costly to use in regulation. Gretchen Goldman, research director for the Center for Science and Democracy, has said studies are already rigorously reviewed by scientific journals and that those peer reviews rarely require raw data to assess the science.
Richard Denison, lead scientist at the Environmental Defense Fund, said the rule could exclude studies that track real-life situations that it would be unethical to reproduce. He gave as an example the monitoring of the Deepwater Horizon oil spill in the Gulf of Mexico in 2010.
“The only way to reproduce that work would be to stage another such oil spill, clearly nonsensical,” he said in a statement.
As for providing all the raw data, Mr. Denison said that would prevent the use of medical records that must be kept confidential by law.
The American Association for the Advancement of Science—-the world’s largest general scientific society and the publisher of the journal Science—said the rule would also exclude many studies that rely on outside funders, because they sometimes limit access to the underlying data.
Daula expressed this view:
If journals required data and code to replicate then it wouldn’t matter. Having a big player demand such transparency may spur journals to adopt such a policy. Thoughts? Controversial politically, but seems in line with ideas advanced on your blog.
I have mixed feelings about this proposal. Overall it seems like a good idea, as long as exceptions for special cases are carved out.
1. Going forward, I strongly support the idea that decisions should be made based on open data and reproducible studies.
2. That said, there are lots of decisions that need to be made based on existing, imperfect studies. So in practice some compromises need to be made.
3. Regarding the example given by the guy from the Environmental Defense Fund, I don’t know how the monitoring was done of the Deepwater Horizon oil spill. But why can’t these data be open, and why can’t the analysis be reproducible?
4. There seems to be some confusion over the nature of “reproducibility,” which has different meanings in different contexts. A simple psychology experiment can actually be reproduced (although there’s never such a thing as an exact replication, given that any attempted replication will include new people and a new context). In some examples of environmental science, you can re-run a lab or field experiment; in other cases (as when studying global warming or massive oil spills), there’s no way to replicate. But the data processing and analysis should still be replicable. I haven’t seen the proposed EPA rules, so I’m not sure what’s meant by “limit the use of findings that can’t be reproduced by others.”
I’d hope that for a study such as the Deepwater Horizon monitoring, there’s be no requirement that a new oil spill be reproduced—but it does seem reasonable for the data to be fully available and the data processing and analysis be replicable.
5. I’m disappointed to see the research director for the Center for Science and Democracy saying that studies are already rigorously reviewed by scientific journals and that those peer reviews rarely require raw data to assess the science.
No kidding, peer reviews rarely require raw data to assess the science! And that’s a big problem. So, no, I don’t think the existence of purportedly rigorous peer review (if you want an environmental science example, see here) is any reason to dismiss a call for open data and reproducibility.
Also, I’d think that any organization called the “Center for Science and Democracy” would favor openness.
6. I can understand the reasoning by which these science organizations are opposing this EPA plan: The current EPA administrator is notorious for secrecy, and from newspaper reports it seems pretty clear that the EPA is making a lot of decisions based on closed-doors meetings with industry. But, if the problem is a closed, secretive government, I don’t think the solution is to defend closed, secretive science.
7. Specific objections raised by the scientists were: (a) “requiring public data would exclude essential studies that involve proprietary information or confidential personal data,” and (b) “rule would also exclude many studies that rely on outside funders, because they sometimes limit access to the underlying data.” I suppose exceptions would have to be made in these cases, but I do think that lots of scrutiny should be applied to claims based on unshared data and unreplicable experiments.
The post Proposed new EPA rules requiring open data and reproducibility appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
