(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
People keep pointing me to this excellent news article by David Brown, about a scientist who was convicted of data manipulation:
In all, 330 patients were randomly assigned to get either interferon gamma-1b or placebo injections. Disease progression or death occurred in 46 percent of those on the drug and 52 percent of those on placebo. That was not a significant difference, statistically speaking. When only survival was considered, however, the drug looked better: 10 percent of people getting the drug died, compared with 17 percent of those on placebo. However, that difference wasn’t “statistically significant,” either.
Specifically, the so-called P value — a mathematical measure of the strength of the evidence that there’s a true difference between a treatment and placebo — was 0.08. . . . Technically, the study was a bust, although the results leaned toward a benefit from interferon gamma-1b. Was there a group of patients in which the results tipped? Harkonen asked the statisticians to look.
It turns out that people with mild to moderate cases of the disease (as measured by lung function) had a dramatic difference in survival. Only 5 percent of those taking the drug died, compared with 16 percent of those on placebo. The P value was 0.004 — highly significant. . . . But there was a problem. This mild-to-moderate subgroup wasn’t one the researchers said they would analyze when they set up the study. . . .
Brown reports the other side:
Harkonen’s defenders also think that context is important.
The press release said the results for the primary endpoint weren’t statistically significant. . . . When the study was published in the New England Journal of Medicine in January 2004, the authors wrote that “a clinically significant survival benefit could not be ruled out.”
In short, Harkonen’s defense team argued that nobody was deceived by the press release. . . . Steven Goodman, the pediatrician and biostatistician, believes that “context” also includes what IPF patients are thinking.
“Part of the issue goes to the level of proof a patient would need when facing a fatal disease with no treatment,” he said. The fact that the study’s main result barely missed statistical significance “is far from proof that the treatment didn’t work. And if I were a patient, I would want to know that.”
And here’s where the story ends:
Things haven’t turned out too well for interferon gamma-1b, either.
InterMune did run another trial. It was big — 826 patients at 81 hospitals — in order to maximize the chance of getting clear-cut results. It enrolled only people with mild to moderate lung damage, the subgroup whose success was touted in the press release.
And it failed. A little more than a year into the study, more people on the drug had died (15 percent) than people on placebo (13 percent). . . . It’s possible that there’s a subgroup of patients, not yet fully identified, who benefit. For example, data suggest that people whose cells still make a fair amount of interferon gamma are helped by the high-priced drug. But it’s unlikely anyone will be trying to figure that out soon.
Certainly not Scott Harkonen.
Things could be worse, though. They used to have the death penalty for forgery.
The post On house arrest for p-hacking appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
