There’s an update (with overview) on the infamous Harkonen case in Nature with the dubious title “Uncertainty on Trial“, first discussed in my (11/13/12) post “Bad statistics: Crime or Free speech”, and continued here. The new Nature article quotes from Steven Goodman:
“You don’t want to have on the books a conviction for a practice that many scientists do, and in fact think is critical to medical research,” says Steven Goodman, an epidemiologist at Stanford University in California who has filed a brief in support of Harkonen……
Goodman, who was paid by Harkonen to consult on the case, contends that the government’s case is based on faulty reasoning, incorrectly equating an arbitrary threshold of statistical significance with truth. “How high does probability have to be before you’re thrown in jail?” he asks. “This would be a lot like throwing weathermen in jail if they predicted a 40% chance of rain, and it rained.”
I don’t think the case at hand is akin to the exploratory research that Goodman likely has in mind, and the rain analogy seems very far-fetched. (There’s much more to the context, but the links should suffice.) Lawyer Nathan Schachtmen also has an update on his blog today. He and I usually concur, but we largely disagree on this one[i]. I see no new information that would lead me to shift my earlier arguments on the evidential issues. From a Dec. 17, 2012 post on Schachtman (“multiplicity and duplicity”):
So what’s the allegation that the prosecutors are being duplicitous about statistical evidence in the case discussed in my two previous (‘Bad Statistics’) posts? As a non-lawyer, I will ponder only the evidential (and not the criminal) issues involved.
“After the conviction, Dr. Harkonen’s counsel moved for a new trial on grounds of newly discovered evidence. Dr. Harkonen’s counsel hoisted the prosecutors with their own petards, by quoting the government’s amicus brief to the United States Supreme Court in Matrixx Initiatives Inc. v. Siracusano, 131 S. Ct. 1309 (2011). In Matrixx, the securities fraud plaintiffs contended that they need not plead ‘statistically significant’ evidence for adverse drug effects.” (Schachtman’s part 2, ‘The Duplicity Problem – The Matrixx Motion’)
The Matrixx case is another philstat/law/stock example taken up in this blog here, here, and here. Why are the Harkonen prosecutors “hoisted with their own petards” (a great expression, by the way)?
The reasoning seems to go like this: Matrixx could still be found guilty of securities fraud for failing to report adverse effects related to its over-the-counter drug Zicam, even if those effects were non-statistically significant. If non-statistically significant effects should have been reported in the Matrixx case, then Harkonen’s having reported the non-statistically significant subgroup is in sync with the government’s requirement. To claim that Matrixx should report non-statistically significant risks, and then to turn around and claim that Harkonen should not report non-statistically significant benefits is apparently inconsistent.
Really?
The two cases are importantly disanalogous on a number of grounds. Specifics can be found in the Matrixx posts cited above.In fact, one might argue that the Matrixx case actually strengthens the case against Harkonen. Giving an overly rosy picture of, or downplaying, information about potential regulatory problems with a drug is likely to be deceptive for investor assessment. Even moving away from the fact that Matrixx concerns security fraud, and granting that the ruling (by the Supreme Court) mentions, as an aside (or obiter dicta), that:
(1)The absence of statistical significance does not preclude there being a warranted ground for inferring (or claiming to have evidence that) a drug caused an adverse side effect.
This is still very different from claiming
(2) The absence of statistical significance (in the case of a post-data subgroup) provides a warranted ground for inferring (or claiming to have evidence that) this drug–with its own serious side effects– has a survival benefit.
But there is a lesson: When it comes to evidence that is relevant to regulation and policy, alterations of methodological standards initially made in the interest of strengthening precautionary standpoints may be (and often are) used later to weaken precautions. Tampering with standards of evidence intended to increase the probability of revealing risks to the public tends to backfire. Admittedly, the obiter dicta at least**, in the Matrixx case, are open to lawyerly undermining the government’s position in the current case.
*Harkonen himself claimed the report was intended for investors.
**Here the “dicta” are throwaway remarks by the Supreme Court on (lack of) statistical significance and causal inference. See earlier post here.
I would also disagree with the alleged ‘transposition fallacy’ remarked on in Schachtman’s new blog, but I don’t have time to do more than record this here.
[i]We agree on the flaw by the Supreme Court in the Matrixx case.
Filed under: PhilStatLaw, PhilStock, statistical tests, Statistics Tagged: philstock
