Biomedical research is a fundamentally empirical discipline, rooted in the fertile soil of quantitative science, highly reliant on statistical methods in assessing the accuracy of its analyses. For this reason alone, I consider it imperative that biomedical researchers learn how statistics fails.
As an engineer, I am committed to the principle of graceful degradation, which can be paraphrased in Hippocratic terms as: "If you must attempt something that may do harm, try your best to ensure that no failure mode, including but not limited to the worst-case scenario, is catastrophic."
Nassim Taleb is a B-school type rather than a biomedical researcher, so some may consider his insights to be irrelevant to their work. I beg to differ, and offer the Vioxx scandal as admittedly anecdotal evidence.
The introduction on the Edge site gives a more eloquent lead-in to Professor Taleb's essay than I can provide:
When Nassim Taleb talks about the limits of statistics, he becomes outraged. "My outrage," he says, "is aimed at the scientist-charlatan putting society at risk using statistical methods. This is similar to iatrogenics, the study of the doctor putting the patient at risk." As a researcher in probability, he has some credibility. In 2006, using FNMA and bank risk managers as his prime perpetrators, he wrote the following:
The government-sponsored institution Fannie Mae, when I look at its risks, seems to be sitting on a barrel of dynamite, vulnerable to the slightest hiccup. But not to worry: their large staff of scientists deemed these events "unlikely."
In [his] essay, Taleb continues his examination of Black Swans, the highly improbable and unpredictable events that have massive impact. He claims that those who are putting society at risk are "no true statisticians", merely people using statistics either without understanding them, or in a self-serving manner. "The current subprime crisis did wonders to help me drill my point about the limits of statistically driven claims," he says.
I have a personal interest in the subprime crisis, which we are now told by the Bush administration will cost us $700 billion to ameliorate, not counting the damage already done to foreclosed homeowners' lives and the housing values of other homeowners, for most the single biggest investment they have made and the foundation of their retirement.
I fall in the latter category, but there's another reason I have a personal stake: that $700 billion is going to come from somewhere, and you can bet your bippy it isn't going to be funded by a reduction in funding for our endless war on terror. I'll bet Bush's war contractor and oilman buddies will make sure of that, regardless of who sits in the Oval Office.
Instead I expect the funding will come from soft targets, like NIH and NHIN. It will impact the funding of healthcare for aging Baby Boomers and the nearly 50 million uninsured in the US, and the demands the aging Boomers will soon make on the Social Security system.
<rant>
I remember a few friends telling me at a party a year or two ago that they didn't favor relief to subprime mortgagees and other homeowners who were impacted by the sour economy to the extent that they were faced with losing their homes - some of whom were my frinds' neighbors. Why help people who made bad decisions? Let them suffer the consequences, even if it left their houses empty and themselves homeless.
The answers to my friends' question should now be apparent to everyone: we should have helped those unfortunates because not helping them weather the storm has brought down home values across the nation, reduced the amount of credit available to fund economic expansion, and damaged the credibility of the US as the engine of the global economy, and now has resulted in a massive bailout that will save the yachts and summer homes of Wall Street multi-millionaires rather than the quality and stability of life on Main Street. For the record, Main Street is a fifteen-minute walk from my home.
</rant>
Back to biomedical research. If you are engaged in this noble work, please read Taleb's essay and ask yourself: what are my Black Swans?
Black Swans are rare but catastrophic events. Taleb gives a great metaphorical example of the Black Swan concept: the life of a turkey. I'm borrowing a graphic from his talk to illustrate, but I'm changing his numbers to make them more realistic (he clearly never lived near a turkey farm, much less raised one himself).
A turkey is hatched in the early Spring, then spends a couple hundred days living the good life, getting bigger and fatter every day. On the Friday before Thanksgiving, the day goes a bit differently for the turkey, and he or she loses everything gained up to that point.
Catastrophic failures in clinical research, such as Vioxx, exhibit a similar life cycle. It is up to everyone involved in such research to know what their projects' Black Swans look like, and make sure the principle of graceful degradation is applied in the analysis and management of the risks inherent in the research.
i think simplification was the main reason why taleb opted for 1001 example in the turkey's case.
the same 1001 days was used in his previous book, when talking about a trader's good performance wiped out by a single bad day.
Posted by: phaeton | December 03, 2008 at 01:54 PM