By Massimo Pigliucci
The next guest on the Rationally Speaking podcast will be our friend Steven Novella, host of The Skeptic’s Guide to the Universe podcast, as well as author of the Neurologica blog and co-editor of the Science-Based Medicine blog. He’s a busy guy, and we appreciate him taking the time to visit us here at Rationally Speaking.
Steven’s writings on skepticism often touch on the issue of evidence- (or, as he puts it, science-) based medicine, something that you might think is a redundant term (isn’t medicine supposed to be based on scientific research anyway?), but that turns out is anything but. Equally poignant are his critiques of a wide variety of quackery and so-called “alternative” medical practices and ideas, from homeopathy to vaccine denialism.
We will ask Steven about all of this, but also about more recent — and perhaps even more disturbing — claims to the effect that much of the so-called evidence-based medicine appears to be a lot more shaky than we would like to think. For instance, a November 2010 article in the Atlantic, penned by David H. Freedman, features the research of John Ioannidis, who specializes in meta-analyses of studies that purport to demonstrate all sorts of science-based medical claims, from the efficacy of aspirin to reduce the likelihood of heart attacks to that of hormone-replacement therapy for menopausal women.
Ioannidis’ research has shown that 40% of papers published in top medical journals are either wrong or make exaggerated claims (and those are the top journals!). This is a sobering statistic, and it apparently hasn’t surprised many people in the medical establishment, though you would think medical doctors and researches would be deeply troubled by it. Ioannidis went so far as to say: “I’m not sure that more than a very small percentage of medical research is ever likely to lead to major improvements in clinical outcomes and quality of life,” and — even more damning — “If we don’t tell the public about these problems, then we’re no better than nonscientists who falsely claim they can heal.” Ouch.
The way I see it, this is an issue regarding causality and control. When you see papers published or research done regarding a topic, it is usually to show the effect that one or a limited set of variables (lifestyle, activity, food, medicine, etc.) has on one or a limited set of aspects of human health. The problem comes when you analyze the huge number of variables that impact the health of the human body (which is nothing less than the interaction of every system of the body with every other system). This makes it difficult to identify with a high level of precision (at least, as high as is common in other scientific disciplines) what the cause-effect relationship is. This difficulty is made even larger when you see just how unfeasible it really is to have controls over other variables that might affect the results of the trial. For example, if you want to test the results of a new drug therapy, you can't simply have the test subject stop eating and sit still for a week while you evaluate the effects of the drug. So, I wouldn't really see this as much of an issue, considering the unique circumstances under which medical research is carried out. The problem comes when you compare the statistics to more precise scientific endeavors (i.e. particle physics).
ReplyDeleteThis is the rationalist's problem: science is inadequate to the task, easily corrupted and agonizingly slow. But it is also, like the famous quote about Democracy, the best game in town.
ReplyDeleteWhat's often missed in the skeptic community is that quacks thrive as much by aggrandizing science as by denegrating it. The quack over promises the cure while simultaneously suggesting that an imperfect cure (the real ones) is false. When science falls short of the exaggerated expectations, the quack steps in with the snake oil. Similarly the creationist argues that the provisional nature of science is proof of its failure. If the theory of evolution has ever been refined, they say, it must be wrong. In other words, if science is science, it is wrong. So, while the 40% failure cited in the post is clearly a sign of a systematic problem, the admission of that fact can strengthen science based medicine's case. So it is the task of the skeptic to lower expectations not raise them.
The same problem holds for rationalism. The Idealist will make all sorts of hyper logical arguments about the foundation of knowledge as a way to soften you up for a whole plate-load of nonsense. We can't know with certainty the relationship between freedom and causality, they say. We don't know anything about the "initial conditions" of the universe, hence everything is spirit and I will never die. I think the proper response is something along the lines of: If Bertrand Russell couldn't find the logical foundation to freaking mathematics, how can you feel justified in making such a radical conclusion about the universe based solely your Scholastic syllogisms?
So, yes, the study made me wince, but getting it out there in the right context aught to make people less likely to embrace quackery not less.
OneMoreDay: Well said. In the political-economic realm, we find all sorts of claims of rationality (or Reason), much of which has little or no evidence to support it. Yet if the mathematical model works, then there it is! Submit or cast your lot with the unwashed heathens!
ReplyDeleteMassimo, this blog and Steve's (NeuroLogica) are the two that I read and comment on most often, so naturally I look forward to this podcast! I especially can't wait to hear Dr. Novella's response to that troubling Atlantic article. (Oy!)
William Andrew:
ReplyDeleteI think that it's more than just causality and control at work here. The problem is with vested human intervention in these studies, and by that I mean bias, interpretation, implementation, etc., whether conscious or unconscious. That the research can be wrong is understandable, that's the nature of science and I doubt anyone would try and refute, or be surprised by, that fact. What Ioannidis specifically claims as problematic, and I tend to agree with his thesis, are the misleading claims of certainty and lack publication of papers refuting earlier claims, both of which interfere with the progress (albeit slow) of the scientific process.
OneDayMore:
I do agree that the recognition of a problem is the first step on the path of progress. The compelling difficulty here is how intimately connected medical research and practice is to the general public. If this were a problem in, say, cognitive neuroscience or cosmology, it would largely be a non-issue as those principally affected are people of technical proficiency who form a small subset of the population. Additionally, and perhaps more importantly, the results of these studies aren't as far reaching and of the same level of immediacy and impact as medical claims.
Given that the public's (and every (most?)) individual's) interest lies in preserving and enhancing their quality of life, and at a desired pace and certainty incommensurate with that of properly conducted science, there can be particularly disturbing consequences. On the one hand, we can expect certain individuals to intelligently question and second-guess the opinions of their doctors, conduct independent research, ask the important questions, and generally be involved in the process.
On the other hand, and what I foresee as perhaps the biggest problem, perhaps cynically, are the seeds of distrust between doctor and patient being sown creating a state of individual, and perhaps public, vulnerability that the quacks can quickly take advantage of. And what promises can science and skepticism make to these people to really comfort them against the onslaught of cure-alls and panaceas that might arise? It's a very delicate problem, to be sure.
Additionally, it serves as little comfort that Ioannidis' conclusions are purely existential; they indicate the existence of a problem, but offer no real construction of a solution. What inferences can we draw from even the well-conducted and provisionally accepted studies if they are so prone to error and imprecision? Superficially, this problem seems somewhat analogous to Incompleteness in logic; given the set of medical axioms we begin with, we can't possibly equip doctor's with the tools and data to cover ever situation encounterable.
I think it's pertinent that we get everyone to come to terms with the fact that even though society is constructed to protect us in a myriad of ways, there is still no certainty of a high quality of life through the intervention of man. Once we come to terms with this fact, the least we should expect is for the words of Laplace to hold true, "The weight of evidence for an extraordinary claim must be proportioned to its strangeness."
Malik B
ReplyDeletePhew, I reread my post and thought I must have written it in my sleep. Glad you got my point. When you say:
"I think it's pertinent that we get everyone to come to terms with the fact that even though society is constructed to protect us in a myriad of ways, there is still no certainty of a high quality of life through the intervention of man. Once we come to terms with this fact, the least we should expect is for the words of Laplace to hold true, "The weight of evidence for an extraordinary claim must be proportioned to its strangeness."
You are getting right at the heart of it! It all boils down to fear of death. Fear of death, a desire to have control -- these are the idols of the chump who sells his rational farm for a handful of magic beans. My favorite Faulkner quote: "The purpose of life is to get ready to be dead a long time." Of course we want good medicine and good industrial regulations to minimize risk, but the zero exposure life is not worth living, and there is no doctor who can cure entropy.
Your LaPlace quote is so similar to that popular Carl Sagan quote.
David Gorski responded to the Atlantic article on Science Based Medicine.
ReplyDeletehttp://www.sciencebasedmedicine.org/?p=7734
The obvious point is that we can tell that 40% of the published papers are wrong only if we now know the right answer.
The Women's Health Initiative found hormone replacement therapy to be risky, but then Carol Tavris wrote that there were many false alarms.
http://journals.lww.com/journalppo/Fulltext/2009/04000/Hormone_Replacement_Therapy__Real_Concerns_and.1.aspx
So Massimo, there's trouble with Physics, there's trouble with Medicine. Is Biology any better? Maybe it's just the trouble with Science, that it's messy and tentative. That's why there are systematic reviews of the literature to sort things out.
Now, if 40% of the medical guidelines are wrong, and following them does more harm than good, that would suck. It would be nice to know the probability that a guideline is wrong before deciding to follow it. Is this risk possible to estimate?
Hello,
ReplyDeleteBehaviorism was the predominant branch of psychology in the mid-twentieth century. Cognitivism attempted to refute it and it became the main branch of psychology. Can the two coexist? Doe one refute the other?
B.F. Skinner said that unobservable and unmeasurable mentalistic explanations of behavior interfered with an examination of the environmental causes of behavior. If we ask why did someone do something? The mentalistic explanation might be, "Because he wanted to." And the examination stops there.
Skinner taught a pigeon to turn in a circle when it saw a sign with the word "turn" and to peck a button when it saw the word "peck." It was trained with food reinforcement when it was underweight.
The Atlantic article reports on flaws in statistical medical research based on groups rather than individuals. Skinner attempted some statistical experiments on groups of rats but he dropped the approach in favor of studying one subject at a time.
Typically, individual organisms were placed in controlled environments. The experimenters attempted to keep all conditions the same, and then change one variable. The results printed out on a graph with a curved line of behavior change over time. Baseline behavior was determined, the variable was changed, and then they returned to baseline conditions. If a behavior was changed and then reversed, they had evidence of an environmental factor that influenced behavior. The line on the chart for an individual could show minute changes in response, but a statistical result of his groups of rats would combine individual variations together and the details were obscured.
Please ask Mr. Novella where are the potential flaws if this experimental design is followed to the letter?