I just finished reading an interesting book review by physicist Martin Blume in a recent issue of Nature. Blume was reviewing Eugenie Samuel Reich’s provocative book “Plastic Fantastic: How the Biggest Fraud in Physics Shook the Scientific World,” and the whole thing prompted some further thoughts about scientific misconduct, objectivity, and the peer review system that is crucial to the advancement of science.
Reich’s book is apparently very well researched (I take Blume’s word for it, since material physics is not my field), but she draws exactly the wrong conclusion from the case study she so thoroughly investigated. The biggest fraud that shook the scientific world wasn’t really the biggest (I would argue that Piltdown man, the fake missing link “discovered” in England in 1912 was much bigger), nor did it really shake the scientific world that much (I am a scientist, and I first read about this case through Blume’s book review). Still, it is an intriguing story in which a postdoc at Bell Laboratories, Jan Hendrik Schön, fabricated data concerning the properties of certain types of plastic (hence the title of Reich’s book) and got away with publishing papers in prestigious journals, including Nature itself, Science and several journals sponsored by the American Physical Society.
Reich, who is a journalist with a background in both science and philosophy, takes the moral of the story to be that “It seems like little more than blind faith to insist that all activity carried out in the name of science will always be self-correcting.” Well, yes, but as Blume points out, no serious scientist (or philosopher, for that matter) actually believes that.
The peer review process that is at the core of science’s ability for self-correction consists of two phases. The first one is the rather institutionalized practice that every editor of a scientific (or other scholarly) journal follows: when an author submits a paper for publication, the editor reads it and sends it out to a minimum of two reviewers who are chosen because they are experts in the particular field to which the paper is pertinent. The reviewers (who are anonymous to the author) send more or less detailed comments to the editor who then makes a judgment as to the suitability of the paper for publication.
But it is the second, more informal and open-ended, component of peer review that is really crucial. The first step relies on the expert advice of a small number of people (the editor and the reviewers), and it is subjected to conflict of interests (maybe one of the reviewers knows the author and dislikes her on personal grounds; or they have been in direct competition for grants, so that the reviewer has an interest in keeping the author from publishing). But after the paper is out everyone in the scientific community can read it, cite it (or not), and criticize it at meetings or in print. This second part of the peer review process is what really matters, because fraudulent papers in the long run end up in one of two categories: they are either completely forgotten because they didn’t really address an important topic at all (in which case the author gets away with the fraud, but there is no lasting damage to the scientific enterprise), or they are discovered because other people tried to replicate or build on the results and failed.
This is exactly what happened with Schön’s papers: they slipped through the first step of the review process, but spectacularly failed to pass muster during the second step. The failure of formal peer review in cases of fraud is not really that surprising, since the system is based on the assumption that authors are not just making stuff up. The aim of peer review is to reject or correct papers that are deficient in their methodology, data analysis, or in how well the author’s conclusions are substantiated by the empirical findings. Making stuff up doesn’t fall into any of those categories. Indeed, the real culprits there are Schön’s senior co-authors, who should have paid more attention to what they were signing off on, especially considering that some of the claims made by Schön were groundbreaking (like the discovery of superconductivity in plastic!).
Yet, contrary to Reich’s conclusion, these stories actually validate the scientific enterprise as particularly effective at uncovering fraud, when it really matters (i.e., for contributions that make a difference to science, as opposed to just adding a line to the c.v. of an individual researcher). The scientific peer review process (both formal and particularly informal) therefore is a bit like what Winston Churchill said of democracy: it is far from being a perfect system, but it is a heck of a lot better than any of the alternatives. Whatever works, as Woody Allen would say.
Rationally Speaking: an archived blog about science & philosophy, by Massimo Pigliucci
You really should base your conclusions on the book, not just on this review. Reich does not, by any means, exclude the "informal" part from the scientific process. The problem was that, although there was a lot of discomfort over Schön's work, and a widespread failure to replicate it, those problems alone did not stop him: he continued to win prestigious prizes and job offers and to be cited with praise. Contrary to Blume's claim, the "informal" process did not work--or at least had not yet done so.
ReplyDeleteWe'd all like to think that "in the long run" the problems would have added up to a rejection of the work. But in the meantime a lot of resources were wasted chasing after a fraud. Schön was eventually derailed only by his sloppy, essentially clerical mistakes--roughly analogous to the photo-manipulation problems in the cloning papers. It is worth asking, as Reich suggests, how long it might have taken to reject the work if he had covered his tracks better.
Does Reich suggest improvements to the peer-review process which would reduce the likelihood of this kind of occurrence?
ReplyDeleteNo, she doesn't really offer any suggestions for improvement. She's more descriptive than prescriptive.
ReplyDeleteMassimo,
ReplyDeleteThanks for this informative post. I didn't have much idea of the peer review process, and never had seen the informal review in that perspective.
I've no idea of the frauds the readers have talked of above.
I agree that valuable resources must have got expended in verification and subsequent rejection of the claims made about plastics, but that's a necessary evil. So many claims that sound extraordinary only serve to provide more-than-gentle push to science.
Take care.
I'm not sure about biggest fraud but perhaps most serious fraud is:
ReplyDeletehttp://www.anesthesiologynews.com/index.asp?section_id=3&show=dept&article_id=12634&ses=ogst
I suspect that peer review works fairly well when it comes to positions where the scientific establishment doesn't have strong vested interests.
ReplyDeleteHowever, there are the non-negotiable and sometimes barely-examined paradigms that are foundational to our shared philosophies and personal investments. These are often off-limits.
There is till some debate over educational psychologist Cyril Burt, but this man's work influenced education policy in the UK and elsewhere for decades. In those terms, his may be the greatest and most serious scientific fraud of all time.
ReplyDeletehttp://en.wikipedia.org/wiki/Cyril_Burt
Readers of Stephen Jay Gould may already be familiar with the Piltdown Man forgeries.
http://en.wikipedia.org/wiki/Piltdown_Man
What about Hwang Woo-Suk?
ReplyDeletehttp://en.wikipedia.org/wiki/Hwang_Woo-Suk
"The biggest fraud that shook the scientific world wasn’t really the biggest (I would argue that Piltdown man, the fake missing link “discovered” in England in 1912 was much bigger)
ReplyDelete"
You would?
It didn't shake the scientific community because those who had a lot at stake in the ev world view were immediately on the hunt for a new (series of) missing link(s). Seems like most missing links have been disproven actually. With so many species extinct, one would think that the logic about what links prove would work just about the opposite from what it is purported to. Instead of each link leading to evidence of some kind of improvement or change, it leads instead to the idea that the individual genomes don't have nearly the flexibility that they use to.
Don,
ReplyDeletethe process did work, Schön is known as a fraud because people looked into his work and couldn't replicate his results.
And my point wasn't at all that Reich doesn't take into account the informal part of the review process, only that her conclusion that the self-correcting mechanism of science doesn't work (direct quote in the review) is not warranted.
Besides, my post wasn't about the book in particular, it was about misconduct and the nature of science in general.
Massimo,
ReplyDeleteThere’s certainly room for reasonable differences of opinion, hinging in part on what one includes in the “normal” correction mechanisms of science. For Schön at least, failure to replicate did not work (or hadn’t yet), but a highly unusual point-by-point comparison of figures in different articles did. Although, as you noted, few practicing scientists would rely on the referee process alone, I think many would put a lot of faith in replication. But I don’t think it works as well as they would like to believe.
In their 1982 book on scientific fraud, “Betrayers of the Truth,” William Broad and Nicholas Wade make this point in a chapter called “The Limits of Replication,” which includes Mark Spector’s work on the kinase cascade at Cornell. As it turns out, he was spiking his samples with a radioactively labeled protein before running gels. This was discovered by his colleague Volker Vogt only after he got his hands on the original gels. As for Schön, Spector’s was highly visible work, but widespread failure to replicate had not uncovered the fraud. Broad and Wade write: “Had it not been for Vogt, the fraud would probably not have been detected until much later, and it might well have escaped attention altogether.”
It would be tautological to say that, because a fraud was found, the process worked. The more productive question, which you started to address, is what actions or procedures are actually effective in uncovering fraud. Even the journals would agree that refereeing is not. It is more troubling that, in this and other cases, the “gold standard” of replication is also ineffective. To my mind, the things that did work, checking original gels and individual data points, lie outside of the usual conception of the self-correcting mechanisms of science.
Disclosure: I was involved in the Schön case, both before it broke, as described in Reich’s book, and on the committee that investigated it afterward. The opinion here, however, is strictly my own.
Don, thanks for the thoughtful comment, it is much appreciated.
ReplyDeleteI agree with Mann'sWord, that "sacrosanct concepts" defy the supposed self-correction of science. I would add they have done so since Darwin, and science is in unrecognized crisis across the board because of these dogmas. In my limited experience with peer-review of my eventually published papers, I found widespread incompetence among principal investigators in the field (aerosol measurements and their analysis), and a feudal society of competing PI's, whose fundamental incompetences were never treated as more than debating points, never to be finally judged and turned out.
ReplyDeleteAs for the crisis I mentioned, read my blog, particularly my "Challenge to Science" posts at
http://theendofthemystery.blogspot.com/2009_04_01_archive.html