Blogosphere beats peer review in the case of stealth creationist paper

Academia is notoriously resistant to change, which to some extent is a good thing. It was therefore no surprise that when Wikipedia became a phenomenon most academics scoffed at it as a passing fad, fatally flawed by its very core idea: anybody, and I mean anybody, can become a Wiki author and post new entries or edit existing ones. Surely, this will inevitably lead to chaos and complete unreliability, the critics said. But a few years ago a study of a sample of entries compared the accuracy of Wikipedia with that of the unquestionably prestigious Encyclopedia Britannica, and Wikipedia was at least as accurate, in some cases more.

Of course the "open access" model does have its limits and defects, and even Wikipedia has to maintain a certain amount of vigilance and label particular entries as contentious or unreliable if there is too much traffic and a lot of editing and counter-editing (typically concerning political issues or individual politicians). Still, from apparent chaos the system has allowed for the emergence of a reasonably reliable first-look reference source that truly exploits the power of the internet.

It seems that the next case will come from another sacred cow of academia: peer review. This is the system used by modern academics -- both in the sciences and the humanities -- to evaluate a scholarly paper before it is published, the chief gateway to insure the high quality of a publication, be it in philosophy, literary criticism, medicine, physics, or what have you. The way it usually works is that an author submits a paper for consideration to the editor of a journal in the appropriate field. The editor makes a first assessment of the manuscript and, if deemed suitable to the journal, sends it out to two or more reviewers, chosen from among people actively engaged in research and scholarship in the field addressed by the submitted paper.

A certain amount of time later (which can be irritatingly long for the authors), the reviews come back with a thumbs up or down verdict, usually accompanied by (anonymous, and sometimes nasty) comments for the authors -- so that they may revise the original manuscript and send it back to either the same journal (if so invited) or to another one. The process repeats itself until either the paper finds its way into a publication or is forever abandoned on the heap of wasted efforts.

The peer review system has its obvious advantages as a gatekeeper for academic publishing quality, but it has equally obvious drawbacks. First of all, the number of reviewers is fairly small, which means that the comments the authors receive may be reflective of the idiosyncratic views of those individuals, and may not necessarily constitute a good assessment of the general value of the paper. Second, often (though not always) the authors don't know who the reviewers are, but the converse is not true, which leads to the temptation of stabbing a rival (or a rival's student) in the back.

One can argue that the real peer review actually takes place over a period of years after the paper (or book) has been published, and it is the result of how, in the long-term, the community at large values the scholarship of the authors. Some papers and books are cited often, some become classics in their field, most are never heard of again -- which in itself is not necessarily an indication of poor quality, but may be a simple reflection of the fact that too many people publish too much.

What I will call the classic peer review system, the one that relies on a small number of editor-selected referees, however, is increasingly under challenge. In the physics community, for instance, it has been normal practice for years to post pre-publication versions of one's paper on internet servers, to get feedback from the rest of the community before formal submission. People can now refer others to these pre-prints by hyperlinks, almost as if they were actual publications, thereby blurring the distinction between formal and informal scholarship. Moreover, an increasing number of open access journals now encourages readers' comments and even rankings to be posted for each paper, occasionally allowing authors to respond and engage in an open dialogue with the community.

This is, I think, a trend that is here to stay, and that will likely completely change the meaning and practice of academic research over the next decade or so. Still, perhaps the most spectacular -- if somewhat under-reported -- case of open peer review showed how the blogosphere can be a more effective guardian of scholarship than a small number of overworked editors and reviewers.

What happened was that two people affiliated with Inje University in Korea, Mohamad Warda and Jin Han, submitted a paper to the prestigious journal Proteomics. The paper was entitled "Mitochondria, the missing link between body and soul: Proteomic prospective evidence," something that should have alerted the Editor, Michael Dunn, and the reviewers that something was amiss (a proteomic paper on dualism and the question of the soul?). Warda and Han's review of the literature was meant as a criticism of the currently accepted theory that the mitochondria (the cellular organelles that are involved in the production of the energy that keeps the metabolism of the organism going) are the result of an evolutionary endosymbiotic event; in other words, that they originated from the engulfment of a bacterial cell by an ancestor of modern plants, animals and fungi.

Warda and Han wrote: "Alternatively, instead of sinking into a swamp of endless debates about the evolution of mitochondria, it is better to come up with a unified assumption. ... More logically, the points that show proteomics overlapping between different forms of life are more likely to be interpreted as a reflection of a single common fingerprint initiated by a mighty creator than relying on a single cell that is, in a doubtful way, surprisingly originating all other kinds of life."

It is difficult to make sense of the badly written phrase (no language editors at Proteomics?), but surely the reviewers should have been a bit surprised by the obviously unscientific phrase "a mighty creator." Regardless of whether one thinks that concepts like soul and divine creators make any sense at all (I don't), they surely do not belong to an ostensibly scientific paper. I am not at all suggesting that Dunn or his reviewers are intelligent design creationists: they simply missed the supernatural references, presumably because they were too busy and distracted by the mountain of very technical language surrounding that specific phrase (though how they missed the title is a bit more difficult to rationalize away).

The happy ending to the story is the result of the normal practice that Proteomics has, as do many other journals, of posting papers on their web site before they are actually printed. According to an article in the National Center for Science Education Reports, the first to note the oddity of Warda and Han's paper was Steven Salzberg, a professor of computer science at the University of Maryland, who blogged about it. That led to blog posts by Attila Cordas, Lars Juhl Jensen and PZ Myers, and eventually to the editor of Proteomics requesting a withdrawal of the paper by the authors, who complied.

Interestingly, the request to withdraw was not based on the creationist claim, but on the fact that the bloggers had uncovered another problem with the paper that had escaped reviewer and referees: the entire body of the article by Warda and Han had been plagiarized from other, already published, sources! Apparently, their only original contributions were writing in really awful English and references to the soul and the mighty creator.

The moral of the story is that the much maligned blogosphere ("you know, anybody can write whatever they want, and nobody's checking") in this case clearly surpassed the official, academically sanctioned system of peer review. My hunch is that this isn't going to be the last time this happens, and that we are looking at the dawn of a new era of academic practice, when papers will be scrutinized by thousands of reviewers within a matter of hours from publication. If we can harness this tremendous intellectual power in a reasonably orderly fashion, we will make the next leap toward a truly worldwide community of scholars and authors.