About Rationally Speaking


Rationally Speaking is a blog maintained by Prof. Massimo Pigliucci, a philosopher at the City University of New York. The blog reflects the Enlightenment figure Marquis de Condorcet's idea of what a public intellectual (yes, we know, that's such a bad word) ought to be: someone who devotes himself to "the tracking down of prejudices in the hiding places where priests, the schools, the government, and all long-established institutions had gathered and protected them." You're welcome. Please notice that the contents of this blog can be reprinted under the standard Creative Commons license.

Monday, February 23, 2009

Blogosphere beats peer review in the case of stealth creationist paper

Academia is notoriously resistant to change, which to some extent is a good thing. It was therefore no surprise that when Wikipedia became a phenomenon most academics scoffed at it as a passing fad, fatally flawed by its very core idea: anybody, and I mean anybody, can become a Wiki author and post new entries or edit existing ones. Surely, this will inevitably lead to chaos and complete unreliability, the critics said. But a few years ago a study of a sample of entries compared the accuracy of Wikipedia with that of the unquestionably prestigious Encyclopedia Britannica, and Wikipedia was at least as accurate, in some cases more.

Of course the "open access" model does have its limits and defects, and even Wikipedia has to maintain a certain amount of vigilance and label particular entries as contentious or unreliable if there is too much traffic and a lot of editing and counter-editing (typically concerning political issues or individual politicians). Still, from apparent chaos the system has allowed for the emergence of a reasonably reliable first-look reference source that truly exploits the power of the internet.

It seems that the next case will come from another sacred cow of academia: peer review. This is the system used by modern academics -- both in the sciences and the humanities -- to evaluate a scholarly paper before it is published, the chief gateway to insure the high quality of a publication, be it in philosophy, literary criticism, medicine, physics, or what have you. The way it usually works is that an author submits a paper for consideration to the editor of a journal in the appropriate field. The editor makes a first assessment of the manuscript and, if deemed suitable to the journal, sends it out to two or more reviewers, chosen from among people actively engaged in research and scholarship in the field addressed by the submitted paper.

A certain amount of time later (which can be irritatingly long for the authors), the reviews come back with a thumbs up or down verdict, usually accompanied by (anonymous, and sometimes nasty) comments for the authors -- so that they may revise the original manuscript and send it back to either the same journal (if so invited) or to another one. The process repeats itself until either the paper finds its way into a publication or is forever abandoned on the heap of wasted efforts.

The peer review system has its obvious advantages as a gatekeeper for academic publishing quality, but it has equally obvious drawbacks. First of all, the number of reviewers is fairly small, which means that the comments the authors receive may be reflective of the idiosyncratic views of those individuals, and may not necessarily constitute a good assessment of the general value of the paper. Second, often (though not always) the authors don't know who the reviewers are, but the converse is not true, which leads to the temptation of stabbing a rival (or a rival's student) in the back.

One can argue that the real peer review actually takes place over a period of years after the paper (or book) has been published, and it is the result of how, in the long-term, the community at large values the scholarship of the authors. Some papers and books are cited often, some become classics in their field, most are never heard of again -- which in itself is not necessarily an indication of poor quality, but may be a simple reflection of the fact that too many people publish too much.

What I will call the classic peer review system, the one that relies on a small number of editor-selected referees, however, is increasingly under challenge. In the physics community, for instance, it has been normal practice for years to post pre-publication versions of one's paper on internet servers, to get feedback from the rest of the community before formal submission. People can now refer others to these pre-prints by hyperlinks, almost as if they were actual publications, thereby blurring the distinction between formal and informal scholarship. Moreover, an increasing number of open access journals now encourages readers' comments and even rankings to be posted for each paper, occasionally allowing authors to respond and engage in an open dialogue with the community.

This is, I think, a trend that is here to stay, and that will likely completely change the meaning and practice of academic research over the next decade or so. Still, perhaps the most spectacular -- if somewhat under-reported -- case of open peer review showed how the blogosphere can be a more effective guardian of scholarship than a small number of overworked editors and reviewers.

What happened was that two people affiliated with Inje University in Korea, Mohamad Warda and Jin Han, submitted a paper to the prestigious journal Proteomics. The paper was entitled "Mitochondria, the missing link between body and soul: Proteomic prospective evidence," something that should have alerted the Editor, Michael Dunn, and the reviewers that something was amiss (a proteomic paper on dualism and the question of the soul?). Warda and Han's review of the literature was meant as a criticism of the currently accepted theory that the mitochondria (the cellular organelles that are involved in the production of the energy that keeps the metabolism of the organism going) are the result of an evolutionary endosymbiotic event; in other words, that they originated from the engulfment of a bacterial cell by an ancestor of modern plants, animals and fungi.

Warda and Han wrote: "Alternatively, instead of sinking into a swamp of endless debates about the evolution of mitochondria, it is better to come up with a unified assumption. ... More logically, the points that show proteomics overlapping between different forms of life are more likely to be interpreted as a reflection of a single common fingerprint initiated by a mighty creator than relying on a single cell that is, in a doubtful way, surprisingly originating all other kinds of life."

It is difficult to make sense of the badly written phrase (no language editors at Proteomics?), but surely the reviewers should have been a bit surprised by the obviously unscientific phrase "a mighty creator." Regardless of whether one thinks that concepts like soul and divine creators make any sense at all (I don't), they surely do not belong to an ostensibly scientific paper. I am not at all suggesting that Dunn or his reviewers are intelligent design creationists: they simply missed the supernatural references, presumably because they were too busy and distracted by the mountain of very technical language surrounding that specific phrase (though how they missed the title is a bit more difficult to rationalize away).

The happy ending to the story is the result of the normal practice that Proteomics has, as do many other journals, of posting papers on their web site before they are actually printed. According to an article in the National Center for Science Education Reports, the first to note the oddity of Warda and Han's paper was Steven Salzberg, a professor of computer science at the University of Maryland, who blogged about it. That led to blog posts by Attila Cordas, Lars Juhl Jensen and PZ Myers, and eventually to the editor of Proteomics requesting a withdrawal of the paper by the authors, who complied.

Interestingly, the request to withdraw was not based on the creationist claim, but on the fact that the bloggers had uncovered another problem with the paper that had escaped reviewer and referees: the entire body of the article by Warda and Han had been plagiarized from other, already published, sources! Apparently, their only original contributions were writing in really awful English and references to the soul and the mighty creator.

The moral of the story is that the much maligned blogosphere ("you know, anybody can write whatever they want, and nobody's checking") in this case clearly surpassed the official, academically sanctioned system of peer review. My hunch is that this isn't going to be the last time this happens, and that we are looking at the dawn of a new era of academic practice, when papers will be scrutinized by thousands of reviewers within a matter of hours from publication. If we can harness this tremendous intellectual power in a reasonably orderly fashion, we will make the next leap toward a truly worldwide community of scholars and authors.

10 comments:

  1. I think having feedback from the blogs is very useful, but I see major problems with letting formal publication decisions on technical papers rest on review by non-specialists. This was an exceptional case. The paper should not even have gone to review.

    ReplyDelete
  2. Ryan,

    but the idea wouldn't be to allow non-specialists to do the extended peer review. It would be members of a community of scientists (or philosophers, or whatever).

    People could register with the online journal, like they do with the physics archive, and comment on papers. And of course the two systems are not mutually exclusive: a pre-screening of papers could be done in the usual way, but then the paper would be subject to further, open-ended review.

    In fact, that's the model that a few colleagues and I are thinking of adopting for a new journal that will come online by the end of the year: http://philosophyandtheoryinbiology.org/

    ReplyDelete
  3. Sounds good. I just think keeping "peer review" and "online review" as separate steps is important. It's one thing to notice that there is some creationist language in a paper and quite another to examine the methods or stats or review of literature. On the other hand, it has always been a system of expert review first, then larger community review second -- it's just been inefficient because it had to happen in print. I think online commentary could be great as a way of making this second part of the procedure more effective

    ReplyDelete
  4. Let's face it, most academic papers are of interest to a small number of specialists; it's not as if such papers will attract the attention of the same folk who write angry letters to the editor of their local newspaper. I would therefore expect that the great majority of comments on any paper would be of some value. Moreover, the useless comments would almost certainly be immediately discernible from their poor spelling, grammar, etc. So why not? Cast a wide net, collect a broad variety of responses, and toss the dreck. Sounds like a more reliable system than the Russian Roulette of the standard system.

    ReplyDelete
  5. I think this must happen if we are to better understand complex systems, where there are no 'specialists' and many different fields of science are called upon to formulate a picture of nature. I think the journal is a wonderful idea, let people submit their ideas – let’s talk about it. Even if the ideas are outlandish we can read about why they are so, or why they are not so. Serious discoveries can be made this way! And if it’s total shit, just chuck it.

    ReplyDelete
  6. @Chris Crawford:
    I think the spelling and grammar problems here arose from the fact that English is not the mother tongue oft he authors. What seems interesting here is that the plagiarism wsa caught by the webizens, because it would be unrealistic to expect a handful of peer reviewers to know all the literature out there, and might not ercognise plagiarism where they see it.
    ---
    Massimo, could you tell us more about this new, forthcoming online journal, please?

    ReplyDelete
  7. Kimpatsu,

    well, for now the best you can do about the new journal is to check the general description on the provisional web site: http://philosophyandtheoryinbiology.org

    We are beginning to receive submissions, and the authors and editors are of a very good caliber, both in biology and in philosophy. I think this is going to be lots of fun!

    ReplyDelete
  8. I'm reminded of Winston Churchill's famous quote: "It has been said that democracy is the worst form of government except all the others that have been tried."

    But I think there's some ambiguity here. Peer review just means review by one's peers, i.e. other specialists who are competent to comment on the work. The peer review system, on the other hand, is the traditional way that academic peer review has been implemented.

    Peer review itself is imperfect. After all, experts can be wrong. But, as Ryan suggests, expert review is an indispensable first step.

    The traditional peer review system has its own shortcomings (its snail's pace for one). It seems that technological innovation has the potential to substantially improve on this. But lots of issues remain. Anonymity, for example, is a double-edged sword.

    ReplyDelete
  9. Massimo,
    IMHO, you should ditch the background on http://life.bio.sunysb.edu/~massimo/p&tb/, it makes reading the text difficult. Also, if you don't have someone else in mind, I could volunteer my services for the development of the website.

    Sorry, no comments specific to the post; I'm usually in agreement with what you say.

    ReplyDelete
  10. Valera,

    thanks for the suggestion and offer of help. That site is only provisional, until Stony Brook Library's people finish the actual journal's site with all the interactive bells and whistles. Stay tuned!

    ReplyDelete

Note: Only a member of this blog may post a comment.