Radical reform for peer review?

scienceforseo.com

by Massimo Pigliucci

A recent piece by Scott Jaschik in “Inside Higher Education” pointed out what a number of my colleagues have been thinking for a while now: the peer review system for scholarly journals doesn’t work very well, needs to be reformed, and really ought to take radical advantage of new technologies. There is, of course, going to be quite a bit of resistance to any change coming from the usual quarters, beginning with older academics who still think of social networking in terms of meeting colleagues after work for a martini (well, okay, nothing wrong with that), administrators who are used to the simple (and simplistic) bean counting operations for tenure and promotion made possible by the current system, and journal publishers who make a ton of money while adding next to nothing in value to people’s publications (after all, they don’t pay for the research, don’t pay the writers, and don’t pay the editors and reviewers — which of course doesn’t stop them from charging an arm and a leg to university libraries).

Of course, since the new technologies are making an overhaul of the system possible, and since there is widespread frustration with the current modus operandi especially among younger faculty, change will happen one way or another — witness the rise of open access and online journals that bypass traditional publishers. It’s only a question of which paths to take, and that’s where the conversation gets interesting.

The most radical suggestion mentioned in the Inside article is the one by Aaron J. Barlow, associate professor of English at the City University of New York, where I work. Barlow is quoted in the article as saying that “peer review — in the sense that people work and a consensus may emerge that a given paper is important or not — doesn’t need to take place prior to publication.” He is, of course, right and as a matter of fact most peer review has always taken place after publication. A lot of bad or simply irrelevant stuff gets published and ends up augmenting someone’s c.v. by a line or two (good for promotion and tenure!), but then dies the common death of much academic scholarship: complete lack of citations by anyone other than the author.

The question that Barlow is raising is whether it wouldn’t be better to skip the preliminary step — the pre-publication filter — and simply leave everything to the community at large. I am sympathetic to that position, particularly because as author, editor and reviewer I have seen my share of unseemly behavior, gender and racial biases, personal vendettas, and so on that certainly don’t belong anywhere within a scholarly environment. But I think pretty much everyone agrees that we already have far too much pyrite to sift through in order to find the gold nuggets, and I shudder as to what would happen if anyone were suddenly able to claim “scholarship” by simply posting their papers on the web and ask people — anyone, not just the relevant expert community? — to comment, “+1” or “like.”

This is the same problem that has been faced by the publishing and journalism industries. These days anyone can self-publish a book at the click of a button, and anyone can set up an online newspaper with free or cheap software and access to a server. But I doubt these new technological possibilities will spell the demise of editors, publishing houses and newspapers like the New York Times, for the simple reason that these “classic” outlets do exercise a very valuable (if flawed, incomplete, sometimes biased) function of filtering a lot of distracting or poor quality nonsense (as the NYT’s famous tagline says, “all the news that’s fit to print,” or to pixellate, as the case may be).

Another approach commented on in the Inside piece is the one currently pursued by Cheryl Ball, the editor of an online journal on rhetoric and technology called Kairos, and an associate professor of English at Illinois State University. Her journal engages the entire editorial board in a lengthy discussion of every submitted paper, at the end of which an editor is assigned to coach the author on how to revise the manuscript to reflect the consensus of the board. This makes the system much more transparent (the author knows that all editors participated in the discussion, no anonymity on either side) and obviously immensely constructive from the point of view of the author and the community at large. But I seriously doubt this sort of model can be expanded to the whole industry. I edit a small online open access journal in philosophy of science, and even with our low number of yearly submissions it would be impossible to get my editorial board to do what Ball has been able to accomplish with hers. Again, the problem being that there are too many authors out there, and that far too high a proportion of submitted papers is simply not up to even minimum standards, or would require a huge amount of work to get there (not to mention, of course, that — again — editors and reviewers are not paid for this, nor do they get much concrete credit from university administrations for engaging in what they do).

I do not know what the solution is, and I suspect that we will see over the next few years increased experimentation on the part of younger editors to either ameliorate the problems with the current system or to overhaul the thing altogether. Some journals already make the author, not just the reviewers, anonymous, to minimize biases (it is well known, for instance, that women and minorities get fewer papers accepted if the reviewers know their names, and that the effect disappears if authorship is kept anonymous). Others publish all submitted papers that are technically correct — meaning that are written in an intelligible manner and include all the necessary documentation — while leaving to readers to judge the intrinsic value of the authors’ findings and opinions. We certainly are on the cusp of a technologically driven revolution in academic publishing, but just as in the already mentioned cases of book publishing and journalism, it remains to be seen exactly what will be left standing and what will have arisen anew once the storm has passed.