When is Peer Review the Gold Standard, and When is it Only Tin?

Fair warning from an anonymous peer reviewer on one of my recent articles…

“The author is hampered by an inaccurate, naïve, and highly simplistic understanding of the basic principles…which leads him to make ludicrous statements like the following…”

As is well-known, “peer review” is the gold standard of academic achievement. It is assumed that peer review gives rigor and legitimacy to new ideas. This assumption persists even in the context of well-publicized fraud scandals involving high fliers in physics, human cloning, and cancer research which indicate that peer reviewers at journals like Science and Nature can be as sloppy as anyone else. Nevertheless, the process does often add to the seriousness of academic publication. Plus, if you did not have peer review, as is often said, you are no better than a newspaper, a blog like Ethnography.com, or (horrors) Wikipedia!
But, as the the quote at the beginning of this article shows, peer review is not always encouraging, nurturing, or in my view, very fair. In other words, sometimes there is only tin beneath the gold plate. Peer reviewers with the cloak of anonymity sometimes let loose on potential competitors. Editors do not always do their part by protecting writers from the more unreasonable attacks. Does this make for better science? Perhaps sometimes. My own view is that in the long-run peer review makes for a more careful and conservative science. But it also discourages challenges to the status quo, even though such challenges are what good science is about in the first place. Most crucially, writers without a thick skin are discouraged from pursuing good ideas further, all because some anonymous reviewer process had a fight with their spouse or teenager that morning, and took it out on you.

Scientific Publication—The Theory
The ideal of peer review is that rational, unbiased, and anonymous experts evaluate the work of others to verify whether an idea is new, rigorous, and important enough for publication. You submit a paper to a journal, and then the editor selects colleagues within your discipline to read what you have written. Anonymity is important to this process (ideally both the reviewer and reviewee do not know who each other are), because it is well-known that there are friendship cliques and elites within the scientific community which may bias review. Reviewers judged by editors as “possible for publication” are then sent to reviewers selected for their expertise and respect. The reviewers then submit their reasons for acceptance or rejection to the editor. Such reasons ideally entail 2-3 pages (single spaced) discussing the strengths and weaknesses of a paper’s data and argument, which are then forwarded anonymously to the author. Often, suggestions are made about literature that may have been missed in the paper, irrespective of whether the paper is accepted or rejected. I have found such suggestions helpful.
Based on 1-3 such reviews, the editor then makes a final decision about whether to accept, reject, or suggest a “revise and resubmit” to the author. Final acceptance of course is important within the scientific community. Besides the status and prestige associated with publication itself, papers published in such “peer reviewed” journals can make a difference in academic promotion and tenure decisions.
With revise and resubmits, a paper often has up to five reviewers (plus the editor) read and make anonymous comments for the author. Because so many minds are focused on the development of the paper, the overall quality, rigor, and accuracy of the argument is improved. The process is often slow. Between first submission, and the final arrival of a paper in print, months, and possibly years may pass. But this care is why your anthropology professor prefers to see you cite the Human Organization, American Anthropologist, or Anthropology Today, rather than Newsweek, CNN’s website, Ethnography.com, Wikipedia, or even Encyclopedia Britannica. All of these sources may be edited for style (ok, maybe not Ethnography.com), but there is not an expert review of the facts.

The result of all this peer reviewed literature is a scientific literature which academics (especially graduate students) pore over in order to find their own innovation. The peer reviewed literature is more valid and reliable because it has been through the rigorous review process. Acceptance rates in the most prestigious journals are often less than 10%, meaning that only the self-described “very best” is published, while the rest is rejected and perhaps submitted to a less prestigious journal, or perhaps find itself into publication in an “edited collection” prepared by a group of colleagues interested in a shared subject. Note that neither of these final two conditions are all that bad. A new idea is still “out there” for the diligent researcher to find.

Scientific Publication—The Practice
Anyway, that’s the theory of peer review. I have been through the process with two separate articles and a book proposal four times in the last six months or so. Only sometimes has it met the ideal. The book proposal has resulted in a contract (yippee), one article on neurology was flat out rejected once (ugh!), and from a second journal received a “rejection but you can submit again.” The third paper about African history was rejected, and the editor recommended I pay attention to one reviewer’s comments, and submit to another journal. All together, the reviews incorporated the opinions of six reviewers. Two were brief and insulting without redeeming value, and dismissed my work in a few short lines. One was insulting, but made good recommendations about things that should be incorporated in the article. One was frustrated with my “sloppiness” but the reviewer thought the paper was worth a “revise and resubmit” which the editor did not give me. The fifth thought the paper was worthwhile, but needed to be fleshed out for the “new parts” more, and the editor gave me the “reject but you can resubmit in a revised form.” The last was the “accept.”
In other words, three of the reviews were constructive, and reflect the very best of the peer review process. Two of them reflect some of the worst impulses found in the review process. The one which was insulting (called me naive, etc) gave good suggestions was somewhere in the middle.

Here is a sampling, with some of my own comments:

“…There is little that is based on original research and no substantial intellectual or theoretical content…I am sorry to be so negative, but this [paper] is simply a non-starter.” (This comment was on a 40+ page paper, and the whole review was only about six sentences long. This reviewer has an ego problem and is lazy).

The second review on the same paper was three pages long, and pointed out in excruciating detail a number of errors on my part:

“Despite this rather frustrating sloppiness [which was pointed out in excruciating detail], I am willing to see the author revise and resubmit…” (ok, ok, you got me this time…I will go back and fix things)

Comments on the sociology and neurology article included the following. First the extremely short dismissive review:

“This leads him to highly fatuous arguments…” (Not as fatuous as your stupid review).

A second comment on the same paper:

“The author is hampered by an inaccurate, naïve, and highly simplistic understanding of the basic principles…which leads him to make ludicrous statements like the following…(this review included some good references to what the reviewer thought were key to the discipline, so he got me on that one. I will cite them, but also note that they present an inaccurate, naïve, and highly simplistic understanding of basic sociological literature…which leads to ludicrous statements. Except I will say this with more respect, and not anonymously.)

The paper was resubmitted to another journal after I took a number of issues raised in the second review into account. I received the following comments back:

“I’m very sympathetic to one of the paper’s central claims…but I don’t believe that the paper as a whole has a sufficiently clear and sustained focus. .. What exactly do the two ideas have in common (apart from a central metaphor) and how do they differ? What can we learn from the comparison … But to make a substantial contribution to this more general debate, it would need to canvas a range of examples, … and to break some ground; advance some new arguments or shed new light on old ones.” (This comment ended in a rejection and resulted from the comment below from the editor. But thanks for the thoughtful comments!)

“I agree with the reviewer`s opinion that the basic line of thought in this paper is interesting and plausible. But I think the reviewer is also probably right that these basic ideas need more sustained development…”(ok, you have a good point. I will do it, and get back to you in a couple of months which incorporate some of the specific points raised—thanks for being encouraging even though this was not an acceptance!)

And finally a note from the one acceptance out of the four submissions:

“I’m not sure if I have a plan to order things differently than they are currently ordered, but it strikes me as potentially a little awkward…” (I think that this reviewer was probably right—but this comment was from the review which resulted in the book contract, which made me pretty happy in the first place)

My own strategy for working with this range of commentary, is to assume that anything complimentary is really correct, suggestions for including other books as a citation should always be followed, and that anyone that includes words like “naïve” or “ludicrous” means that I have a really good paper and should try again, and that the reviewer is in need of psychiatric help.

Why we Need Peer Review
So there you have peer review, from the nasty to the constructive. If you are ever asked to do peer a review, I would urge you to avoid the nasty side. Be constructive in your comments, even if your conclusion is to “reject.” Remember too, that many papers go through many iterations—papers are only rarely accepted on the “first try.” My own experience is that papers might be accepted on the second to fifth try. And usually—though not always—the peer review process is a constructive part of the development of a paper. Also remember, there are a lot of journals out there, and a rejection is sometimes the luck of the draw. How could the editor have known that the reviewer he met a conference five years before had tortured frogs as a child, and was also going through a bad divorce? Ignore the comments about being naïve, simplistic and ludicrous, and fix what is fixable, while also recognizing that you cannot please every reviewer all the time.

While peer review eliminates poor scholarship, in my view the greatest contribution peer review offers is in its ability to encourage and nurture good scholarship in others. Some of the more prestigious journal in sociology note this, telling reviewers that despite the fact that 90% of the submissions are not published, their comments are important because eventually many papers are published somewhere. Indeed, many of the most important and revolutionary ideas are first described in remoter areas of the academic literature—it is only after validation there that they make their way into the more “prestigious” mainstream literature. This I hope is the case with the two papers described above. I hope in the next few months to finish revisions, turn them around, and seek publication somewhere in the scientific literature. Rejections are part of the academic game. It is just too bad that nastiness is too.

 

First posted July 2008 at Ethnography.com