Fair warning from an anonymous peer reviewer on one of my academic articles…
The author is hampered by an inaccurate, naïve, and highly simplistic understanding of the basic principles…which leads him to make ludicrous statements like the following…
Yes, that’s me: inaccurate, naïve, and highly simplistic! And so forth. If you share that sentiment, do not read further!
I actually posted a blog about this for the first time in July 2008 after being pummeled in the peer review process. Some anonymous yahoos out in peer-review land accused me of the above transgression and more. What can I say? Only that someone else later thought about the same paper that:
This is a strong paper that makes some interesting connections between advances in contemporary neural science and some early observations from America’s first sociologists. While it treads ground familiar to anyone who has taken introductory sociology (elementary patterns of socialization, affectivity as a social product, empathic understanding), the paper marries this more familiar work to recent ideas emerging from neural science. This makes it a novel contribution. In particular, the claim that sociologists have known of the imminently social character of human beings all along and didn’t need fMRI to discover it, strikes me as a bold one, but a claim that is worth making, especially since it reaffirms the value and relevance of sociological concepts to those beyond the discipline’s boundaries.
But even that journal rejected the paper, too. My neurology paper may be bold, but that review was not enough to get the paper actually accepted for a long time. For a long time, anonymous reviewers sent the paper went down in flames. This phoenix of a paper actually went through six years of peer reviews, in the process collecting a range of laudatory and insulting reviews.
It is somehow believed that “peer review” is the gold standard of academic achievement. Really?
Here is what a couple of hotshot editors are reported by Wikipedia to have said:
Drummond Rennie, deputy editor of Journal of the American Medical Association is an organizer of the International Congress on Peer Review and Biomedical Publication, which has been held every four years since 1986. He remarked:
“There seems to be no study too fragmented, no hypothesis too trivial, no literature too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print.”
Richard Horton, editor of the British medical journal The Lancet, said:
“The mistake, of course, is to have thought that peer review was any more than just a crude means of discovering the acceptability—not the validity—of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong.”
This assumption persists even in the context of well-publicized fraud scandals involving well-funded high fliers in physics, human cloning, and cancer research all indicate that peer reviewers at journals like Science and Nature are somehow sloppier than the social scientific journals I typically send my papers to–so far as I know, none of the journals where I have published have such a sordid record. They are just nasty.
Nevertheless, peer review does often add to the seriousness of academic publication. Plus, if you did not have peer review, as is often said, you are no better than a newspaper, a blog or (horrors) Wikipedia!
But, that does not mean that peer review is always encouraging, nurturing, or even fair. Sometimes peer review is only tin beneath the gold plate. Peer reviewers with the cloak of anonymity permit insecure scientists the chance to level the artillery at potential competitors.
On top of that, editors do not always do their part by protecting writers from the more unreasonable attacks. Does this make for better science? Maybe. My own view is that in the long-run peer review makes for a more careful and conservative science-if that is “better” I’ll let you decide. But in the short run it often adds fuel to the insecurities of the most vulnerable in our midst—the graduate students, untenured faculty, and others who are kept at arms length by self-described tenured “gatekeepers.”
In short, peer review can discourage challenges to the status quo, even though such challenge is what good science is about in the first place. Most crucially, writers without a thick skin are discouraged from pursuing ideas further (whether good or bad), all because some anonymous reviewer had a fight with their spouse or teenager that morning, and took it out on you.
Scientific Publication—The Theory.
But I still think that the ideal of peer review is good. It is that rational, unbiased, and anonymous experts evaluate the work of others to verify whether an idea is new, rigorous, and important enough for publication. You submit a paper to a journal, and then the editor selects colleagues within your discipline to read what you have written.
Anonymity is important to this process (ideally both the reviewer and reviewee do not know who each other are), because there are friendship cliques and elites within the scientific community that bias review. Papers judged by editors as “possible for publication” are sent to reviewers selected for their expertise. The reviewers then submit their reasons for acceptance or rejection to the editor. Such reasons ideally entail 2-3 pages (single spaced) discussing the strengths and weaknesses of a paper’s data and argument, which are then forwarded anonymously to the author. Often, suggestions are made about literature that may have been missed in the paper, irrespective of whether the paper is accepted or rejected.
Hopefully, this results in a “revise and resubmit,” though “reject” is more common. With revise and resubmit, a paper often has up to five reviewers (plus the editor) read and make anonymous comments for the author. Because so many minds are focused on the development of the paper, the overall quality, rigor, and accuracy is often improved. Survive this, and you get a final acceptance which is important to anacademic community which controls jobs, promotions, and the distribution of status.
Between first submission, and the final arrival of a paper in print, months, and possibly years may pass. But this care is why your anthropology professor prefers to see you cite the American Journal of Sociology, American Anthropologist, or Social Forces, rather than Newsweek, CNN’s website, Ethnography.com, Wikipedia, or even Encyclopedia Britannica. All of these sources may be edited for style. Most importantly, though,there is no expert review of the scholarly reasoning.
The result of all this peer-reviewing literature is a scientific literature which academics (and especially graduate students) pore over. The peer-reviewed literature is considered valid and reliable because it has been through the “rigorous” review process. Acceptance rates in the most prestigious journals are often less than 10%, meaning that only the self-described “very best” is published, while the rest is rejected and perhaps submitted to a less prestigious journal, or perhaps find itself into publication in an “edited collection” prepared by a group of colleagues interested in a shared subject. Note that neither of these final two conditions are all that bad, since they do indeed put a new idea “out there” for those diligent graduate students to find. This is particulalry the case in the modern world of the internet where with a little navigation skill, and the help of Interlibrary Loan services, publications from anywhere in the world can be located, often in hours. Still, the stamp of approval from a “prestigious” journal makes it more likely to be noticed by a wide audience tuned.
Scientific Publication—The Practice.
Anyway, that’s the theory of peer review. As I indicated above, in 2007-2008, I went through the process with two separate articles and a book proposal four times in six months. Only sometimes did the process meet the ideal. The book proposal resulted in a contract, and eventually a book, Schooling, Bureaucracy, and Childhood. One article on neurology was flat out rejected once, and then received from another journal a “rejection but you can submit again,” which eventually resulted in a flat out rejection, and another round of peer review. The third paper about African history was rejected, and the editor recommended I pay attention to one reviewer’s comments, and submit to another journal.
All together, the reviews during this time incorporated the opinions of six reviewers. Two were brief, insulting, and without redeeming value. They dismissed my work in a few short lines. One was insulting, but made good recommendations about things that should be incorporated in the article. One was frustrated with my “sloppiness” but the reviewer thought the paper was worth a “revise and resubmit” which the editor did not give me. The fifth thought the paper was worthwhile, but needed to be fleshed out for the “new parts” more, and the editor gave me the “reject but you can resubmit in a revised form.” The last was the “accept.”
In other words, three of the reviews were constructive, and reflect the very best of the peer review process. Two of them reflect the worst impulses found in the review process. The one which was insulting (called me naive, etc) still gave some good suggestions.
Here is a sampling, with some of my own ripostes:
…There is little that is based on original research and no substantial intellectual or theoretical content…I am sorry to be so negative, but this [paper] is simply a non-starter. (This comment was on a 40+ page paper, and the whole review was only about six sentences long. This reviewer has an ego problem and is lazy).
The second review on the same paper was three pages long, and pointed out in excruciating detail a number of errors on my part:
Despite this rather frustrating sloppiness [which was pointed out in excruciating detail], I am willing to see the author revise and resubmit… (ok, ok, you got me this time…I will go back and fix things)
Comments on the sociology and neurology article included the following. First the extremely short dismissive review:
This leads him to highly fatuous arguments… (Not as fatuous as your stupid review).
A second comment on the same paper:
The author is hampered by an inaccurate, naïve, and highly simplistic understanding of the basic principles…which leads him to make ludicrous statements like the following…(this review included some good references to what the reviewer thought were key to the discipline, so he got me on that one. I will cite them, but also note that they present an inaccurate, naïve, and highly simplistic understanding of basic sociological literature…which leads to ludicrous statements. Except I will say this with more respect, and not anonymously.)
The neurology paper was resubmitted to another journal after I took a number of issues raised in the second review into account. I received the following comments back:
I’m very sympathetic to one of the paper’s central claims…but I don’t believe that the paper as a whole has a sufficiently clear and sustained focus. .. What exactly do the two ideas have in common (apart from a central metaphor) and how do they differ? What can we learn from the comparison … But to make a substantial contribution to this more general debate, it would need to canvas a range of examples, … and to break some ground; advance some new arguments or shed new light on old ones. (This comment ended in a rejection and resulted from the comment below from the editor. But thanks for the thoughtful comments!)
I agree with the reviewer`s opinion that the basic line of thought in this paper is interesting and plausible. But I think the reviewer is also probably right that these basic ideas need more sustained development… (ok, you have a good point. I will do it, and get back to you in a couple of months which incorporate some of the specific points raised—thanks for being encouraging even though this was not an acceptance!)
And finally a note from the one acceptance out of the four submissions:
I’m not sure if I have a plan to order things differently than they are currently ordered, but it strikes me as potentially a little awkward…(I think that this reviewer was probably right—but any type of acceptance makes me pretty happy!)
My own strategy for working with this range of commentary, is to assume that anything complimentary is really correct, suggestions for including other books as a citation should always be followed, and that anyone that includes words like “fatuous,” “naïve,” or “ludicrous” means that I have a really good paper that justifiably ruffled feathers, and I should try again. As for the reviewer? That person is in need of psychiatric help.
What I like about Anonymous Peer Review.
So there you have peer review, from the nasty to the constructive. If you are ever asked to do peer a review, I would urge you to avoid the nasty side—visit a therapist instead. Be constructive in your comments, even if your conclusion is to “reject.” Remember too, that many papers go through many iterations—papers are only rarely accepted on the “first try.” My own experience is that papers might be accepted on the second to fifth try. Or even the twelfth try.
The mirror neuron holds my record, having been rejected by a motley collection of psychology, sociology, and biological journals. Who would have guessed that it would be eventually accepted by a Philosophy journal? But of all the papers I have written, in my view it is the most original—and also the most difficult to get published. First keystrokes were in 2007, first submission in 2008, and actual publication was in 2014!
Usually—though not always—the peer review process is a constructive part of developing a paper. There are a lot of journals out there, and a rejection is sometimes the luck of the draw. How could the editor have known that the reviewer he met a conference five years before had tortured frogs as a child, and was also going through a bad divorce? Ignore the comments about being naïve, simplistic and ludicrous which probably tell you more about the reviewer’s mental health than the quality of your paper. Fix what is fixable, while recognizing that good papers by definition displease.
While peer review sometimes (but not always) eliminates some poor scholarship, in my view the greatest contribution peer review offers is its capaicty to encourage and nurture good scholarship. Some of the more prestigious journal in sociology note this, telling reviewers that despite the fact that 90% of the submissions are not published, Comments are important because eventually many papers are published somewhere. What they don’t note is that they are also rejecting some of the best sociology.
Indeed, many of the most important and revolutionary ideas were first described in remoter areas of the academic literature. In part this happens because the papers were first nastily received at the prestigious “mainstream” journals which are so heavily vetted by the big shots. It is only after validation in the nether reaches of a discipline that the conservative review process means that they great ideas make their way into the more “prestigious” mainstream literature.
Which still doesn’t explain how fraudulent papers get through the “rigorous” process. I still can’t figure out how the fraudulent writers for Nature and the other journals managed to get their papers published when so many anonymous reviewers come after my papers with chainsaws!
Reference List: The Fatuous, Naïve, and Bold
Waters, Tony (2014). Of Looking Glasses, Mirror Neurons, and Meaning. Perspectives on Science. Behind Paywall available here:
Prepublication Version available here: https://www.academia.edu/5064752/Of_Looking_Glasses_and_Mirror_Neurons–Manuscript_Version
Waters, Tony (2012). Schooling, Childhood, and Bureaucracy: Bureaucratizing the Child. New York: Palgrave MacMilllan.
Free Chapter Available Here. Also can recommend to libraries! http://www.palgraveconnect.com/pc/doifinder/10.1057/9781137269720.0005
Waters, Tony (2009). Social Organization and Social Status in Nineteenth and Twentieth Century Rukwa, Tanzania. African Studies Quarterly 11(1)