Fair warning from an anonymous peer reviewer of one of my academic articles…
The author is hampered by an inaccurate, naïve, and highly simplistic understanding of the basic principles…which leads him to make ludicrous statements like the following…
Yes, that’s me: inaccurate, naïve, and highly simplistic! And so forth. If you share that sentiment, do not read further.
I posted a blog about peer review for the first time in July 2008 after being pummeled in the peer review process. Some anonymous yahoos out in peer-review land accused me of the above transgression and more. What can I say? Only that someone else later thought about the same paper the following.
This is a strong paper that makes some interesting connections between advances in contemporary neural science and some early observations from America’s first sociologists. While it treads ground familiar to anyone who has taken introductory sociology (elementary patterns of socialization, affectivity as a social product, empathic understanding), the paper marries this more familiar work to recent ideas emerging from neural science. This makes it a novel contribution. In particular, the claim that sociologists have known of the imminently social character of human beings all along and didn’t need fMRI to discover it, strikes me as a bold one, but a claim that is worth making, especially since it reaffirms the value and relevance of sociological concepts to those beyond the discipline’s boundaries.
But even the journal receiving that review rejected the paper. My neural science paper may have been bold, but that review was not enough to get the paper actually accepted for a long time. I guess reviewers and editors were more into the “naïve” evaluation, and so the paper went down in flames repeatedly. This phoenix of a paper actually went through four or five years of peer reviews, in the process collecting a range of laudatory and insulting reviews.
It is somehow believed that “peer review” is the gold standard of academic achievement. Really?
Here is what a couple of hotshot editors are reported by the non-peer reviewed Wikipedia to have said:
Drummond Rennie, deputy editor of Journal of the American Medical Association is an organizer of the International Congress on Peer Review and Biomedical Publication, which has been held every four years since 1986. He remarked:
“There seems to be no study too fragmented, no hypothesis too trivial, no literature too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print.
Richard Horton, editor of the British medical journal The Lancet, said:
“The mistake, of course, is to have thought that peer review was any more than just a crude means of discovering the acceptability—not the validity—of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong.
But the assumption about the “gold standard” persists even in the context of well-publicized scandals involving well-funded high fliers in fields like physics, human cloning, and cancer research that all indicate that peer reviewers at journals like Science and Nature are somehow sloppier than the social scientific journals I typically send my papers to. Indeed, so far as I know, none of the journals where I publish have such a sordid record (certainly SocJournal does not!), but the journals I send my papers to can indeed send me back nasty reviews.
Despite all this, I still think peer review does often add to the seriousness of academic publication. Plus, if you did not have peer review, as is often said, you are no better than a newspaper, a blog or (horrors) Wikipedia!
But, that does not mean that peer review is always encouraging, nurturing, or even fair. Sometimes peer review is only tin beneath the gold plate. Peer review with the cloak of anonymity permit insecure scientists the chance to level the artillery at potential competitors. Editors in turn do not always do their part by protecting writers from the more unreasonable attacks. Does this make for better science? Maybe. My own view is that in the long-run peer review makes for a more careful and conservative science–if that is “better.” But in the short run it often adds fuel to the insecurities of the most vulnerable in our midst—the graduate students, untenured faculty, and others who are kept at arms length by self-described anonymous tenured “gatekeepers.”
In short, peer review discourages challenges to the status quo, even though such challenge is what good science is about in the first place. Most crucially, writers without a thick skin are discouraged from pursuing ideas further (whether good or bad), all because some anonymous reviewer had a fight with their spouse or teenager that morning, and took it out in the peer review.
Scientific Publication—The Theory. But I still think that the ideal of peer review is good. The theory is that rational, unbiased, and anonymous experts evaluate the work of others to verify whether an idea is new, rigorous, and important enough for publication. You submit a paper to a journal, and then the editor selects unbiased experts within your discipline to read what you have written.
Anonymity is important to this process (ideally both the reviewer and reviewee do not know who each other are), because there are friendship cliques (and elites) within the scientific community. Papers judged by editors as “possible for publication” are thusly sent to reviewers selected for their expertise without the name of the author. The reviewers then submit their reasons for acceptance or rejection to the editor. Such reviews ideally entail 2-3 pages (single spaced) discussing the strengths and weaknesses of a paper’s data and argument, which are then sent anonymously back to the author. Often, suggestions are made about literature that is missing, irrespective of whether the paper is accepted or rejected.
Authors hope that this results in a “revise and resubmit,” though “reject” is more common. With revise and resubmit, a paper often has about five reviewers (plus the editor) who read and make anonymous comments. Because so many minds are focused on the development of the paper, the overall quality, rigor, and accuracy is often improved. Survive this, and you might get a final acceptance which is important to an academic community which controls jobs, promotions, and the distribution of status.
Between first submission, and the final arrival of a paper in print, months, and possibly years may pass. This care is why your professor prefers to see you cite the American Journal of Sociology, American Anthropologist, or Social Forces, rather than Newsweek, CNN’s website, Wikipedia, or even Encyclopedia Britannica (Socjournal of course is in its own category which will indeed be apparent to the PhDs graduating in 2020!). All of these sources may edit for content and style, though they may not go through the formal peer review process which is so careful and conservative.
The result in the peer-reviewed literature is a science which academics (and especially graduate students) pore over. The peer-reviewed literature is more considered valid and reliable because it has been through a “rigorous” review process. Acceptance rates in the most prestigious journals are often less than 10%, meaning that only the peer-described “very best” is published. Often unsaid is that the rest is rejected but often submitted to a less prestigious journal, or perhaps finds itself into publication in an “edited collection” prepared by a group of colleagues interested in a shared subject. Note that neither of these final two conditions are all that bad, since they do indeed put a new idea “out there” for that diligent graduate student to find. Indeed, today “out there” includes the readily accessible world-wide-web open to everyone, while that prestigious publication is tucked by a pay-wall accessible only to those with expensive library privileges. Still, the stamp of approval from a “prestigious” journal makes it more likely to be noticed by the “right” audience.
Scientific Publication—The Practice Anyway, that’s the theory of peer review. As I indicated above in my 2008 blog, in 2007-2008, I went through the process with two separate articles and a book proposal four times in six months, and only sometimes did the process meet the ideal. The book proposal resulted in a contract, and eventually a book, Schooling, Bureaucracy, and Childhood (2012). The article on neural sciences was flat out rejected once within a few weeks, and then shortly after received from a “rejection but you can submit again.” This “reject but resubmit” eventually resulted in another flat out rejection after another round of peer review. The third paper about African history was rejected, and the editor recommended I pay attention to one reviewer’s comments, and submit to another journal. I did that, and it was published about a year later.
Altogether, the reviews during just those six months incorporated the opinions of six reviewers. Two reviews were brief, insulting, and without redeeming value. They dismissed my work in a few short lines. One was insulting, but made good recommendations about things that should be incorporated in the article. One was frustrated with my “sloppiness” but the reviewer thought the paper was worth a “revise and resubmit” which the editor did not give me. The fifth thought the paper was worthwhile, but needed to be fleshed out more, and the editor gave me the “reject but you can resubmit in a revised form.” The last was the “accept.” In other words, three of the reviews were constructive, and reflect the very best of the peer review process. Two of them reflect the worst impulses found in the review process. The one which was insulting (called me naive, etc) still gave some good suggestions.
Here is a sampling, with some of my own ripostes, which six years later in 2014 still make me feel better!
…There is little that is based on original research and no substantial intellectual or theoretical content…I am sorry to be so negative, but this [paper] is simply a non-starter. (This comment was on the 40+page Africa Studies paper, and the whole review was only about six sentences long. This reviewer has an ego problem and is lazy).
The second review on the same paper was three pages long, and pointed out in excruciating detail a number of errors on my part:
Despite this rather frustrating sloppiness [which was pointed out in excruciating detail], I am willing to see the author revise and resubmit… (ok, ok, you got me this time…I will go back and fix things).
Comments on the neural science article included the following. First the extremely short dismissive review:
This leads him to highly fatuous arguments… (Not as fatuous as your silly review).
A second comment on the same paper:
The author is hampered by an inaccurate, naïve, and highly simplistic understanding of the basic principles…which leads him to make ludicrous statements like the following. (This review included some good references to what the reviewer thought were key to the discipline. I will cite the suggested references, but also note that they present an inaccurate, naïve, and highly simplistic understanding of basic sociological literature…which also leads to ludicrous statements. Except I will say this with more respect, and not anonymously).
The neural science paper was then resubmitted to another journal. After I fixed a number of issues later raised in the second review, I received the following comments back in 2010:
I’m very sympathetic to one of the paper’s central claims…but I don’t believe that the paper as a whole has a sufficiently clear and sustained focus… What exactly do the two ideas have in common (apart from a central metaphor) and how do they differ? What can we learn from the comparison… But to make a substantial contribution to this more general debate, it would need to canvas a range of examples,… and to break some ground; advance some new arguments or shed new light on old ones. (This comment ended in a rejection and resulted from the comment below from the editor. But thanks for the thoughtful comments!)
The editor in the rejection letter responded:
I agree with the reviewer’s opinion that the basic line of thought in this paper is interesting and plausible. But I think the reviewer is also probably right that these basic ideas need more sustained development… (Ok, you have a good point. I will do it, and in the submission to another journal incorporate some of the specific points raised—thanks for being encouraging even though this was not an acceptance!)
And finally a note from the one acceptance out of the four submissions, on my book proposal about “Bureaucratizing the Child”
I’m not sure if I have a plan to order things differently than they are currently ordered, but it strikes me as potentially a little awkward…(I think that this reviewer was probably right—but any type of acceptance after so much rejection makes me pretty happy!)
My own strategy for working with this range of commentary, is to assume that anything complimentary is entirely correct, suggestions for including other books as a citation should generally be followed, and that anyone that includes words like “fatuous,” “naïve,” or “ludicrous” means that I have a really good paper that justifiably ruffled feathers, and I should try again. As for the reviewer? That person is in need of psychiatric help.
What I like about Anonymous Peer Review. So there you have peer review, from the nasty to the constructive. If you are ever asked to do a peer review, I would urge you to avoid the nasty side—visit a therapist instead. Be constructive in your comments, even if your conclusion is to “reject.” Remember too, that many papers go through many iterations—papers are only rarely accepted on the “first try.” My own experience is that papers might be accepted on the second to fifth try. Or even the twelfth try.
My mirror neuron article holds my record, having been rejected by a motley collection of psychology, sociology, and biological journals between 2008 and 2012. Who would have guessed that it would be eventually accepted by a Philosophy of Science journal? But of all the papers I have written, in my view it is the most original—and perhaps this is why it was most difficult to get published. First keystrokes were in 2007, first submission in 2008, and actual publication was in 2014!
Usually—though not always—the peer review process is a constructive part in developing a paper. There are many journals out there, and a rejection is sometimes the luck of the draw. How could the editor have known that the reviewer he met a conference five years before had tortured frogs as a child, or was also going through a bad divorce? So ignore the comments about being naïve, simplistic and ludicrous which probably tell you more about the reviewer’s mental health than the quality of your paper. Fix what is fixable, while recognizing that good papers by definition displease.
While peer review sometimes (but not always) eliminates some poor scholarship, in my view the greatest contribution peer review offers is the capacity to encourage and nurture good scholarship. Some of the more prestigious journal in sociology note this, telling reviewers that despite the fact that 90% of the submissions are not published, comments are important because eventually many papers are published elsewhere. What they don’t note is that they are also rejecting some of the best sociology, because of the inherently conservative nature of peer review.
Indeed, many of the most important and revolutionary ideas were first described in remoter areas of the academic literature. In part this happens because the papers were first nastily received at the prestigious “mainstream” journals which are so heavily vetted by the big shots. It is only after validation in the nether reaches of a discipline that the great ideas make their way into the more “prestigious” mainstream literature.
Which still doesn’t explain how fraudulent papers get through the “rigorous” process at places like Nature and Science. I still can’t figure out how the fraudulent writers managed to get their papers published when so many anonymous reviewers come after my papers with chainsaws!
Reference List
Waters, Tony (2014). Of Looking Glasses, Mirror Neurons, and Meaning. Perspectives on Science. Behind Paywall available here:
http://www.mitpressjournals.org/doi/pdf/10.1162/POSC_a_00152
Prepublication Version available here: https://www.academia.edu/5064752/Of_Looking_Glasses_and_Mirror_Neurons–Manuscript_Version
Waters, Tony (2012). Schooling, Childhood, and Bureaucracy: Bureaucratizing the Child. New York: Palgrave MacMilllan.
Free Chapter Available Here. Also can recommend to libraries! http://www.palgraveconnect.com/pc/doifinder/10.1057/9781137269720.0005
Waters, Tony (2009). Social Organization and Social Status in Nineteenth and Twentieth Century Rukwa, Tanzania. African Studies Quarterly 11(1)
http://asq.africa.ufl.edu/files/Waters-V11Is1.pdf
Tony Waters is czar and editor of Ethnography.com. He came to us from the Sociology department at California State University at Chico where he has been a professor since 1996. In 2016 though he suddenly found himself with a new gig at Payap University in northern Thailand where he is on the faculty of the Peace Studies Department. He has also been a guest professor in Germany, and Tanzania. In the past, his main interests have been international development and refugees in Thailand, Tanzania, and California. This reflects a former career in the Peace Corps (Thailand), and refugee camps (Thailand and Tanzania). His books include: Crime and Immigrant Youth (1999), Bureaucratizing the Good Samaritan (2001), The Persistence of Subsistence Agriculture: Life Beneath of the Marketplace (2007), When Killing is a Crime (2007), and Schooling, Bureaucracy, and Childhood: Bureaucratizing the Child (2012). His hobby is trying to learn strange languages–and the mistakes that that implies. Tony is a prolific academic, you can read more of his work at academia.edu.or purchase one (or more!) of his books from Amazon.com.