Editorial pubs.acs.org/cm
Pre-Publication Peer ReviewEvidence and Editors he publishing world is in flux. From open access,1 to the advent of reports-style journals and mega-journals,2,3 to post-publication peer review (i.e., Pubmed Commons,4 PubPeer5), many scientists are openly, and in some cases vociferously, discussing the best way(s) of communicating their science to peers, to their communities, and to those who support and fund research. Vigorous discussions are important to air challenges and problems, but emotions can run high, and we all know in practice that best decisions are never made when one’s blood is boiling. No system designed and run by humans will ever be perfect, and I am concerned that the calls to modify or eliminate anonymous prepublication peer review are not balanced by thoughtful consideration of its positive and constructive aspects. Again, I do recognize that anonymous prepublication peer review is not flawless, as made clear by recent events in the publishing world, but let us not throw the baby out with the bathwater, out of a rash need “to do something”. As editors and practicing scientists, all ACS editors wear both hats. Speaking about my own experience, I am extremely grateful to pre-publication peer review, as the anonymous comments made by various referees over the years have been, for the most part, insightful and educational, and have pushed us to strive for better. Of course, peer review can sting, and on the weekend, a manuscript submitted to an ACS journal was declined after review. One of the reviewers seemingly missed the point, but that made us realize that we were not sufficiently clear. Since we do not want future readers to arrive at the same conclusion, we have brushed ourselves off and will run two additional experiments, rewrite the egregious portions of the manuscript, and move on. Careful analysis by the editor also helped focus our attention on the weakness. As noted in 2012, publications that have been declined, presumably revised, and resubmitted received significantly more citations due to, as described by the authors of this study, inputs from editors and reviewers and extra work that led to improvements of the manuscript:6 “Several mechanisms (for the increased number of citations of resubmitted papers) could be involved, but perhaps the most likely explanation is that inputs from editors, reviewers, and the greater amount of time spent working on resubmissions signif icantly improve the citation impact of the f inal product. There are indications of the value of peer review f rom publication and editorial practice.7,8 Our results suggest that it extends to citation impact... Perhaps more important, these results should help authors endure the f rustration associated with long resubmission processes and encourage them to take the challenge.7,9” Two blogs on the subject of peer review caught my eye this week, written by two science editors, Teri Odom at ACS Photonics, entitled “How to Remove Bias From Peer Review”,10 and another by Hilda Bastion at PLoS Medicine, “Weighing Up Anonymity and Openness in Publication Peer Review”;11 the two started from different points on the spectrum of arguments
T
© 2015 American Chemical Society
and yet converged on a similar conclusion regarding the role of editors. Both blogs are worth reading, and from them I drew the following conclusions. i. Editors review the reviews. The now well-known episode of the terribly sexist peer review (female authors being told by the one reviewer to bring on one or two male coauthors) brought to light a rather nasty case of an editor not “reviewing the reviews”.12,13 An important task of an editor is to review the reviews, judge the validity of the recommendations of the referees, and consider possible sources of bias. Ultimately, the editors provide rigor and careful appraisal of the manuscript and the reviews, and oversee the application of accepted standards to both author and reviewer. Authors choose journals that have editors whom we trust to be thoughtful, tough, and fair since they can parse and evaluate the reviews, and spend the time making the best decision with the data at their disposition. Teri Odom, executive editor of ACS Photonics, recently wrote the following regarding the task of editors to essentially “review the peer review”:10 “Editors of reputable scientif ic journalspracticing scientists and professional editorshave an obligation to ensure a fair review process and to protect authors f rom inappropriate and unsubstantiated comments. If editors are not held accountable, increased incidents of poor peer review and bias are bound to occur.” Odom then proposed consideration of a more formal system to protect against demeaning comments:10 “Perhaps a system of review for editors and reviewers would provide a check for demeaning comments. For example, if an editor fails to communicate that such commentary is not acceptable, then she or he should step down f rom that position. And if a reviewer makes outrageously biased comments more than once, then he or she would be barred f rom publishing in and reviewing for that family of journals.” Bastion discussed the accountability of editors as gatekeepers of publication, since the impact of bias could have broad impact on careers and could drive people from science:11 “Substantiating our statements, and being accountable for what we say and how we say it when we are gatekeepers for publication, is decisive for me. That’s all the more important for people whose work or critique loses out because of status bias, and those who may be repelled from publishing and science by reviewer aggressiveness.” ii. Should all anonymous peer review be double blinded? Should we reveal reviewer identity? Bastion’s blog is a meta-analysis of 17 studies regarding anonymity in peer review. Bastion calls for evidence-based decision making since grounding a critical decision, such as modifying the peer review process, on opinion alone could lead to unintended consequences:11 Published: June 9, 2015 3783
DOI: 10.1021/acs.chemmater.5b01917 Chem. Mater. 2015, 27, 3783−3784
Chemistry of Materials
Editorial
“Determining practice based on a consensus of practitioner opinion doesn’t always end well, though, does it? Even wellintentioned interventions can cause harmor have no real consequences at all, leaving the problems untouched. And there’s a lot that can go wrong with editorial review.” Bastion summarized the results of the various studies with respect to three questions, below. As an initial disclaimer, dissatisfaction was expressed with respect to some of studies, including problems such as nonrandomization and lack of tests for multiplicity, and thus concrete evidence to enable the scientific community to make informed decisions may be lacking. Q: Does double-blinding (i.e., masking author identity, keeping peer review anonymous) make a difference to the outcome of peer review? A: With the data available, it was clear that failure of blinding was common (i.e., the reviewers were able to identify the authors), and overall, masking author names had little effect on the outcome. Q: Did disclosing peer reviewer identities affect the outcome, and quality and collegiality of the referee reports? A: It was noted that, on average, the reviews were not deemed to be better, although there was a greater likelihood of substantiating their points and criticisms when it was clear that the reviewer identity would be shared. Reviews did not appear to be less critical, on average. The one important outcome that stands out, however, was that many potential peer reviewers declined the invitation to review if there was a chance they would be named, which is an important unintended consequence. If I know that I am going to suggest declining the paper of a colleague who may be about to review my grant proposal, would I take that risk? Many would not, and thus we may eliminate a large fraction of the population of potential reviewers by disclosing peer reviewers. Q: Does carrying out double-blinding, or sharing peer review identity (above 2 items) affect bias in the reviews? A: The data was insufficient to definitively answer this question. Bastion concludes by saying that we do not have the evidence, either way, to make a decision regarding anonymity in peer review:11 “Author and peer reviewer anonymity haven’t been shown to have an overall benef it, and they may cause harm.” “...we don’t have an overwhelming evidence basis for anything. In my opinion, the weight tilts in one direction at the moment, with a trade-of f of potential harms we don’t know enough about. This ongoing knowledge gap is letting science and scientists down.”
spotting and rooting out bias when making decisions and recommendations. We feel the weight on our shoulders, as we should. I look forward to more discussion on the topic.
■
Jillian M. Buriak, Editor-in-Chief
AUTHOR INFORMATION
Notes
Views expressed in this editorial are those of the author and not necessarily the views of the ACS.
■
REFERENCES
(1) Geiger, F.; Pedersen, J. How Open is Open Access? J. Phys. Chem. Lett. 2015, 6, 1246−1248. (2) Kamat, P. V.; Schatz, G. C. Know the Difference: Scientific Publications versus Scientific Reports. J. Phys. Chem. Lett. 2015, 6, 858−850. (3) Buriak, J. M. Mega-Journals and Peer Review: Can Quality and Standards Survive? Chem. Mater. 2015, 27, 2243−2243. (4) http://www.ncbi.nlm.nih.gov/pubmedcommons/. (5) https://pubpeer.com. (6) Calgano, V.; Demoinet, E.; Gollner, K.; Guidi, L.; Ruths, D.; de Mazancourt, C. Flows of Research Manuscripts Among Scientific Journals Reveal Hidden Submission Patterns. Science 2012, 23, 1065− 1069. (7) Clapham, P. Publish or Perish. Bioscience 2005, 55, 390. (8) Bakanic, V.; McPhail, C.; Simon, R. J. The Manuscript Review and Decision-Making Process Am. Sociol. Rev. 1987, 52, 631. (9) Weber, E. J.; Katz, P. P.; Waeckerle, J. F.; Callaham, M. L. Author Perception of Peer Review: Impact of Review Quality and Acceptance on Satisfaction. JAMA 2002, 287, 2790. (10) http://chronicle.com/blogs/conversation/2015/05/07/how-toremove-bias-from-peer-review/. (11) http://blogs.plos.org/absolutely-maybe/weighing-upanonymity-and-openness-in-publication-peer-review/. (12) http://blogs.plos.org/everyone/2015/05/01/plos-one-updatepeer-review-investigation/. (13) http://news.sciencemag.org/scientific-community/2015/04/ sexist-peer-review-elicits-furious-twitter-response.
■
CONCLUSIONS As scientists, we demand rigor of ourselves when carrying out experiments and making conclusions based upon the evidence. We submit our manuscripts for peer review, where they are put under the spotlight and refereed by experts in our field. We should demand the same level of rigor with respect to analyzing peer review, as we consider potential changes. Evolution of opinions through open discussion is important and is how science is best practiced, but if not based upon rational evidence and thorough analysis, alterations should not be made in a rash fashion. I do believe, however, that as a result of these discussions, the critical role of the editor in the process is starting to draw more attention, in particular with respect to 3784
DOI: 10.1021/acs.chemmater.5b01917 Chem. Mater. 2015, 27, 3783−3784
