Atmospheric Sciences Editors' Vox

Stuff My Reviewers Say

After going through a little more than a decade's worth of reviews, Noah Diffenbaugh shares insights about the peer review process and how reviewer feedback ultimately improved his publications.


Editor’s Note: This post coincides with Peer Review Week (September 19-26), a global event that celebrates the indispensable role peer review plays in maintaining scientific quality. The theme for this years event is Recognition for Review. 

I recently received a particularly critical set of reviews. The manuscript was rejected, and the editor’s decision definitively closed the door to resubmission.

All rejections hurt, and there is always at least a split-second—and sometimes much longer—in which I consider quitting science altogether. But this rejection was particularly deflating, both because I had invested two years in the science, and because two of the three reviewers dismissed the very conception of the analysis. Indeed, one reviewer essentially said that the most convincing aspect was to clarify that what we had attempted should never be attempted again.

In response to my persistent moping, one of my graduate students sent me a link to the Stuff My Reviewers Say Tumblr. (The actual address doesn’t use the word “stuff.”) The mission statement reads, “Collecting the finest real specimens of reviewer comments since 1456.”Authors post reviewer comments to the site, and both the quotations and the hashtags are priceless. After an hour of reading through the archives, I was laughing so hard that I cried—the tears were, as my teenage children would say, literally rolling down my cheeks and off my chin.

I decided to re-read the reviews that I have received as first author, starting with my first submission back in the fall of 2001. Looking back at 15 years of reviews was both sobering and instructive. For one thing, it reminded me that my most recent negative reviews were not that unusual, just the most recent. But it also reminded me how fundamentally my science has benefited from the generous contributions of so many volunteer reviewers, both in the case of each and every paper, and over the longer term development of my research. Taken together, the community of almost entirely anonymous reviewers has acted as an indispensable and almost completely uncredited co-author and mentor.

This has also been my overwhelming observation during my time as editor of Geophysical Research Letters (GRL). My conservative estimate is that I have read ~4,000 reviews in the past seven years. Ultimately, all reviews provide a final evaluation. But reading these thousands of reviews has convinced me that in the vast majority of cases, the primary function of peer review is to provide honest and genuine guidance, insight and advice about the science that has been submitted. It is truly heartening to see just how constructive reviews tend to be, and how much effort reviewers invest in helping the authors. Indeed, even when a reviewer recommends rejection, in most cases they go on to provide a host of detailed, constructive suggestions for how to improve both the science and the presentation.

Although AGU and other organizations are trying to find ways to better acknowledge the hard work of reviewers, the reality is that whether the paper is accepted or rejected, most of the reviewer effort is never known to anyone other than the authors and editor. And in most cases the authors don’t know who so graciously gave them such valuable and generous feedback. And yet, the voluntary generosity of peer reviewers is an indispensable pillar of science, both institutionally and in the case of essentially every published paper.

I note that GRL does occasionally receive reviews that contain careless—or in some cases downright nasty—comments. Although I am glad to report that these are the rare exception, personal attacks and mean language are unprofessional and inappropriate, and don’t have a place in scientific discourse. On behalf of the GRL Editorial Board, I would like to remind the community to be careful about the language that is used, and to refrain from demeaning critiques and personal attacks. And, on the flip side, I would like to remind authors that reviews (and editor decisions) are offered in good faith within the confines of the peer-review process—it is a violation of that good faith (and copyright) for authors to post the  content of reviews publicly without permission. But it is OK to post small snippets, as on the site above.

Going through the reviews I have received, I did find some gems, which I have assembled into a personal Top 10 List. These are all from manuscripts on which I was the first author. And in all cases, they made the paper much better. So thank you, anonymous reviewers—whoever you are!

The Top 10 “Stuff My Reviewers Say” (my personal highlights, 2002–2016)

10. “Two reviewers pointed out that the main conclusion of the paper is tautological or meaningless.”

9. “I am personally a big fan of the single complex figure that tells the whole story, but this figure is beyond me.”

8. “Fig. 1 is pretty terrible… At a minimum, spend some time to make a better-looking pie chart.”

7. “The paper appears at present to be somewhat overcomplicated given it is using some basic maths to make its points.”

6. “One general challenge is that the abstract and conclusions don’t really say anything new”

5. “The use of ‘unprecedented’ is inappropriate or even nonsensical when referring to a subset of the period of record.  For example, my birthday was unprecedented in the past 11 months.”

4. “This section only demonstrates that the crank can be turned, not that the results have any credibility.”

3. “This paper is a source of serious confusion for me… I am not even able to judge the scientific merits of this paper as it stands, though the parts I understand appear to be questionable.”

2. “The approach that is used can be compared to a tapas or dim sum meal; lots of little morsels, but not much depth in any instance.”

1. “As written, the paper reads like a first draft in which you unloaded your stream of consciousness on the computer.”

—Noah S. Diffenbaugh, Editor-in-Chief, Geophysical Research Letters; email: [email protected]

  • wd73383

    Although it is obvious, the key to getting useful reviews is to fire any reviewer who is unqualified. For example, you don’t want a theoretical researcher to review a book on engineering practice no matter how “great” that researcher is.

  • davidlaing

    Regardless of the fact that peer review is volunteered, every reviewer has a Reality Tunnel, or a way of looking at the world that is conditioned by what he/she has been told by others, or has read in books, i.e., influenced by the opinions of others. Whether consciously or not, reality tunnels create filters of orthodoxy in the mind of the reviewer that can unjustifiably exclude valid ideas from consideration as possible explanations of perceived reality. As a concrete example, greenhouse warming theory is regarded by many in the climate field as sacrosanct and beyond questioning, despite the fact that it is mostly theoretical, and there is a lack of hard evidence that shows the effect unequivocally. Nevertheless, ideas regarding alternative sources of warming, such as ozone depletion, are routinely removed from consideration by the peer review process. This serves to perpetuate the sense that the matter is closed to further debate, a very unscientific conclusion. It would be helpful if peer reviewers could suppress their reality tunnels enough to allow new ideas to come forward for debate because that is, after all, what science is all about, the free exchange of reasonable ideas in the ongoing attempt to understand the structure and function of the Earth system, rather than the promulgation of a certain human viewpoint about what that system is probably doing. Earth is never wrong. Scientists claiming to understand Earth often are. Science, as an (ideally) impartial search for Earth’s truths, should be free of such human bias as unfortunately pervades the peer-review process.

    • Brakenridge

      Very few Earth Scientists who are practicing scientists would describe “theoretical” in this way (as being uncertain). Theory is our highest level of understanding. To say that some idea is “mostly theoretical” says, to scientists, that it is very well understood (e.g., understanding of nuclear fission is described by theory; understanding of the mid-oceanic ridges is described by theory). And no matter is “closed to debate” in science; even our most basic and fundamental theories are subject to being revised, expanded, or even discarded. Climate science is simply following the path other fields have followed…hypotheses are generated, tested, causes and effects are identified, proved (or disproved), increasingly accurate understanding leads to increasingly confident predictions. No scientist I know ever claims to “understand Earth”, but they may be able to accurately predict certain causes and effects within nature, including the how CO2 and O3 each affect atmospheric temperature. Of course human bias is very much a part of each individual scientist; what is unique in science, is that we all agree that “Earth is never wrong”, as the above writes; that the most convincing writer will only “win” temporarily, if he or she is wrong about the Earth.

      • davidlaing

        The theory of greenhouse warming is just that, theory, or a mental model of how Earth works, in this case, Arrhenius’s speculation that a geometric increase in atmospheric carbon dioxide concentration results in an arithmetic increase in global temperature. Angstrom experimentally disproved Arrhenius’s theory in 1900, and no hard data from the Earth system has tested the theory as stated since, thanks in large part to a lot of persuasive reasoning by Guy Callendar, Gilbert Plass, Syukuro Manabe, and others. Theory is OK, but it MUST be supported by hard data, or it is simply tilting at windmills, but never mind. Before long, Earth’s natural behavior will show whether or not Arrhenius was right. If he was wrong, no amount of creative arm-waving by the defenders of greenhouse warming will avail, and a new cause for global warming will have to be sought. Ozone depletion by non-explosive volcanoes producing HCl and HBr, which yield Cl and Br when photodissociated on polar stratospheric clouds are prime candidates for this.