Reviewers are critical to ensure quality and rigor in scientific publications. In most cases, their contributions are hidden because of our masked peer review system but, while many choose to remain anonymous to authors, it’s important that we know who they are and where they are. To mark Peer Review Week 2021, we wanted to share how AGU has been tracking and analyzing reviewer demographics. We shared previous work on reviewer demographics [Hanson and Lerback, 2017; Lerback et al., 2020] and, over the past few years, have continued to work on expanding the reviewer pool to engage a wider range of perspectives and ultimately publish more inclusive science benefiting larger sector of society. Expanding our reviewer pool also helps to supply an increasing demand for peer reviews as scientific output and submissions increase.
Sources of Demographic Information
AGU has several methods for identifying reviewer demographics. First, we use our member database to match authors and reviewers in our journal submission system with their member profiles containing gender, age, and race/ethnicity information. When gender is not given in the member profile, we use the gender-name database Gender API, which uses first name and country to guess and we keep guesses with +90% confidence score. Some of the race/ethnicity categories are based on U.S. census categories (e.g., Asian American), so our current race/ethnicity data only applies to U.S.-based reviewers. However, we are in the process of updating our system so that the categories accurately describe the global population. Country of residence is determined from a person’s profile in the submission system. For those that don’t have it, we use their email suffix (when it identifies a country).
Gender of Reviewers
The number of invitations to review a paper sent to women (both total counts and percentages) has been increasing since 2016, although the rate at which they agree to review is slightly lower than male invitees, so the proportion of women agreeing to review papers (“final (agreed) reviewers” in the chart below) has nominally increased each year:
It’s tricky to develop a specific target for invitations to women, as reviewing receives less professional credit than publishing articles, leading research groups, chairing department committees, and other activities. Some members of the community are worried that over-burdening women scientists with reviewing will take their attention away from activities that more overtly advance their careers.
AGU is currently working on expanding co-reviewing opportunities, which will allow senior scientists to partner with those earlier in their careers to work together on reviews.
However, by increasing reviewing opportunities for early career scientists, which, in the Earth and space sciences includes a higher proportion of women than older cohorts, younger scientists will benefit more from reviewing papers than those advanced in their careers. AGU is currently working on expanding co-reviewing opportunities, which will allow senior scientists to partner with those earlier in their careers to work together on reviews. This will help train early career scientists in peer review and expand future opportunities to review.
Geographic Region of Reviewers
An interesting comparison is between the location of the invited reviewer and the geographic region where our corresponding authors are located, as shown in the chart below. Though we’ve seen an increase in the proportion of review invitations sent to regions where we see the fastest growth in authors submitting papers, namely China, the increase in invitations isn’t as large as the increase in accepted papers from China. Additionally, the proportions of invitations sent to U.S.-and Europe-based reviewers have been declining but are still higher than their representation in our pool of authors whose papers are published.
Age of Reviewers
It’s important to disaggregate gender data by age group to see where specific improvements can be made. These next charts show age groups of reviewers, authors, and AGU members.
People invited to review papers are predominately mid- to late-career men (40s-60s). The largest cohort of women invited to review are in their 30s.
We typically compare invited reviewers to authors of accepted papers and members, shown below. Both authors and members are skewed slightly younger than invited reviewers, which indicates editors are more comfortable inviting those in the mid- and later-career stages. This could be due to the expertise of those in these stages (including being known published authors) or because these reviewers are a part of the editor’s professional network.
Increasing reviewing opportunities to younger scientists could help decrease the burden on mid-career scientists who likely have more teaching, administration, and/or family-related duties while having the benefit of training younger scientists earlier in their career on how to do a good peer review.
Reviewer Workload
Another interesting way to assess the diversity of our reviewer pool is to calculate the number of invitations per person within various demographic groups. For example, are we inviting the same men more often than we’re inviting the same women? We can calculate this by dividing the number of total invitations by the number of distinct email addresses belonging to that demographic group, with the chart below showing the resulting invitations per person.
The results show that we invite fewer women than men and much fewer unknown gender (majority China-based) than either gender. We also see that as the years progress, we are inviting the same men and women less often.
This decrease in invites per person is also seen in invited reviewer country-region. The chart below shows editors are inviting people in the U.S. more often than those in all other country-regions. However, we also find that we are inviting people by region less often indicating that in 2020, we expanded our reviewer pool.
Reviewing during the COVID-19 pandemic
Across the scientific community, there was considerable concern about how the pandemic would affect people’s capacity to submit papers and their ability to review. In May of 2020, we looked at the initial impact of the pandemic on our submissions; we now have enough data from the past year to understand how this has impacted our community.
Submissions increased overall and across all demographic groups (based only on the corresponding author) with proportions varying only slightly with no statistically significant difference between in-pandemic and pre-pandemic periods. Editors and associate editors invited slightly more reviewers in their 20s and 30s and slightly fewer in their 40s and 50s compared to previous years. Reviewer agree rates dropped a few points among women and increased a few points among men in 2020, as shown in the chart below.
Overall, paper submissions and reviewer activities increased (more submissions, more peer reviews) likely because stay-at-home orders were conducive to these types of desk-based activities (see survey conducted by the American Geosciences Institute on work habits in 2020). Though many survey-based studies showed that women (and men) with children reported less productivity and decreased job satisfaction, we found that these effects of the pandemic weren’t reflected in AGU submission and reviewing rates.
Increasing and Diversifying our Reviewer Pool
The pandemic illuminated the need to expand the reviewer pool, and also made AGU and other scholarly publishers—and quite acutely society at large—consider ways to make processes and systems more inclusive and equitable. Peer review plays a seemingly small part, but the effects on advancing the scientific record could be massive. So, what are some ways we’re trying to increase reviewer representation in key demographic groups?
- We ask authors to consider suggesting reviewers who are in historically underrepresented groups in the Earth and space sciences, such as women, early career scientists, and racial minorities.
- We encourage editors to invite more author-suggested reviewers, not just for those papers but for other submissions. Though there is a worry that some authors suggest their “friends,” these names help to expand the reviewer pool when used for other submissions on a similar topic.
- We are deliberately increasing editor and associate editor appointments in target demographic groups. This chart shows that they do invite more reviewers from their region:
- We’ve enforced term limits and renewals to help our editorial boards to make room for new perspectives.
- We’ve created author and reviewer resources in other languages (e.g., webinars in Chinese and Spanish and author resources in Chinese and Japanese.
- We encourage editors to engage younger women for associate editor and reviewer roles when the pool of advanced career women is overworked.
The pandemic impelled us to be even more thoughtful about who contributes to decisions about what gets published. Reviewers are the backbone to that endeavor.
Our community takes very seriously our responsibility to publish accurate science that takes into account multiple perspectives and opinions, especially as human-caused climate change threatens to upend our way of life.
The pandemic impelled us to be even more thoughtful about who contributes to decisions about what gets published, which ultimately contributes to the solutions to address our society’s most pressing challenges. Reviewers are the backbone to that endeavor.
—Paige Wooden ([email protected], 
0000-0001-5104-8440), Senior Program Manager, Publications Statistics, American Geophysical Union
Citation:
Wooden, P. (2021), Reviewing reviewers, Eos, 102, https://doi.org/10.1029/2021EO163498. Published on 23 September 2021.
Text © 2021. The authors. CC BY-NC-ND 3.0
Except where otherwise noted, images are subject to copyright. Any reuse without express permission from the copyright owner is prohibited.