Gender and Discipline Shape Length, Content and Tone of Grant Peer Review Reports
Stefan Müller, Gabriel Okasa, Michaela Strinzel, Anne Jorstad, Katrin Milzow, Matthias Egger · 2025
Read Gender and Discipline Shape Length, Content and Tone of Grant Peer Review Reports by Stefan Müller, Gabriel Okasa, Michaela Strinzel, Anne Jorstad, Katrin Milzow, Matthias Egger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00103.txt.
Introduction
Stefan Müller and five co-authors published a 2025 study of how gender and field shape the peer review reports written for grant proposals. They analyzed 39,280 reports sent to the Swiss National Science Foundation from 2016 to 2023, covering 11,385 proposals in 21 fields. Machine learning sorted more than 1.3 million sentences by what they judged and how warmly they judged it.
Reports in social sciences and humanities ran longer and more critical, with less weight on the applicant's record. Reports in the technical fields were shorter and warmer, with more weight on track record. Women reviewers wrote longer reports that tracked the stated criteria more closely and sounded more positive. Women applicants also drew slightly warmer reports. The authors warn that these gaps matter for fairness in funding.
Catalog record
| CREATOR | Stefan Müller, Gabriel Okasa, Michaela Strinzel, Anne Jorstad, Katrin Milzow, Matthias Egger |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-00103
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Gender and Discipline Shape Length, Content and Tone of Grant Peer Review Reports Stefan Müller, Gabriel Okasa, Michaela Strinzel, Anne Jorstad, Katrin Milzow, Matthias Egger Abstract Peer review by experts is central to the evaluation of grant proposals, but little is known about how gender and disciplinary differences shape the content and tone of grant peer review reports. We analyzed 39,280 review reports submitted to the Swiss National Science Foundation between 2016 and 2023, covering 11,385 proposals for project funding across 21 disciplines from the Social Sciences and Humanities (SSH), Life Sciences (LS), and Mathematics, Informatics, Natural Sciences, and Technology (MINT). Using supervised machine learning, we classified over 1.3 million sentences by evaluation criteria and sentiment. Reviews in SSH were significantly longer and more critical, with less focus on the applicant’s track record, while those in MINT were more concise and positive, with a higher focus on the track record, as compared to those in LS. Compared to male reviewers, female reviewers write longer reviews that more closely align with the evaluation criteria and express more positive sentiments. Female applicants tend to receive reviews with slightly more positive sentiment than male applicants. Gender and disciplinary culture influence how grant proposals are reviewed–shaping the tone, length, and focus of peer review reports. These differences have important implications for fairness and consistency in research funding. \NewTableCommand \tinytableDefineColor [3] Introduction Peer review is the cornerstone of the scientific review process, aiming to ensure fair allocation of funding, based on merit, scientific quality and potential. Typically, experienced external reviewers provide structured assessments of proposals that inform the recommendations made by evaluation panels. The process of grant peer review has, however, been criticized for many years [ 1 ] . Reviewers often remain anonymous, applicants may not see full review reports, and the weight given to the external reports in final funding decisions is generally unclear [ 2 , 3 , 4 ] . Compared to journal article review, researchers tend to have less trust in the grant review process [ 5 ] . Moreover, peer review of both articles and grant proposals may be influenced by the gender and other characteristics of applicants and reviewers, potentially introducing bias [ 6 , 7 , 8 ] . Approaches to peer review may vary across research areas or disciplines. Disciplinary differences and cultures shape how proposals are reviewed, with different fields emphasizing the applicants’ track record, methodological rigor, or originality. Finally, the well-documented low inter-rater reliability in peer review underscores the difficulty of achieving consistent evaluations across experts [ 9 , 10 ] . Although billions of research dollars are distributed competitively using peer review, the process remains under-researched. While journal peer review and editorial practices have been the focus of a growing body of empirical research [ 11 , 12 , 13 , 14 , 15 ] , the evidence on grant peer review is scarce. Simon Wessely has argued that studying grant peer review may be more important than studying publication practices: whereas published articles reflect completed research, rejected grant proposals often represent studies that will never be done [ 16 ] . Funding success also has significant impacts on academic careers. An analysis based on funding outcomes at the Netherlands Organization of Scientific Research shows that applicants just above the funding threshold accumulated more than twice as much funding in subsequent years as applicants just below the threshold [ 17 ] . Decisions based on grant peer review can have lasting effects on research and careers. Studying the process is essential to ensure it is as fair and as transparent as possible. The emergence of Large Language Models (LLMs) [ 18 ] has allowed increasingly sophisticated analyses of peer review reports [ 19 ] . Beyond thematic analysis, LLMs can assess the tone or sentiment of reviews by quantifying whether the language used is supportive, neutral, or critical [ 20 ] . Sentiment analysis can help funding agencies detect implicit positivity or negativity that might not be evident from review scores. We defined key content categories and trained four annotators to assign categories to 3,000 randomly sampled sentences from peer review reports submitted to the Swiss National Science Foundation’s (SNSF) project funding scheme [ 21 ] . Using these annotations, we fine-tuned six transformer machine learning models. After demonstrating that classifiers performed reliably [ 21 ] , we applied the models to over 1.3 million sentences from almost 40,000 peer review reports. This study examines how the length, content, and sentiment of peer review reports vary across fields and disciplines, as well as by the gender of applicants and reviewers. It is among the first to analyze the structure and textual content of grant peer review reports, extending research beyond numerical scores and ratings. In doing so, it offers novel insights for reviewers, applicants, and funding agencies alike. Results We present the results in several steps. We start with descriptive statistics on the characteristics of reviews, reviewers and applicants. Next, we focus on the differences in the length of reviews across disciplines, the gender of reviewers, and the gender of the corresponding applicants. Afterwards, using descriptive and mixed-effects regression analyses, we assess how content related to the SNSF evaluation criteria, and the sentiment of peer review reports differ across disciplines and gender. Gender is self-reported, and during the period of analysis, the SNSF recorded only two categories for gender: female or male. Moreover, we report on the role of the review score, and on robustness analyses. LS MINT SSH Total Reviewers 15,006 (100%) 15,441 (100%) 8,833 (100%) 39,280 (100%) Gender of Reviewer Male 11,525 (76.8%) 13,555 (87.8%) 5,433 (61.5%) 30,513 (77.7%) Female 3,481 (23.2%) 1,886 (12.2%) 3,400 (38.5%) 8,767 (22.3%) Country of Affiliation United States of America 3,783 (25.2%) 3,371 (21.8%) 2,051 (23.2%) 9,205 (23.4%) Germany 1,187 (7.9%) 1,443 (9.3%) 805 (9.1%) 3,435 (8.7%) Great Britain and N. Ireland 1,072 (7.1%) 1,178 (7.6%) 1,072 (12.1%) 3,322 (8.5%) France 723 (4.8%) 866 (5.6%) 247 (2.8%) 1,836 (4.7%) Italy 529 (3.5%) 929 (6.0%) 282 (3.2%) 1,740 (4.4%) Canada 513 (3.4%) 583 (3.8%) 341 (3.9%) 1,437 (3.7%) Netherlands 583 (3.9%) 364 (2.4%) 419 (4.7%) 1,366 (3.5%) Australia 527 (3.5%) 387 (2.5%) 278 (3.1%) 1,192 (3.0%) Other 2,730 (18.2%) 3,722 (24.1%) 1,344 (15.2%) 7,796 (19.8%) Unknown 3,359 (22.4%) 2,598 (16.8%) 1,994 (22.6%) 7,951 (20.2%) Applicants 4,264 (100%) 4,077 (100%) 3,044 (100%) 11,385 (100%) Gender Male 3,074 (72.1%) 3,401 (83.4%) 1,916 (62.9%) 8,391 (73.7%) Female 1,190 (27.9%) 676 (16.6%) 1,128 (37.1%) 2,994 (26.3%) Age Mean (SD) 48.2 (7.5) 46.9 (8.2) 48.2 (7.9) 47.7 (7.9) Affiliation Cantonal University 3,383 (79.3%) 1,507 (37.0%) 2,010 (66.0%) 6,900 (60.6%) ETH Domain 544 (12.8%) 2,150 (52.7%) 183 (6.0%) 2,877 (25.3%) UAS/UTE 66 (1.5%) 221 (5.4%) 547 (18.0%) 834 (7.3%) Other 167 (3.9%) 160 (3.9%) 279 (9.2%) 606 (5.3%) Hospital 86 (2.0%) 2 (0.0%) 3 (0.1%) 91 (0.8%) Unknown 18 (0.4%) 37 (0.9%) 22 (0.7%) 77 (0.7%) Academic Rank Full Professor 1,098 (25.8%) 1,173 (28.8%) 1,254 (41.2%) 3,525 (31.0%) Associate Professor 858 (20.1%) 534 (13.1%) 468 (15.4%) 1,860 (16.3%) Assistant Professor 416 (9.8%) 446 (10.9%) 312 (10.2%) 1,174 (10.3%) UAS/UTE Professor 27 (0.6%) 84 (2.1%) 328 (10.8%) 439 (3.9%) Senior Postdoc/Group Leader 1,550 (36.4%) 1,540 (37.8%) 599 (19.7%) 3,689 (32.4%) Other Professorship 290 (6.8%)