Jagged AI in Scientific Peer Review: Evidence from POMP Data Analysis
Jin Wook Lee, William Szegda, Zhisheng Song, Edward L. Ionides · 2026
Read Jagged AI in Scientific Peer Review: Evidence from POMP Data Analysis by Jin Wook Lee, William Szegda, Zhisheng Song, Edward L. Ionides (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-07855.txt.
Description
Jin Wook Lee and three co-authors measure how well AI review agents handle scientific peer review of statistical data analyses. They also probe where their ability is jagged: strong in one place, weak in another. The test bed is seventy-two student projects fitting partially observed Markov process models, drawn from a University of Michigan graduate course, with reports, code, and human peer reviews openly available.
Four AI agents, all built on Claude Code with different instruction files, were compared with the human reviewers. The agents caught plausible technical errors and invalid inference methods that humans missed, but were weaker on interpretive errors, narrative coherence, and domain-informed critique. Changing the instructions shifted which weaknesses the agents emphasized without removing the jagged pattern.
Catalog record
| CREATOR | Jin Wook Lee, William Szegda, Zhisheng Song, Edward L. Ionides |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.07855