Evaluating large language model performance in Risk of Bias assessments: A cross-sectional validation study
Siddharth Gandhi, Arveen Shokravi, Yashan Chelliahpillai, Michael Balas · 2026
Read Evaluating large language model performance in Risk of Bias assessments: A cross-sectional validation study by Siddharth Gandhi, Arveen Shokravi, Yashan Chelliahpillai, Michael Balas (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/plos-pone-0353155.txt.
Description
Gandhi and colleagues tested a leading AI chatbot on judging bias in clinical trials in a 2026 study. Fifty trials drawn from fifty reviews were rated by the original reviewers, by a masked human panel, and by the chatbot using structured prompts from the standard bias tool.
The chatbot called far more trials high risk than either human group did. Agreement with humans was modest at best. Overcalling bias can distort evidence summaries just as undercalling can. The authors' verdict is caution: AI bias ratings are not yet a stand-in for trained reviewers, though they might serve as a first pass with human checks.
Catalog record
| CREATOR | Siddharth Gandhi, Arveen Shokravi, Yashan Chelliahpillai, Michael Balas |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38084 |
| LICENSE | CC_BY |
| RIGHTS | CC BY |
| SOURCE | PLOS |