Evaluating large language model performance in Risk of Bias assessments: A cross-sectional validation study

Siddharth Gandhi, Arveen Shokravi, Yashan Chelliahpillai, Michael Balas · 2026

Read Evaluating large language model performance in Risk of Bias assessments: A cross-sectional validation study by Siddharth Gandhi, Arveen Shokravi, Yashan Chelliahpillai, Michael Balas (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/plos-pone-0353155.txt.

Description

Gandhi and colleagues tested a leading AI chatbot on judging bias in clinical trials in a 2026 study. Fifty trials drawn from fifty reviews were rated by the original reviewers, by a masked human panel, and by the chatbot using structured prompts from the standard bias tool.

The chatbot called far more trials high risk than either human group did. Agreement with humans was modest at best. Overcalling bias can distort evidence summaries just as undercalling can. The authors' verdict is caution: AI bias ratings are not yet a stand-in for trained reviewers, though they might serve as a first pass with human checks.

Catalog record

CREATORSiddharth Gandhi, Arveen Shokravi, Yashan Chelliahpillai, Michael Balas
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38084
LICENSECC_BY
RIGHTSCC BY
SOURCEPLOS

Source: https://doi.org/10.1371/journal.pone.0353155

Browse the full library · About the Tower of Babel