Variability among large language models in assessing CONSORT compliance of published randomized clinical trials
Daniel Y Tsybulnik, Justin J Gillette, Thomas F Heston · 2026
Read Variability among large language models in assessing CONSORT compliance of published randomized clinical trials by Daniel Y Tsybulnik, Justin J Gillette, Thomas F Heston (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/plos-pone-0358873.txt.
Description
Daniel Y Tsybulnik, Justin J Gillette, and Thomas F Heston ask whether language models agree with each other when grading clinical trial reports, in PLOS ONE in 2026. Journals expect trials to follow the CONSORT reporting checklist, and automated checking would help editors, if the checkers were consistent. The team had three leading models independently score twenty randomized trials from immunology journals against thirty-seven checklist points.
The models diverged sharply. One awarded an average compliance score of about eighty-one percent, a second about sixty-four, and the third about fifty-five. Against a strict ninety percent quality bar, the most generous model passed five papers while the other two passed none at all. Agreement between models was too weak for interchangeable use. The authors warn that swapping one model for another silently changes the verdict, so any editorial workflow needs a fixed, validated checker.
Catalog record
| CREATOR | Daniel Y Tsybulnik, Justin J Gillette, Thomas F Heston |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30017 |
| LICENSE | CC_BY |
| RIGHTS | CC BY |
| SOURCE | PLOS |