Two-stage prompting framework with predefined verification steps for evaluating diagnostic reasoning tasks on two datasets
Ming Shao, Haichun Zhang · 2025
Read Two-stage prompting framework with predefined verification steps for evaluating diagnostic reasoning tasks on two datasets by Ming Shao, Haichun Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/pmc-12738563.txt.
Description
Ming Shao and Haichun Zhang test a simple trick to make chatbots diagnose more carefully: make them double-check, in a 2025 study. Language models are used in medicine despite shaky diagnostic reasoning.
Their framework runs in three steps: first diagnosis, a preset verification pass, then a final call. It faced 589 exam-style cases and 300 hard journal cases, answered by two leading models, each sampled five times. Blinded doctors graded the output. Verification lifted accuracy by up to 5.2 points and cut uncertainty and inconsistency further. Wrong medical knowledge fell the most among error types, by 63 percent. The method also beat plain chain-of-thought prompting. More datasets are needed, the authors note.
Catalog record
| CREATOR | Ming Shao, Haichun Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | CC BY 4.0 |
| RIGHTS | CC BY |
| SOURCE | Europe PMC |