Same Patient, Different Words, Different Diagnosis? Evaluating Semantic Stability in Clinical LLMs
Mahdi Alkaeed, Adnan Qayyum, Nabeel Abo Kashreef, Muhammad Bilal, Junaid Qadir · 2026
Read Same Patient, Different Words, Different Diagnosis? Evaluating Semantic Stability in Clinical LLMs by Mahdi Alkaeed, Adnan Qayyum, Nabeel Abo Kashreef, Muhammad Bilal, Junaid Qadir (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30646.txt.
Description
Alkaeed and colleagues test whether medical language models stay stable when the same patient case is rephrased. Small phrasing changes can flip answers, a danger in safety-critical care, yet vector likeness often misses negation, timing, or severity gaps. Their meaning check uses meaning-based logic to filter truly meaning-preserving rewrites, refined by model judging and checked by a medical expert. Across sixteen general and medical models, domain focus proves mixed: some focused models rank most robust, while strong general baselines stay strong.
Catalog record
| CREATOR | Mahdi Alkaeed, Adnan Qayyum, Nabeel Abo Kashreef, Muhammad Bilal, Junaid Qadir |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72685 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30646