Same Patient, Different Words, Different Diagnosis? Evaluating Semantic Stability in Clinical LLMs

Mahdi Alkaeed, Adnan Qayyum, Nabeel Abo Kashreef, Muhammad Bilal, Junaid Qadir · 2026

Read Same Patient, Different Words, Different Diagnosis? Evaluating Semantic Stability in Clinical LLMs by Mahdi Alkaeed, Adnan Qayyum, Nabeel Abo Kashreef, Muhammad Bilal, Junaid Qadir (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30646.txt.

Description

Alkaeed and colleagues test whether medical language models stay stable when the same patient case is rephrased. Small phrasing changes can flip answers, a danger in safety-critical care, yet vector likeness often misses negation, timing, or severity gaps. Their meaning check uses meaning-based logic to filter truly meaning-preserving rewrites, refined by model judging and checked by a medical expert. Across sixteen general and medical models, domain focus proves mixed: some focused models rank most robust, while strong general baselines stay strong.

Catalog record

CREATORMahdi Alkaeed, Adnan Qayyum, Nabeel Abo Kashreef, Muhammad Bilal, Junaid Qadir
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72685
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.30646

Browse the full library · About the Tower of Babel