When Chain-of-Thought Backfires: Evaluating Prompt Sensitivity in Medical Language Models

Binesh Sadanandan, Vahid Behzadan · 2026

Read When Chain-of-Thought Backfires: Evaluating Prompt Sensitivity in Medical Language Models by Binesh Sadanandan, Vahid Behzadan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-25960.txt.

Description

Sadanandan and Behzadan probe how prompt style affects a medical language model. They test MedGemma at 4B and 27B sizes on MedMCQA and PubMedQA. Chain-of-thought prompting cuts accuracy by 5.7 percent versus direct answers. Few-shot examples cut it by 11.9 percent and raise position bias. Shuffling the answer options flips most predictions. Cutting context from the front wrecks accuracy, while back cuts keep 97 percent of it. A simple cloze score beats all prompting styles, they report. The models know more than their text shows.

Catalog record

CREATORBinesh Sadanandan, Vahid Behzadan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37664
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.25960

Browse the full library · About the Tower of Babel