Counterfactual Cultural Cues Reduce Medical QA Accuracy in LLMs: Identifier vs Context Effects
Amirhossein Haji Mohammad Rezaei, Zahra Shakeri · 2026
Read Counterfactual Cultural Cues Reduce Medical QA Accuracy in LLMs: Identifier vs Context Effects by Amirhossein Haji Mohammad Rezaei, Zahra Shakeri (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20102.txt.
Description
Amirhossein Haji Mohammad Rezaei and Zahra Shakeri test whether irrelevant cultural cues can derail medical AI. A medical language model should not change a correct diagnosis because a question mentions the patient's culture. Yet culture-related tokens might bias its reasoning. The authors build a counterfactual benchmark to measure the effect cleanly.
They expand a set of medical exam questions into many variants. Variants insert cultural identifiers, contextual cues, or both, for three patient groups. Comparing accuracy across variants isolates whether identifiers or context do the damage. Equitable healthcare needs models robust to such non-decisive information. The benchmark gives developers a concrete fairness test for medical question answering.
Catalog record
| CREATOR | Amirhossein Haji Mohammad Rezaei, Zahra Shakeri |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-20102