Counterfactual Cultural Cues Reduce Medical QA Accuracy in LLMs: Identifier vs Context Effects

Amirhossein Haji Mohammad Rezaei, Zahra Shakeri · 2026

Read Counterfactual Cultural Cues Reduce Medical QA Accuracy in LLMs: Identifier vs Context Effects by Amirhossein Haji Mohammad Rezaei, Zahra Shakeri (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20102.txt.

Description

Amirhossein Haji Mohammad Rezaei and Zahra Shakeri test whether irrelevant cultural cues can derail medical AI. A medical language model should not change a correct diagnosis because a question mentions the patient's culture. Yet culture-related tokens might bias its reasoning. The authors build a counterfactual benchmark to measure the effect cleanly.

They expand a set of medical exam questions into many variants. Variants insert cultural identifiers, contextual cues, or both, for three patient groups. Comparing accuracy across variants isolates whether identifiers or context do the damage. Equitable healthcare needs models robust to such non-decisive information. The benchmark gives developers a concrete fairness test for medical question answering.

Catalog record

CREATORAmirhossein Haji Mohammad Rezaei, Zahra Shakeri
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-20102

Browse the full library · About the Tower of Babel