Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Images
Daniel Wolf, Heiko Hillenhagen, Billurvan Taskin, Alex Bäuerle, Meinrad Beer, Michael Götz, Timo Ropinski · 2025
Read Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Images by Daniel Wolf, Heiko Hillenhagen, Billurvan Taskin, Alex Bäuerle, Meinrad Beer, Michael Götz, Timo Ropinski (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00549.txt.
Description
Daniel Wolf and six co-authors showed in 2025 that leading vision-language models cannot tell left from right in medical scans. Judging where body parts sit relative to each other is basic to clinical reading. The team tested GPT-4o, Llama 3.2, Pixtral and JanusPro on such questions, and every model failed the task. Adding drawn markers, letters or colors onto the images helped only modestly.
Scores on medical images stayed far below those on everyday photos. The models seemed to answer from stored anatomy knowledge rather than from the picture in front of them. That habit produces confident, wrong conclusions. The authors released a benchmark dataset, MIRP, to measure this skill systematically. They frame relative position reading as a gate skill that clinical AI must pass before wider use.
Catalog record
| CREATOR | Daniel Wolf, Heiko Hillenhagen, Billurvan Taskin, Alex Bäuerle, Meinrad Beer, Michael Götz, Timo Ropinski |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-00549