Visual concept ranking uncovers medical shortcuts used by large multimodal models

Joseph D. Janizek, Sonnet Xu, Junayd Lateef, Roxana Daneshjou · 2026

Read Visual concept ranking uncovers medical shortcuts used by large multimodal models by Joseph D. Janizek, Sonnet Xu, Junayd Lateef, Roxana Daneshjou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-05096.txt.

Description

Medical AI can pick up shortcuts, leaning on image features that correlate with disease instead of the disease itself. Joseph D Janizek and 3 co-authors built an auditing method for that failure mode in this 2026 computer vision paper. Their test bed is large multimodal models asked to classify malignant skin lesions from clinical photographs.

The authors first show the models perform unevenly across demographic subgroups when given example images in the prompt. Their method, Visual Concept Ranking, then ranks which visual concepts the model depends on, generating hypotheses about its feature use. Each hypothesis can be checked by manually intervening on the images. Extra experiments on chest radiographs and natural images round out the study. The point is practical: hospitals need audits that name the shortcut before a model is trusted with patients.

Catalog record

CREATORJoseph D. Janizek, Sonnet Xu, Junayd Lateef, Roxana Daneshjou
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-05096

Browse the full library · About the Tower of Babel