The Scaffold Effect: How Prompt Framing Drives Apparent Multimodal Gains in Clinical VLM Evaluation

Doan Nam Long Vu, Simone Balloccu · 2026

Read The Scaffold Effect: How Prompt Framing Drives Apparent Multimodal Gains in Clinical VLM Evaluation by Doan Nam Long Vu, Simone Balloccu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-28387.txt.

Introduction

Vu and Balloccu show that merely mentioning MRI scans in a prompt can swing clinical vision-language models, before any image is shown. They test 12 open models on two cohorts for mood disorders and cognitive decline. Smaller models gained up to 0.66 F1 once neuroimaging context was added, nearly matching far bigger ones. The calibration gains also arrived with the text mention alone. An expert review found low faithfulness throughout, with invented clinical details. Metric gains, they warn, are not proof of real image use.

Catalog record

CREATORDoan Nam Long Vu, Simone Balloccu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81406
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.28387

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Scaffold Effect: How Prompt Framing Drives Apparent Multimodal Gains in Clinical VLM Evaluation Doan Nam Long Vu, Simone Balloccu Abstract Trustworthy clinical AI requires that performance gains reflect genuine evidence integration rather than surface-level artifacts. We evaluate 12 open-weight vision-language models (VLMs) on binary classification across two clinical neuroimaging cohorts, FOR2107 (affective disorders) and OASIS-3 (cognitive decline). Both datasets come with structural MRI data that carries no reliable individual-level diagnostic signal. Under these conditions, smaller VLMs exhibit gains of up to 58% F1 upon introduction of neuroimaging context, with distilled models becoming competitive with counterparts an order of magnitude larger. A contrastive confidence analysis reveals that merely mentioning MRI availability in the task prompt accounts for 70-80% of this shift, independent of whether imaging data is present, a domain-specific instance of modality collapse we term the scaffold effect . Expert evaluation reveals fabrication of neuroimaging-grounded justifications across all conditions, and preference alignment, while eliminating MRI-referencing behavior, collapses both conditions toward random baseline. Our findings demonstrate that surface evaluations are inadequate indicators of multimodal reasoning, with direct implications for the deployment of VLMs in clinical settings 1 1 1 https://github.com/long21wt/scaffold-effect . 1 Introduction The application of vision-language models to clinical decision-making has attracted growing interest, with recent work exploring their potential for diagnostic classification from multimodal patient data 25 ; 31 ; 21 . A natural expectation is that providing richer inputs should improve model performance, but only insofar as the additional modalities supply relevant diagnostic evidence. While spurious correlations and hallucination in VLMs have received growing attention 43 ; 9 , to our knowledge no prior work has directly examined how models respond when a clinically related but diagnostically uninformative modality is introduced alongside structured tabular data. This distinction matters: a model changing its behaviour upon receiving an uninformative modality is not integrating evidence but responding to context, a failure mode that standard multimodal benchmarks are not designed to detect. We investigate this using two clinical cohorts where prior work has established that neuroimaging markers carry no reliable individual-level diagnostic signal 36 ; 35 , and where we further exclude trivially discriminative features in consultation with domain experts, ensuring models must reason rather than exploit shortcuts. Across 12 open-weight VLMs, smaller models exhibit substantial genuine, calibrated performance gains upon introduction of neuroimaging context, with Qwen2.5-VL-3B 38 improving up to 58% F1 score, with distilled models becoming competitive with counterparts an order of magnitude larger. Yet these gains carry no diagnostic grounding: the overwhelming majority of the shift is driven by the textual mention of MRI availability alone, persisting even when the MRI visualization is replaced by an unrelated out-of-domain image. We characterize this as a scaffold effect , a domain-specific instance of modality collapse 30 ; 26 arising from a convergence of prompt sensitivity 23 ; 12 and the priming effect 14 ; 39 that produces confident, evidence-styled justifications decoupled from actual diagnostic inputs. Critically, this vulnerability persists across model families and resists correction via preference alignment, suggesting it reflects a structural property of how these models process domain-specific multimodal prompts rather than an incidental artifact of any single architecture. Our contributions are: • A systematic evaluation of 12 open-weight VLMs on FOR2107 16 and OASIS-3 19 across five input conditions, revealing genuine calibrated gains of up to 58% F1 gains not attributable to benchmark memorization. • A contrastive confidence analysis showing \approx 70–80% of the performance shift stems from textual priming, characterized as a scaffold effect via phrase-level probe and false-modality ablation. • An expert case study with a clinical psychologist, showing that multimodal context measurably improves reasoning trace quality, yet faithfulness remains below acceptable thresholds, indicating that gains are partly driven by confidence rather than evidence-grounded inference. • Evidence that preference alignment via MPO which suppresses MRI-referencing behavior but induces symmetry-without-recovery , collapsing both conditions toward random baseline rather than selectively removing modality dependence. Our findings reveal the vulnerabilities in the reliability of VLMs for clinical diagnosis. We demonstrate that surface evaluations are fragile indicators of genuine multimodal reasoning and can exaggerate a model’s diagnostic ability. Figure 1: Overview of the proposed VLM pipeline. On FOR2107 and OASIS-3 we only change the label from MDD to Cognitive Decline and Control to Cognitive Normal. 2 Related Works We review related work on prompt sensitivity, the priming effect and modality collapse in vision-language models (VLMs). 2.1 Prompt Sensitivity in VLMs While prompt sensitivity has been extensively studied in large language models (LLMs) 11 ; 23 , it remains underexplored in VLMs. To the best of our knowledge, only 12 have explicitly demonstrated that VLMs suffer from severe prompt sensitivity, a vulnerability that leads to inconsistent classification performance and a concerning reliance on language priors. In healthcare, sensitivity to instruction phrasing can severely impact reliability: prompt variations cause F1 fluctuations of up to 0.25 in LLMs 2 and accuracy variations of up to 6% in VLMs on health benchmarks 12 . 2.2 Priming Effect in VLMs The priming effect is a cognitive phenomenon in which prior exposure to a stimulus influences subsequent judgments or behaviors, often without conscious awareness 24 . Rather than altering decision-making through explicit alternative descriptions, priming operates by activating related concepts, associations, or response tendencies that shape how later inputs are interpreted 17 . In the context of VLMs, priming in vision-language models refers to how prompts, both textual and visual cues, shape model outputs, mirroring the priming concept from cognitive science. 39 directly investigate this phenomenon in large-scale VLMs, demonstrating that model responses systematically shift in the direction intended by an accompanying image, suggesting that VLMs actively incorporate visual information into language processing rather than treating it as incidental. Recent research has further revealed both the power and limitations of priming in adapting VLMs to new tasks and domains 14 ; 17 . As these models increasingly incorporate multimodal inputs, such priming effects can be further intensified by interactions between textual and visual signals 6 ; 41 . For instance, 41 show that in fact-checking settings, VLMs may favor previously introduced textual context over salient visual evidence, and related behaviors have been observed in general VQA tasks 29 . 2.3 Modality Collapse in VLMs Modality collapse refers to the phenomenon whereby a unimodal model achieves comparable accuracy to a multimodal model on a vision-language task, revealing that one modality is not meaningfully utilized 13 ; 26 . A recent survey by 30 provides a systematic taxonomy of contributing factors, including dataset bias, model behavior, lack of fine-grained supervisory signal, and task setup, and reviews methods for quantifying modality contribution and cross-modal interaction. A consistent finding across this literature is that the text modality dominates, with visual input failing to influence predictions even when it is ostensibly required by the task 44 . Our work moves from detection to diagnosis. Prior modality collapse work establishes that text dominates, we instead identify which part of the textual input triggers the collapse in a clinical setting, and show that conventional suppression strategies cannot disentangle the trigger from the model’s broader inferential capacity. 3 Methodology Figure 1 illustrates our pipeline, integrating two primary data modalities: tabular clinical data (provided in CSV format) and structural brain MRI (sMRI) scans. It

Read the complete text