DoubleTake: Contrastive Reasoning for Faithful Decision-Making in Medical Imaging Daivik Patel, Shrenik Patel Abstract Accurate decision-making in medical imaging requires reasoning over subtle visual differences between confusable conditions, yet most existing approaches rely on nearest-neighbor retrieval that returns redundant evidence and reinforces a single hypothesis. We introduce a contrastive, document-aware reference selection framework that constructs compact evidence sets optimized for discrimination rather than similarity by explicitly balancing visual relevance, embedding diversity, and source-level provenance using ROCO embeddings and metadata. While ROCO provides large-scale image–caption pairs, it does not specify how references should be selected for contrastive reasoning, and naive retrieval frequently yields near-duplicate figures from the same document. To address this gap, we release a reproducible reference selection protocol and curated reference bank that enable a systematic study of contrastive retrieval in medical image reasoning. Building on these contrastive evidence sets, we propose Counterfactual-Contrastive Inference (CCI) , a confidence-aware reasoning framework that performs structured pairwise visual comparisons and aggregates evidence using margin-based decision rules with faithful abstention. On the MediConfusion benchmark, our approach achieves state-of-the-art performance , improving set-level accuracy by more than 15% relative to prior methods while substantially reducing confusion and maintaining high conditional accuracy. † † affiliationnotice: Equal contribution 1 Introduction Medical image decision-making is often framed as a recognition problem, yet many of the most consequential errors arise after recognition has already succeeded ( Topol 2019 ) . In clinical practice, failures frequently occur not because a finding is missed, but because two plausible diagnoses are confused. These errors emerge in regimes where images share global structure, modality, and anatomical context, and differ only in subtle, localized visual cues. In such settings, the core challenge is not identifying what is present, but determining which of several visually similar hypotheses is better supported by the evidence. Benchmarks such as MediConfusion are designed to isolate this challenge ( Sepehri et al. 2024 ) . Rather than evaluating images independently, MediConfusion organizes data into pairs of visually confusable images that require different answers. This structure exposes a class of failures that conventional metrics often obscure. A model may achieve strong image-level accuracy while repeatedly assigning the same prediction to both images in a pair. From a clinical perspective, such behavior is unacceptable: confusing look-alike conditions is precisely the error that decision support systems are meant to avoid. Set-level accuracy, which requires both images in a confusion pair to be classified correctly, therefore provides a more stringent and clinically aligned measure of discriminative competence. Recent advances in vision-language modeling suggest that performance in medical imaging is increasingly limited by reasoning under ambiguity rather than coarse recognition. In confusion-heavy regimes, systems often collapse toward a single answer due to shortcut correlations or dominant priors, yielding high confusion rates despite reasonable marginal accuracy. Retrieval augmentation has emerged as a common strategy for improving medical image reasoning, motivated by the intuition that relevant examples can provide useful context ( Lewis et al. 2020 ) . In practice, retrieval is almost universally implemented through nearest-neighbor search in an embedding space, selecting the most similar images to the query. While effective for recall-oriented tasks, this approach is poorly suited to discriminative decision making, where evidence diversity is as important as relevance ( Carbonell & Goldstein 1998 ) . Nearest-neighbor retrieval optimizes for similarity rather than contrast, and frequently returns near-duplicate images that share diagnosis, visual structure, and even document provenance. In medical corpora derived from the literature, this redundancy is exacerbated by multiple figures from the same article depicting the same condition. Rather than clarifying decision boundaries, such evidence reinforces a single hypothesis and suppresses alternatives, increasing decisiveness without increasing correctness. Addressing this misalignment requires rethinking retrieval not as a similarity operation, but as a process of evidence construction tailored to discrimination. The main contributions of this work are as follows. (1) We propose a contrastive, document-aware reference selection framework that constructs compact evidence sets optimized for discrimination rather than similarity by explicitly balancing visual relevance, embedding diversity, and source-level provenance. To support reproducibility and systematic evaluation, we release a triad selection protocol and curated reference bank derived from ROCO, enabling controlled study of contrastive retrieval in medical image reasoning. (2) Building on these contrastive evidence sets, we introduce Counterfactual-Contrastive Inference (CCI) , a confidence-aware reasoning framework that aggregates structured pairwise visual comparisons using margin-based decision rules with faithful abstention. We further introduce a pair-level adjudication mechanism to resolve ambiguity in confusion-pair settings without imposing artificial label constraints. Together, these components achieve state-of-the-art performance on MediConfusion , substantially improving set-level accuracy while reducing confusion rates. Figure 1 : Overview of the proposed framework. Given a query image, a contrastive triad of reference images is retrieved from an external medical image bank. Structured pairwise comparisons are aggregated by Counterfactual-Contrastive Inference (CCI) using confidence-aware voting, margin-based decisions, and adjudication for ambiguous cases. 2 Related Work 2.1 Medical Visual Question Answering Benchmarks Medical visual question answering (VQA) has been studied through a range of benchmarks, including ImageCLEF VQA-Med ( Hasan et al. 2018 ; Ben Abacha et al. 2019 ) , PathVQA ( He et al. 2020 ) , and knowledge-enhanced datasets such as SLAKE ( Liu et al. 2021 ) . 2.2 Retrieval for Medical Image Reasoning Retrieval augmentation is commonly used in medical image reasoning to provide contextual exemplars when decisions hinge on subtle visual cues. Most existing systems implement retrieval via similarity search in an embedding space, returning the top- k nearest neighbors of a query image, sometimes augmented with captions or metadata. While effective for recall-oriented tasks, this similarity-first objective is poorly aligned with confusion-style benchmarks such as MediConfusion, where the goal is to discriminate between visually similar conditions rather than retrieve more of the same. This misalignment is amplified in literature-derived corpora, where figures are often templated and multiple near-duplicate panels originate from the same document. As a result, naive top- k retrieval frequently returns redundant, provenance-collapsed evidence that reinforces a single hypothesis instead of exposing alternatives. Our work addresses this gap by framing retrieval as evidence construction for discrimination , with explicit controls for diversity and document provenance. 2.3 Contrastive Reasoning at Inference Time The term contrastive is typically used to describe representation learning objectives for medical vision–language alignment, for example via CLIP-style pretraining and medical variants such as MedCLIP ( Wang et al. 2022 ) . More recent unified medical CLIP-style models include UniMed-CLIP ( Khattak et al. 2024 ) . These approaches improve representation quality, including in zero-shot settings ( Tiu & et al. 2022 ; Jang et al. 2022 ) , but they do not by themselves specify how retrieved evidence should be selected for discriminative decision making. Our framing is orthogonal: we do not introduce a new contrastive training objective. Instead, we construct contrastive evidence sets at inference time , forcing the model to compare the query against carefully selected alternatives that probe decision boundaries. This perspective complements recent instruction-tuned and few-shot medical multimodal systems ( Zhang et al. 2023 ; Moor et al. 2023 ) as well as earlier vision–language baselines ( Li et al. 2019 ; Vaswani et al. 2017 ) . 2.4 Reliability via Abstention + Evidence Aggregation Selective prediction and abstention are well-established tools for improving reliability in high-stakes settings, allowing models to defer decisions when evidence is insufficient ( Geifman & El-Yaniv 2017 ; Angelopoulos & Bates 2021 ) . In medical imaging, this is particularly important because multiple visually plausible hypotheses often coexist. Confusion-pair benchmarks such as MediConfusion make this challenge explicit by evaluating whether models can distinguish look-alike conditions rather than merely achieve high marginal accuracy. In this setting, decisions arise from multiple structured visual comparisons whose confidence and consistency may vary and can be miscalibrated ( Guo et al. 2017 ) , and naive aggregation can collapse to identical predictions across a pair. We address these failure modes with confidence-aware aggregation and faithful abstention. 3 Method 3.1 System Overview We present a two-stage inference framework for discriminative medical image decision making. Given a query image x and a binary multiple-choice question with two candidate answer options \mathcal{Y}=\{A,B\} (where A and B denote the two answer strings provided by the benchmark for that question), the system outputs a prediction \hat{y}\in\{A,B,\bot\} , where \bot denotes abstention. The framework combines contrastive, document-aware reference selection , which constructs compact evidence sets optimized for discrimination, with Counterfactual-Contrastive Inference (CCI) , which aggregates structured visual comparisons into a reliable decision. For a given query image x , the reference selection component retrieves a fixed triad of reference images R(x)=\{r_{1},r_{2},r_{3}\} from a large external medical image bank. Each reference induces a counterfactual comparison that probes how the decision would change relative to a plausible alternative, producing a grounded vote with an associated confidence. CCI filters unreliable signals, aggregates evidence using a margin-based rule, and invokes adjudication only when evidence is ambiguous. In datasets organized into visually confusable pairs, a final pair-level adjudicator resolves ambiguous outcomes. Figure 1 provides an overview of the full pipeline. 3.2 Contrastive, Document-Aware Reference Selection Effective discrimination requires evidence that is not only relevant to the query, but also diverse and non-redundant. Standard nearest-neighbor retrieval optimizes for similarity and frequently returns near-duplicate images, often originating from the same document, which reinforces a single hypothesis and obscures subtle distinctions. We therefore design reference selection explicitly as evidence construction for discrimination . Reference bank and representations. We construct a reference bank derived from ROCO that provides a large, redundant universe of medical images with document provenance ( Pelka et al. 2018 ) . Formally, let \mathcal{B}=\{(r_{j},c_{j},d_{j},m_{j})\}_{j=1}^{N} denote the reference bank, where r_{j} is an image, c_{j} its caption, d_{j} a document identifier (e.g., PMC article ID), and m_{j} its imaging modality. All images are embedded using a frozen encoder \phi(\cdot) (CLIP ViT-B/32) ( Radford et al. 2021 ; Dosovitskiy et al. 2020 ) . For a query image x , cosine similarity is defined as s(x,r)=\frac{\langle\phi(x),\phi(r)\rangle}{\|\phi(x)\|\|\phi(r)\|}. Because ROCO contains many visually redundant figures, we apply near-duplicate suppression. A candidate r is removed if there exists a previously retained image r^{\prime} such that s(r,r^{\prime})>\tau_{\mathrm{dup}},\qquad\tau_{\mathrm{dup}}=0.99. This prevents trivial duplicates from dominating the retrieved set. Candidate pool and rank bands. Let \mathcal{C}(x) denote the candidate pool after modality gating and deduplication (defined below). We rank all r\in\mathcal{C}(x) in descending order of similarity s(x,r) and write \mathcal{C}_{[a,b]}(x) for the subset of candidates whose similarity rank lies in [a,b] (1-indexed, inclusive). When selecting r_{2} and r_{3} , we additionally exclude any already selected references and, whenever possible, enforce distinct document IDs. Triad retrieval policy. We retrieve exactly three references per query. This fixed budget encourages diversity and stabilizes downstream inference. The three references play complementary roles. The anchor reference provides a stable visual baseline and is selected as the most similar non-duplicate candidate, r_{1}=\arg\max_{r\in\mathcal{C}(x)}s(x,r), where \mathcal{C}(x) denotes the candidate pool after deduplication and gating. This reference typically aligns with the query in modality and anatomy. The hard negative induces discriminative contrast. From a mid-similarity band of candidates (ranks 20–200), we select the image that is maximally distinct from the anchor in embedding space, r_{2}=\arg\min_{r\in\mathcal{C}_{[20,200]}(x)}\left|\left\langle\tilde{\phi}(r),\tilde{\phi}(r_{1})\right\rangle\right|,\qquad\tilde{\phi}(r)=\frac{\phi(r)}{\|\phi(r)\|}. This ensures that the reference remains relevant to the query while forcing comparison along subtle, discriminative dimensions rather than shared structure. The boundary probe tests decision robustness against broader but still plausible alternatives. It is drawn from a wider similarity band (ranks 200–1000) and scored by combining relevance to the query, dissimilarity to the anchor, and lightweight semantic plausibility derived from metadata: \mathrm{score}(r;x,r_{1})=\big(\kappa(c_{r};x)+1\big)\,s(x,r)\,\big(1-s(r,r_{1})\big), where \kappa(c_{r};x)\in\mathbb{N}_{0} is the raw lexical overlap count between the caption c_{r} and the question text associated with x (after lowercasing and stopword removal), i.e., the number of shared tokens. In practice, overlap thresholds (e.g., 0, 1–2, and \geq 3 shared tokens) may be used to stage candidate selection, but the scoring signal itself is the overlap count. The boundary probe is selected as r_{3}=\arg\max_{r\in\mathcal{C}_{[200,1000]}(x)}\mathrm{score}(r;x,r_{1}). Together, the triad approximates a compact evidence set that maximizes relevance while penalizing redundancy under provenance and modality constraints. Document awareness and modality gating. To reduce redundancy from a single source, we prefer references drawn from different documents whenever feasible (i.e., enforcing d_{r_{i}}\neq d_{r_{j}} for i\neq j when possible). We also apply modality gating via keyword matching over the question text and answer options to estimate a modality \hat{m}(x) (e.g., “CT”, “MRI”, “X-ray”, “ultrasound”), and restrict candidates to those whose captions mention the same modality: \mathcal{C}(x)=\{r\in\mathcal{B}\mid c_{r}\ \text{contains keywords for }\hat{m}(x)\}. If no modality keyword is detected in the question/options text (or if caption keywords are missing), we skip modality filtering and fall back to the full deduplicated candidate set. This keeps comparisons focused on clinically meaningful distinctions while avoiding brittleness from metadata sparsity. Fallbacks. If enforcing distinct document IDs is infeasible within a rank band, we relax the document constraint while keeping modality gating. If a rank band becomes empty after filtering (e.g., due to deduplication), we expand to the nearest available ranks within the same modality until a valid reference is found. Use of captions and release. Captions and metadata are used solely for reference selection and are never shown to the vision-language model during inference, eliminating text leakage. We release the triad selection protocol and curated ROCO reference bank to enable reproducible study of contrastive retrieval in medical imaging. 3.3 Counterfactual-Contrastive Inference Given a contrastive triad R(x) , CCI aggregates pairwise visual comparisons into a reliable prediction. Each comparison acts as a counterfactual probe, asking how the decision would change relative to a specific alternative. Rather than treating all comparisons equally, CCI explicitly models confidence, decisiveness, and ambiguity. Structured pairwise visual comparison. For each reference r_{i} , a vision-language model f_{\theta} compares the query x against r_{i} in a structured manner, producing o_{i}=(v_{i},\alpha_{i},e_{i}), where v_{i}\in\{A,B,\bot\} is the predicted answer, \alpha_{i}\in[0,100] is a self-reported confidence score elicited from f_{\theta} using a fixed prompt template (shared across all methods) that requests an integer confidence in [0,100] for the selected option, and e_{i} is a structured evidence record containing the query’s modality and anatomy, concrete visual findings, and explicit differences relative to the reference. We treat \alpha_{i} as a relative weighting signal for aggregation (not as a calibrated probability). Voting over available comparisons. Some comparisons abstain (or fail to yield a valid vote). We therefore define \mathcal{I}(x) as the set of comparison indices that produce a non-abstaining vote, \mathcal{I}(x)=\{i\mid v_{i}\in\{A,B\},\alpha_{i}\geq p\} and discard all other comparisons. Confidence-weighted aggregation and margin rule. Let W_{A}=\sum_{i\in\mathcal{I}(x)}\alpha_{i}\,\mathds{1}[v_{i}=A],\qquad W_{B}=\sum_{i\in\mathcal{I}(x)}\alpha_{i}\,\mathds{1}[v_{i}=B] denote confidence-weighted evidence for each answer. Define total evidence mass W=W_{A}+W_{B} and decisiveness margin M=|W_{A}-W_{B}| . CCI returns \hat{y}(x)=\begin{cases}\bot,&W= 60, output Answer : - Step 4 – Confidence and Key Evidence Provide your confidence level (0-100) and the single most important piece of visual evidence that supports your answer . Key evidence must name a specific visual feature in the QUERY image , not a conclusion . Output format ( follow exactly ): Modality and Anatomy : < one line > QUERY Findings : < finding 1> < finding 2> < finding 3> Differences vs REFERENCE : < difference 1> < difference 2> < difference 3> ( optional ) < difference 4> ( optional ) Answer : < A or B or -> Confidence : <0-100> Key evidence : < specific visual feature in QUERY , not a conclusion > Rules : Base the final answer only on the QUERY image . Use the REFERENCE image only to identify discriminative visual features . Do not reference captions , metadata , or prior cases . Do not add explanations beyond the required fields . Abstain (-) if evidence is insufficient or confidence is below 60. B.2 Aggregation Prompt ( aggregate_decision ) ⬇ You have made { n } pairwise comparisons for a medical image question . Question : { q } A : { a } B : { b } Your pairwise comparison votes : { votes } Based on the consistency of your comparisons , provide your final answer . If the votes are inconsistent and average confidence is low , output Final Answer : - Output format : Final Answer : < A or B or -> B.3 Pair-level Adjudicator Prompt ( pair_adjudicator ) ⬇ You are analyzing TWO medical images for the same clinical question . Question : { q } A : { a } B : { b } Image 1: { meta1 } Image 2: { meta2 } Both images were independently predicted as : { pred } Your task : Evaluate each image primarily on its own visual evidence to determine the most appropriate answer . You may consider visual differences between the two images only to assess whether the same decision should reasonably apply to both images . IMPORTANT GUIDELINES : - Each image should be judged on its own merits . - There is NO requirement that the two images have different answers . - There is NO requirement that the two images have the same answer . - Use cross - image comparison only if it reveals a clear , clinically meaningful distinction . - If evidence is ambiguous or insufficient for either image , abstain (-). - Abstention is a valid and preferred outcome when confidence is low . - Base all decisions strictly on visual evidence . Provide : 1. Brief visual assessment of each image ( key findings only ). 2. Whether any observed differences justify different answers . 3. A final answer for each image . Output format ( follow exactly ): Visual assessment Image 1: < key visual features > Visual assessment Image 2: < key visual features > Do observed differences justify different answers ? < Yes or No > Final Answer Image1 : < A or B or -> Final Answer Image2 : < A or B or -> Rules : - Do not rely on captions , metadata , or prior cases . - Do not force disagreement between images . - Do not guess ; abstain (-) if uncertain .