BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptation

Huanyang TongWenzhou University [email protected], Kai LiuWenzhou University [email protected], Fangjun KuangWenzhou Business College [email protected], Huiling ChenWenzhou University [email protected] · 2026

Read BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptation by Huanyang TongWenzhou University [email protected], Kai LiuWenzhou University [email protected], Fangjun KuangWenzhou Business College [email protected], Huiling ChenWenzhou University [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15736.txt.

Introduction

Huanyang Tong and three colleagues present BiomedAP, a dual-anchor frame for medical vision-language adaptation. Few-shot diagnosis is strong yet brittle: prompt shifts can break it. Past frames tune visual and textual prompts apart, leaning on golden prompts. In clinics, noisy descriptions make that split unstable. BiomedAP fuses modalities layer by layer with gated fusion, damping stray cues. A dual-anchor rule pulls prompts toward centers from expert templates and visual prototypes. On 11 benchmarks it tops baselines with competitive accuracy and far stronger prompt-shift robustness.

Catalog record

CREATORHuanyang TongWenzhou University [email protected], Kai LiuWenzhou University [email protected], Fangjun KuangWenzhou Business College [email protected], Huiling ChenWenzhou University [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28651
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.15736

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptation Huanyang TongWenzhou University [email protected], Kai LiuWenzhou University [email protected], Fangjun KuangWenzhou Business College [email protected], Huiling ChenWenzhou University [email protected] Abstract Biomedical Vision–Language Models (VLMs) have shown remarkable promise in few-shot medical diagnosis but face a critical bottleneck: fragility to prompt variations .Existing adaptation frameworks typically optimize visual and textual prompts as independent streams, relying on ideal “Golden Prompts”. In clinical reality, where descriptions are often noisy and heterogeneous, this modality isolation leads to unstable cross-modal alignment. To address this, we propose BiomedAP, a vision-informed dual-anchor framework with gated cross-modal fusion.BiomedAP enforces synergistic alignment through two mechanisms: (1) Gated Cross-Modal Fusion, which enables layer-wise interaction between modalities, acting as a dynamic noise regulator to suppress irrelevant textual cues; and (2) a Dual-Anchor Constraint that regularizes learnable prompts toward stable semantic centroids derived from both expert templates (High Anchors) and few-shot visual prototypes (Low Anchors). Extensive experiments across 11 benchmarks demonstrate that BiomedAP consistently surpasses baselines, achieving competitive few-shot accuracy and markedly enhanced robustness under prompt perturbations. Our code is available at: https://github.com/tongdiedie/BiomedAP . Keywords: Vision-Language Models; Prompt Learning; Parameter-Efficient Fine-Tuning; Few-shot Learning † † footnotetext: * First author †Corresponding author Figure 1 : Comparison of adaptation strategies. (a) No Adaptation : Direct inference without task-specific alignment. (b) Single-Anchor Adaptation : Relies solely on elaborate expert descriptions (High Anchor), leading to overfitting and fragility under prompt variations. (c) Dual-Anchor Adaptation (Ours) : Integrates both expert knowledge (High Anchor) and vision-intrinsic semantics derived from visual prototypes (Low Anchor), ensuring semantic stability across diverse clinical inputs. 1 Introduction Vision–language integration has emerged as a paradigm shift in medical AI 23 ; 29 ; 33 . Foundation VLMs, such as CLIP 27 and BiomedCLIP 34 , align images with text to enable flexible recognition across heterogeneous diseases. Two mainstream adaptation lines are commonly used: context-token prompt learning (CoOp/CoCoOp) 35 ; 36 and prompt-/adapter-style PEFT (e.g., VPT, Tip-Adapter, MaPLe) 10 ; 12 ; 32 , with biomedical variants such as Biomed-DPT 25 . These methods are attractive because they retain most pretrained knowledge while requiring only a small number of trainable parameters, making them especially suitable for data-limited medical settings. However, adapting these models to specific clinical tasks remains challenging. Full fine-tuning is computationally prohibitive and prone to overfitting 5 ; 6 , and standard PEFT often falls short of the reliability required for medical diagnosis 21 ; 22 . In real-world clinical settings, this challenge is further compounded by two recurring limitations. (1) Modality isolation limits semantic grounding. Many pipelines rely on late-stage matching, coupling visual and textual representations only at the final similarity computation 25 . This lack of intermediate interaction can amplify background confounders or miss subtle lesion patterns (Figure 1 (a–b)). (2) Prompt fragility from single-anchor supervision. Existing methods typically treat expert-crafted “Golden Prompts” as the sole optimization target. Optimizing exclusively against a High Anchor entangles semantics with surface forms, leading to brittle performance when clinical narratives become concise, noisy, or deviate from expert phrasing. To overcome these challenges, we propose BiomedAP to learn invariant diagnostic semantics from heterogeneous text. As illustrated in Figure 1 (c), BiomedAP introduces Gated Cross-Modal Fusion to deepen vision–text interaction during encoding. A lightweight gate modulates textual injection based on consistency with visual evidence. We further impose a Dual-Anchor Constraint with two centroids: a High Anchor (Expert) for diagnostic specificity and a Low Anchor (Core) capturing vision-informed intrinsic semantics . The Low Anchor stabilizes noisy prompts, while the High Anchor preserves fine-grained clinical details. In summary, our main contributions are: • We identify modality isolation and single-anchor fragility as primary obstacles to robust biomedical PEFT. • We introduce BiomedAP with Gated Cross-Modal Fusion to enable layer-wise interaction and suppress prompt-induced noise. • We propose a Dual-Anchor Constraint that regularizes prompts toward both expert and vision-informed semantics, improving stability. • Experiments on 11 benchmarks demonstrate consistent gains in few-shot learning and base-to-novel generalization, with superior robustness to prompt perturbations. 2 Related Work Parameter-Efficient Adaptation of Vision-Language Models. Parameter-efficient fine-tuning (PEFT) adapts large vision-language models (VLMs) with only a small number of trainable parameters 3 ; 6 ; 27 ; 34 . Textual prompt learning methods, such as CoOp and CoCoOp, optimize learnable context tokens in the language branch 36 ; 35 , whereas visual prompt tuning methods inject trainable tokens into the image encoder 10 ; 31 ; 37 . Recent work has further explored richer multimodal prompt parameterizations and improved generalization objectives for efficient VLM adaptation 12 ; 11 ; 13 ; 20 . However, many PEFT methods still adapt visual and textual streams in a largely decoupled manner, with cross-modal interaction occurring only at the final matching stage 27 . Recent work has begun to explore more tightly coupled visual-textual prompt optimization, suggesting that stronger mutual interaction can improve cross-modal alignment 24 . This limitation is particularly restrictive in biomedical recognition, where reliable prediction often depends on tight alignment between localized visual evidence and clinical semantics 1 ; 7 . Biomedical Prompt Learning and Medical VLM Adaptation. Recent biomedical adaptation methods extend prompt learning to medical tasks using domain-specific corpora, expert templates, or LLM-generated descriptions, as broadly summarized in recent surveys of medical vision-language modeling 18 . Recent work has also begun to explicitly study few-shot adaptation as a dedicated evaluation setting for medical vision-language models 28 . Methods such as BiomedCoOp and Biomed-DPT show that biomedical prompt tuning substantially improves few-shot transfer over generic prompting baselines 14 ; 25 . Related biomedical VLMs, including MedCLIP, Med-Flamingo, and BiomedCLIP, further demonstrate the value of domain-aligned pre-training for medical vision-language reasoning 29 ; 23 ; 34 . Nevertheless, most existing biomedical prompt learners still rely on late interaction and optimize prompts primarily in the text space, making them vulnerable to prompt fragility when clinical descriptions are noisy, concise, or phrased differently from training templates. This concern is consistent with recent studies on robustness in medical vision-language models 9 . Cross-Modal Interaction for Robust Grounding. A complementary line of work improves vision-language alignment through deeper cross-modal interaction. Representative models such as BLIP and BLIP-2 employ cross-attention and query-based fusion to strengthen multimodal grounding beyond late-stage matching 16 ; 15 , and recent parameter-efficient multimodal learning methods similarly emphasize adaptive feature exchange across modalities 17 . In contrast to these approaches, BiomedAP is designed for few-shot biomedical adaptation under heterogeneous clinical text. It combines intermediate-layer gated cross-modal fusion with dual-anchor regularization, enabling prompt learning to be both visually grounded and semantically stabilized. 3 Method 3.1 Overview We introduce BiomedAP , a parameter-efficient framework that adapts frozen vision–language backbones to medical tasks 5 ; 6 . BiomedAP addresses two common failure modes: (i) modality isolation and (ii) prompt fragility . As illustrated in Figure 2 , we keep the pre-trained encoders frozen and optimize two lightweight components: (1) Gated Cross-Modal Prompt Fusion for context-aware prompt refinement, and (2) Dual-Anchor Constraint (DAC) for semantic regularization. Figure 2 : Overall framework of BiomedAP. DAC constructs a High-Quality (Expert) anchor and a Low-Quality (Core) anchor, whose features ( f_{high} , f_{low} ) regularize the learnable text context to reduce prompt drift. Training uses a Confidence-Aware Adaptive Loss to balance anchor regularization

Read the complete text