From Panel to Pixel: Zoom-In Vision-Language Pretraining from Biomedical Scientific Literature
Kun Yuan, Min Woo Sun, Zhen Chen, Alejandro Lozano, Xiangteng He, Shi Li, Nassir Navab, Xiaoxiao Sun, Nicolas Padoy, Serena Yeung-Levy · 2025
Read From Panel to Pixel: Zoom-In Vision-Language Pretraining from Biomedical Scientific Literature by Kun Yuan, Min Woo Sun, Zhen Chen, Alejandro Lozano, Xiangteng He, Shi Li, Nassir Navab, Xiaoxiao Sun, Nicolas Padoy, Serena Yeung-Levy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-02566.txt.
Introduction
Kun Yuan and nine co-authors train AI models that read medical figures and text together. Biomedical vision language models learn from web scale scientific papers. Most training, though, pairs a whole figure with its caption and drops the fine links between small regions and words.
Clinicians rely on exactly those fine links when they zoom into local structures. The team's zoom in pretraining keeps panel level detail, matching small figure regions with the text that describes them. Models trained this way could support diagnosis and research tools that point to the right spot, not just the right page. Fine links like these are what let a tool point to a spot, not just a page, when it counts.
Catalog record
| CREATOR | Kun Yuan, Min Woo Sun, Zhen Chen, Alejandro Lozano, Xiangteng He, Shi Li, Nassir Navab, Xiaoxiao Sun, Nicolas Padoy, Serena Yeung-Levy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-02566
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Panel to Pixel: Zoom-In Vision-Language Pretraining from Biomedical Scientific Literature Kun Yuan, Min Woo Sun, Zhen Chen, Alejandro Lozano, Xiangteng He, Shi Li, Nassir Navab, Xiaoxiao Sun, Nicolas Padoy, Serena Yeung-Levy Abstract There is a growing interest in developing strong biomedical vision-language models. A popular approach to achieve robust representations is to use web-scale scientific data. However, current biomedical vision-language pretraining typically compresses rich scientific figures and text into coarse figure-level pairs, discarding the fine-grained correspondences that clinicians actually rely on when zooming into local structures. To tackle this issue, we introduce Panel2Patch, a novel data pipeline that mines hierarchical structure from existing biomedical scientific literature, i.e., multi-panel, marker-heavy figures and their surrounding text, and converts them into multi-granular supervision. Given scientific figures and captions, Panel2Patch parses layouts, panels, and visual markers, then constructs hierarchical aligned vision-language pairs at the figure, panel, and patch levels, preserving local semantics instead of treating each figure as a single data sample. Built on this hierarchical corpus, we develop a granularity-aware pretraining strategy that unifies heterogeneous objectives from coarse didactic descriptions to fine region-focused phrases. By applying Panel2Patch to only a small set of the literature figures, we extract far more effective supervision than prior pipelines, enabling substantially better performance with less pretraining data. 1 Introduction Generalist vision–language foundation models are rapidly becoming central to biomedical research by learning unified representations across radiology 4 , cellular microscopy 44 , and surgical videos 41 ; 42 from large-scale image–text pairs. A widely adopted strategy for constructing supervision is inspired by how human experts are trained: exposure to textbooks and educational articles that provide rich multimodal content capturing expert knowledge, visual patterns, and their semantic associations, enabling strong generalization to tasks such as visual question answering 21 and image captioning 9 . However, these models often struggle when tasks require fine-grained discrimination between similar attributes, spatial configurations, or subtle semantic distinctions. This gap stems from the nature of scientific corpora, where figures are mostly multi-panel and captions provide high-level summaries, leading to coarse figure-level pairs for pretraining 26 ; 16 ; 18 that favor thematic alignment but lack the localized grounding clinicians rely on when zooming into specific structures, as illustrated in Fig. 1 . Figure 1 : Our Panel2Patch pipeline generates additional fine-grained vision-language supervision signals and enhances the multi-modal representation through cross-level message passing. Table 1 : Comparison of vision-language data generation pipelines. Our Panel2Patch produces multi-panel, single-panel, and region-level correspondences, covering all granularities with automatic hierarchical captions and no additional annotation cost. Dataset Multi-panel Single-panel Region Captions Grounding Human Annotation Cost FG-CLIP 34 \times \checkmark \checkmark CogVLM2 Yolo-World No High GRIT 22 \times \checkmark \times Annotated Captions GLIP No High FineCLIP 30 \times \checkmark \checkmark BLIP2 YOLO Yes Moderate MEDTRINITY-25M 35 \times \checkmark \checkmark GPT-4v + LLaVA_MedCap Expert Models & GT Yes Moderate Biomedica 16 \checkmark \times \times Literature Captions N/A Yes Low Open-PMC-18M 2 \checkmark \checkmark \times Literature Captions YOLO No Low Ours \checkmark \checkmark \checkmark Hierarchical Captions QWen2.5-VL No Low Existing vision-language data generation pipelines face a fundamental tradeoff between scalability and fine-grained supervision, as shown in Tab. 1 . Natural computer vision methods 12 ; 34 ; 33 ; 30 achieve fine-grained region-level alignment through human annotations, pretrained detectors, or specialized captioners, but such annotation-intensive pipelines incur high computational costs and poorly generalize to biomedical domains due to domain gaps and imaging diversity. Conversely, scalable approaches that mine from biomedical scientific literature sacrifice granularity. They either treat multi-panel figures as single coarse instances 44 ; 16 or decompose them into panels 2 while still retaining figure-level captions that reference multiple elements, thus resulting in weak and coarse image-text alignment and missing the fine-grained region-level correspondences. Fundamentally, none of these methods produce hierarchical supervision spanning all three granularities, i.e., multi-panel figures, individual panels, and fine-grained regions, with corresponding hierarchical captions that capture both global context and localized semantics. This absence of multi-level, semantically grounded supervision limits current models’ ability to learn fine-grained visual-textual correspondences in a scalable manner, hindering precise biomedical multimodal understanding. To address these challenges, we propose a comprehensive approach that breaks the scalability-granularity tradeoff through two synergistic contributions. First, we introduce Panel2Patch , a data generation pipeline that exploits a key insight: scientific figures already contain hierarchical visual structure and explicit localization cues that can be automatically extracted . Rather than requiring expensive annotations or specialized detectors, Panel2Patch leverages the inherent pedagogical design of scientific literature, i.e., multi-panel layouts, panel identifiers, and embedded visual markers (arrows, bounding boxes, zoom-in crops), to automatically mine supervision at three granularities: figure, panel, and patch/region. Using off-the-shelf vision-language models with set-of-marks (SoM) prompting, it parses panel structures, decomposes captions, and localizes patches by detecting the visual markers authors use to highlight regions of interest. This yields hierarchical image-text pairs from global figure context to localized patch descriptions, achieving multi-level, fine-grained supervision at scale with minimal cost, as shown in Tab. 1 . Panel2Patch establishes a methodology for mining hierarchical supervision from any domain where visual documentation follows pedagogical conventions, offering a scalable alternative to manual annotation pipelines for biomedicine and other scientific fields to build their foundation models. We also design a hierarchical zoom-in pretraining framework that uses these multi-granular pairs to train a single CLIP-style encoder whose panel-level embeddings serve as the primary representation for biomedical downstream tasks. The encoder is optimized jointly with auxiliary figure- and patch-level objectives and inter-level message passing, so that panel embeddings are explicitly refined by global figure context and fine-grained regional evidence, through top-down context propagation from figures to panels with global guidance and bottom-up evidence aggregation from patches to panels with localized cues. This hierarchical learning strategy produces a unified encoder where the figure and patch levels serve as contextual scaffolding that enhances the panel-level representations, while maintaining robust visual understanding across all granularities. Together, these achieve data-efficient learning, outperforming prior biomedical VLP models trained on \geq 10\times larger corpora. Our contributions are threefold: • We propose Panel2Patch, a scalable pipeline that automatically extracts hierarchical supervision (figure, panel, patch) from scientific literature’s pedagogical structure. • We design a hierarchical zoom-in pretraining framework with inter-level message passing that enhances biomedical multimodal representations using multi-granularity context and fine-grained correspondences. • We achieve state-of-the-art performance on multiple external standard biomedical benchmarks, i.e., PatchCamelyon 28 , \mu -bench 15 , MedMNIST 36 , LC25000 3 , and Chexpert 7 , across different biomedical specialties, using 60% less data than prior work, which paves a efficient path towards biomedical foundation model. 2 Related Work 2.1 Biomedical Vision-Language Pretraining Recent biomedical vision–language models mostly scale literature-derived image–text pairs. Early work such as PMC-CLIP 14 trains on 1.6M figure–caption pairs, while BiomedCLIP 44 and BIOMEDICA 16 expand to 15M and 24M pairs, respectively, improving domain-specific representations via larger corpora and richer metadata. However, these pipelines treat each scientific figure as a single image-level instance, even when figures are multi-panel with captions that mix global context and panel-specific details, leading to coarse supervision misaligned with how clinicians actually zoom into panels and local regions. To obtain finer supervision, MedTrinity-25M 35 augments biomedical corpora with expert-verified ROIs and GPT-4V/LLaVA-based captions, while FineCLIP 30 and FG-CLIP 34 ; 33 build region–text datasets using