BioCAP : Exploiting Synthetic Captions Beyond Labels in Biological Foundation Models
Ziheng Zhang Xinyue Ma Arpita Chowdhury Elizabeth G Campolongo, Matthew J Thompson Net Zhang Samuel Stevens Hilmar Lapp, Tanya Berger-Wolf Yu Su Wei-Lun Chao Jianyang Gu · 2025
Read BioCAP : Exploiting Synthetic Captions Beyond Labels in Biological Foundation Models by Ziheng Zhang Xinyue Ma Arpita Chowdhury Elizabeth G Campolongo, Matthew J Thompson Net Zhang Samuel Stevens Hilmar Lapp, Tanya Berger-Wolf Yu Su Wei-Lun Chao Jianyang Gu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20095.txt.
Introduction
Ziheng Zhang and eleven co-authors test whether descriptive captions can improve biological foundation models beyond simple species labels. Images and captions can each reveal different traits of a species, so training with both should stress real diagnostic features and weaken false correlations. The hard part is getting faithful, specimen-specific captions at large scale.
The team generates synthetic captions with multimodal large language models, guided by visual information drawn from Wikipedia and by taxon-specific format examples to reduce invented detail. Training with those captions produces BioCAP, a model that performs strongly on species classification and on matching text with images. The work shows captions carry useful signal that labels alone miss.
Catalog record
| CREATOR | Ziheng Zhang Xinyue Ma Arpita Chowdhury Elizabeth G Campolongo, Matthew J Thompson Net Zhang Samuel Stevens Hilmar Lapp, Tanya Berger-Wolf Yu Su Wei-Lun Chao Jianyang Gu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-20095
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BioCAP : Exploiting Synthetic Captions Beyond Labels in Biological Foundation Models Ziheng Zhang Xinyue Ma Arpita Chowdhury Elizabeth G Campolongo, Matthew J Thompson Net Zhang Samuel Stevens Hilmar Lapp, Tanya Berger-Wolf Yu Su Wei-Lun Chao Jianyang Gu Abstract This work investigates descriptive captions as an additional source of supervision for biological multimodal foundation models. Images and captions can be viewed as complementary samples from the latent morphospace of a species, each capturing certain biological traits. Incorporating captions during training encourages alignment with this shared latent structure, emphasizing potentially diagnostic characters while suppressing spurious correlations. The main challenge, however, lies in obtaining faithful, instance-specific captions at scale. This requirement has limited the utilization of natural language supervision in organismal biology compared with many other scientific domains. We complement this gap by generating synthetic captions with multimodal large language models (MLLMs), guided by Wikipedia-derived visual information and taxon-tailored format examples. These domain-specific contexts help reduce hallucination and yield accurate, instance-based descriptive captions. Using these captions, we train BioCAP ( i.e . , BioCLIP with Captions), a biological foundation model that captures rich semantics and achieves strong performance in species classification and text-image retrieval. These results demonstrate the value of descriptive captions beyond labels in bridging biological images with multimodal foundation models 1 1 1 Models, data, and code available at imageomics.github.io/biocap . . Figure 1: Left : Different strategies to create captions for biological images. Wikipedia offers rich domain knowledge, but descriptions are often generic and not directly grounded in the given image. Multimodal large language models (MLLMs) may hallucinate details when conditioned solely by images (wrong color description in this example). Incorporating Wikipedia-derived visual information and taxon-tailored format examples as contexts helps generate accurate, image-specific captions. Right : Using these descriptive captions as additional supervision, BioCAP captures fine-grained biological semantics. Please refer to Figure 6 for detailed comparisons with other models. 1 Introduction Multimodal foundation models learn from vast datasets of paired visual and textual inputs ( Radford et al. 2021 ; Liu et al. 2023 ; Zhai et al. 2023 ; Hurst et al. 2024 ) . They demonstrate strong generalization across various downstream tasks, such as classification and visual question answering (VQA) ( Yue et al. 2024 ; Mai et al. 2025 ) . In general domains, the web provides a nearly limitless supply of such paired supervision ( Gadre et al. 2023 ; Schuhmann et al. 2022 ) . In scientific applications, however, such resources are unevenly distributed. Biomedical imaging benefits from clinical reports and radiology notes associated with image instances that offer detailed natural language supervision ( Wang et al. 2022 ; Li et al. 2023 ; Zhang et al. 2024 ) . By contrast, many other scientific domains—including organismal biology, astronomy, geology, and material science ( Stevens et al. 2024 ; Parker et al. 2024 ; Vivanco Cepeda et al. 2023 ; Harnik et al. 2025 ) —often lack instance-level textual descriptions and must solely rely on symbolic labels ( e.g . , species names). This leaves the potential of natural-language captions untapped for scientific multimodal learning. In this work, we focus on organismal biology. From the representation learning perspective, each species can be described by an underlying latent vector in the morphospace, which encodes its ground-truth biological characteristics, i.e . , traits ( Budd 2021 ) . Images and descriptive captions can be viewed as two projections of this latent vector. Each captures certain traits while also introducing noise ( e.g . , pose, lighting). Aligning both modalities encourages the model to recover the shared latent structure and focus on potentially diagnostic characters, thereby suppressing reliance on spurious correlations to noise. While the potential exists, in practice, the effectiveness relies on the availability of captions that are both instance-specific and faithful. Noisy or hallucinated captions can, instead, introduce contradictory signals that harm multimodal alignment ( Huang et al. 2021 ) . Biological research has produced massive repositories of organismal images, but most are annotated only with species names ( Stevens et al. 2023b ; Yang et al. 2024 ) . Collecting instance-based captions at scale is inherently challenging as expert-level annotation for millions of images demands exhaustive human labor ( Van Horn et al. 2018 ) . A natural solution is to leverage recent progress in multimodal large language models (MLLMs) to automatically generate image-aligned captions ( Fan et al. 2023 ; Lai et al. 2024 ) . However, biological species differentiation often depends on subtle morphological details. Without proper guidance, MLLMs tend to hallucinate about these details due to the aforementioned noise in the images. Figure 1 shows such an example, where the model incorrectly describes the color of the Calliope Hummingbird. Upon this observation, we suggest that domain knowledge has to be incorporated in the context. Specifically, we collect species-level visual information from Wikipedia ( Wikipedia contributors 2025 ) as ground-truth characters. As the model needs to attend to different traits to describe different species, we also curate format examples based on taxonomic classes to encourage explicit focus. These domain-specific contexts help MLLMs generate accurate and trait-focused descriptions grounded in the input image. As shown in Figure 1 , contextualizing the model with Wikipedia-derived visual information and taxon-tailored format examples corrects the earlier hallucination and substitutes it with accurate biological details. Building on these curated contexts, we generate instance-level synthetic captions for the large-scale TreeOfLife-10M dataset ( Stevens et al. 2023b ; Van Horn & Mac Aodha 2021 ; Gharaee et al. 2023 ; Encyclopedia of Life 2023 ) . We then train the BioCAP model ( i.e . , BioCLIP with Captions) with species names and captions as complementary supervision. As shown on the right of Figure 1 , BioCAP demonstrates a rich understanding of biological semantics. Further comparisons with BioCLIP (trained without captions) and the initial CLIP checkpoint in Figure 6 show improved semantic alignment of BioCAP . Quantitatively, BioCAP is evaluated on species classification and biological natural-language benchmarks, where it outperforms BioCLIP by 8.8\% and 21.3\% , respectively. These results answer the core question of this work: descriptive captions, when grounded in biological knowledge, provide essential supervision that bridges organismal images with multimodal foundation models. 2 Related Work 2.1 Multimodal Foundation Models for Scientific Images Multimodal large language models (MLLMs) such as GPT-4o ( Hurst et al. 2024 ) and LLaVA ( Liu et al. 2023 ) have demonstrated a powerful ability in tasks involving images and natural language like captioning, VQA, and open-ended reasoning. This paradigm has been successfully extended to specialized domains: LLaVA-Med ( Li et al. 2023 ) adapts the LLaVA framework to radiology for domain-specific VQA, ChemVLM ( Li et al. 2025 ) reasons about molecular structures, and BiomedGPT ( Zhang et al. 2024 ) unifies diverse biomedical tasks under a single multimodal backbone. In parallel, CLIP-style contrastive frameworks have also been adapted to a variety of scientific imagery domains. BioCLIP ( Stevens et al. 2024 ; Gu et al. 2025 ) and BioTrove-CLIP ( Yang et al. 2024 ) align organismal images with taxonomic names. CLIBD ( Gong et al. 2025 ) uses DNA barcodes as supervision for biodiversity tasks. MedCLIP ( Wang et al. 2022 ) extracts standardized UMLS entities from radiology reports, and