BiCLIP: Bidirectional and Consistent Language–Image Processing for Robust Medical Image Segmentation
Saivan Talaei *, Fatemeh Daneshfar *, Abdulhady Abas Abdullah, Mustaqeem Khan ( ✉ ) · 2026
Read BiCLIP: Bidirectional and Consistent Language–Image Processing for Robust Medical Image Segmentation by Saivan Talaei *, Fatemeh Daneshfar *, Abdulhady Abas Abdullah, Mustaqeem Khan ( ✉ ) (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-00156.txt.
Introduction
Talaei and colleagues present BiCLIP, a system for robust medical image segmentation in messy real-world clinics with scarce labels and hardware-degraded images. Its two-way fusion lets visual features repeatedly refine text accounts for stronger meaning alignment. An augmentation consistency rule steadies learning by regularising middle representations against perturbed input views. On QaTa-COV19 and MosMedData+ it beats image-only and multimodal baselines, holding up on just 1% of labelled data and resisting motion blur and low-dose CT noise.
Catalog record
| CREATOR | Saivan Talaei *, Fatemeh Daneshfar *, Abdulhady Abas Abdullah, Mustaqeem Khan ( ✉ ) |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19922 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.00156
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BiCLIP: Bidirectional and Consistent Language–Image Processing for Robust Medical Image Segmentation Saivan Talaei *, Fatemeh Daneshfar *, Abdulhady Abas Abdullah, Mustaqeem Khan ( ✉ ) Abstract Medical image segmentation is essential for computer-assisted diagnosis and treatment planning, playing a key role in many real-world clinical workflows. Recently, multimodal vision-language methods have gained attention by leveraging textual descriptions to enhance semantic understanding. However, their robustness under realistic clinical conditions such as limited annotations and acquisition-related degradations remains insufficiently explored. We propose BiCLIP (Bidirectional and Consistent Language-Image Processing), a vision-language framework designed to improve robustness in medical image segmentation. BiCLIP introduces a bidirectional multimodal fusion mechanism that allows visual features to iteratively refine textual representations, strengthening semantic alignment. Additionally, an augmentation consistency objective regularizes intermediate representations across perturbed input views, promoting stable learning behavior. We evaluate BiCLIP on the QaTa-COV19 and MosMedData+ benchmarks, where it consistently outperforms strong image-only and multimodal baselines. Experiments show that BiCLIP remains robust when trained on only 1% of labeled data and withstands clinically motivated corruptions such as low-dose CT noise and motion blur. † † footnotetext: * These authors contributed equally to this work. 1 Introduction Deep learning approaches have become central to medical image segmentation, facilitating automatic segmentation of anatomical structures and disease-related regions with an accuracy that is similar to expert annotations. Early convolutional architectures, including U-Net [ 1 ] , were successful in integrating local spatial details with global contextual information, leading to ubiquitous use in medical imaging applications. Despite this progress, purely image-only models still depend on image quality and acquisition conditions [ 5 ] . In addressing these limitations, text-guided segmentation approaches use linguistic information to form complementary semantic context and enhance the robustness and interpretability across diverse medical imaging tasks [ 4 , 6 ] . A growing body of work has investigated the integration of vision-language representations for medical image segmentation. Zhao et al. [ 7 ] propose Cap2Seg, which combines image captioning with lesion segmentation to produce textual descriptions for prediction without depending on external reports. As for foundation models, Hu et al. [ 9 ] present LGA, a lightweight language-guided adapter that provides textual embeddings to the Segment Anything Model with minimal fine-tuning. Wang et al. [ 10 ] develop an attribute-aware referring segmentation framework to improve robustness under missing textual prompts. In parallel to these efforts, Wang et al. [ 11 ] propose MedLangViT, which explores improved multimodal fusion, providing better integration between linguistic and visual cues for medical image segmentation. Despite recent advances, existing vision–language segmentation methods often rely on a unidirectional fusion approach. In these methods, textual cues condition visual representations without allowing visual evidence to refine semantic cues. Additionally, they rarely incorporate explicit mechanisms to enhance robustness. As a result, learned representations may still be adversely affected by annotation scarcity and appearance variations. The method proposed here, BiCLIP, introduces a Bidirectional Multimodal Fusion (BMF) module that facilitates interaction between image and text representations in a bidirectional manner. Furthermore, an Image Augmentation Consistency (IAC) module regularizes intermediate features across perturbed inputs, thereby promoting robust representation learning. Our contributions are summarized as follows. (1) We develop a BMF module that facilitates mutual exchange between visual and linguistic representations, enabling semantic cues to adapt in response to visual information. This interaction supports robustness in settings with limited supervision and degraded image quality, where static textual conditioning may be insufficient. (2) We introduce an IAC module that constrains intermediate features to remain consistent across both weak and strong perturbations, improving stability under appearance changes. (3) We conduct extensive robustness evaluations under low-annotation regimes and clinically motivated corruption settings, including simulated low-dose CT noise and motion blur, demonstrating the effectiveness of the proposed design choices. 2 Method As shown in Fig. 1 , BiCLIP processes a medical image and its associated text using parallel visual and textual encoders. The resulting embeddings are integrated through the proposed BMF module to form fused representations, which are used to generate a pseudo image encoding cross-modal semantics. This pseudo image both refines the textual representation via a cycle-consistency objective and is concatenated with the original image for segmentation using a U-Net backbone. An IAC module further enforces representation stability under image perturbations. Details of each component are provided in the following subsections. Figure 1: Overview of the BiCLIP framework. (a) Overall multimodal segmentation architecture; (b) pseudo-image generator for bidirectional vision-language interaction; (c) IAC module for robust learning. 2.1 Bidirectional Multimodal Fusion Module Given an input image of size 224\times 224\times 3 and its associated clinical description, BiCLIP constructs aligned visual and textual representations for bidirectional multimodal fusion. The text is encoded using a frozen CXR-BERT model and projected to a compact text embedding, while the image is processed by a lightweight convolutional encoder to obtain a global visual embedding. These complementary embeddings serve as inputs to the proposed BMF module. To enable image-aware language modeling, the BMF module allows visual information to refine the textual representation. The text embedding \mathbf{t} and image embedding \mathbf{i} are concatenated to form a joint multimodal representation \mathbf{z}=[\mathbf{t};\mathbf{i}] , which aggregates semantic information from both modalities. This fused representation is passed through a multilayer perceptron g_{\text{BMF}}(\cdot) , which models cross-modal interactions and predicts a refinement term for the text embedding. The refined text embedding is then obtained via residual addition, preserving the original linguistic structure while incorporating visual cues: \displaystyle\Delta\mathbf{t} \displaystyle=g_{\text{BMF}}(\mathbf{z}),\qquad\mathbf{t}^{\prime}=\mathbf{t}+\Delta\mathbf{t}. (1) The refined text embedding \mathbf{t}^{\prime} is then transformed into a pseudo image \hat{\mathbf{x}} via a pseudo-image generator, which encodes cross-modal semantics and serves as an intermediate representation bridging language and vision; this generator is further supervised using a generated ground-truth signal to guide the pseudo-image formation. To close the bidirectional fusion loop, the pseudo image is mapped back into the text embedding space using an image-to-text head h(\cdot) , which projects visual content into the language embedding space, yielding the updated textual representation \hat{\mathbf{t}} . A cycle-consistency loss enforces alignment between the original and refined text embeddings, \displaystyle\mathcal{L}_{\text{cycle}}=\left\|{\mathbf{t}}-\hat{\mathbf{t}}\right\|_{2}^{2}. (2) Finally, the pseudo image is concatenated with the original image and forwarded to a U-Net-based segmentation network for mask prediction. 2.2 Image Augmentation Consistency Module Once the pseudo image \hat{\mathbf{x}} is obtained, it is combined with the original image \mathbf{x} along the channel dimension to produce a multimodal input \mathbf{x}_{\text{cat}} . During training, two augmented variants of \mathbf{x}_{\text{cat}} are generated to improve robustness to appearance variations. To ensure spatial alignment between inputs and masks, a spatial augmentation function \mathcal{A}_{g} is applied jointly to the multimodal input and the segmentation mask \mathbf{y} : \displaystyle(\mathbf{x}_{g},\mathbf{y}_{g}) \displaystyle=\mathcal{A}_{g}(\mathbf{x}_{\text{cat}},\mathbf{y}), (3) Two appearance-perturbed views are then generated from the augmented input \mathbf{x}_{g} . In particular, the real image component \mathbf{x}_{g}^{r} is processed with a weak augmentation \mathcal{A}_{w} and a strong augmentation \mathcal{A}_{s} to introduce different levels of appearance variation, while the pseudo image component \mathbf{x}_{g}^{p} is normalized using \mathcal{N}_{p} to serve as a stable semantic reference. After the weak and strong augmentations are applied, the resulting outputs are concatenated with the normalized \mathbf{x}_{g}^{p} : \displaystyle\mathbf{x}_{w} \displaystyle=\mathrm{concat}\!\big(\mathcal{A}_{w}(\mathbf{x}_{g}^{r}),\,\mathcal{N}_{p}(\mathbf{x}_{g}^{p})\big),\quad\mathbf{x}_{s}=\mathrm{concat}\!\big(\mathcal{A}_{s}(\mathbf{x}_{g}^{r}),\,\mathcal{N}_{p}(\mathbf{x}_{g}^{p})\big). (4) These two augmented views are forwarded through the same U-Net backbone. Let F(\cdot) denote the decoder feature map extracted from the final upsampling stage of the network. Accordingly, the weakly and strongly augmented inputs \mathbf{x}_{w}