Tell2Reg: Establishing spatial correspondence between images by the same language prompts
Wen Yan, Qianye Yang, Shiqi Huang, Yipei Wang, Shonit Punwani, Mark Emberton, Vasilis Stavrinides, Yipeng Hu, Dean Barratt · 2025
Read Tell2Reg: Establishing spatial correspondence between images by the same language prompts by Wen Yan, Qianye Yang, Shiqi Huang, Yipei Wang, Shonit Punwani, Mark Emberton, Vasilis Stavrinides, Yipeng Hu, Dean Barratt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-03118.txt.
Introduction
Wen Yan and eight co-authors align medical images by asking a large vision model the same word prompt on both scans. Past alignment nets guess shifts or warps, which need costly labeled training. Their method asks the model to mark the same named region in each image, and that pair defines the match.
No training or labels are needed at all. On a hard task, prostate scans across patients with wide shape and shade change, it beat untaught methods and matched weakly taught ones. Word sense seems to carry place sense, a first hint of a deep link.
Catalog record
| CREATOR | Wen Yan, Qianye Yang, Shiqi Huang, Yipei Wang, Shonit Punwani, Mark Emberton, Vasilis Stavrinides, Yipeng Hu, Dean Barratt |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-03118
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Tell2Reg: Establishing spatial correspondence between images by the same language prompts Wen Yan, Qianye Yang, Shiqi Huang, Yipei Wang, Shonit Punwani, Mark Emberton, Vasilis Stavrinides, Yipeng Hu, Dean Barratt Abstract Spatial correspondence can be represented by pairs of segmented regions, such that the image registration networks aim to segment corresponding regions rather than predicting displacement fields or transformation parameters. In this work, we show that such a corresponding region pair can be predicted by the same language prompt on two different images using the pre-trained large multimodal models based on GroundingDINO and SAM. This enables a fully automated and training-free registration algorithm, potentially generalisable to a wide range of image registration tasks. In this paper, we present experimental results using one of the challenging tasks, registering inter-subject prostate MR images, which involves both highly variable intensity and morphology between patients. Tell2Reg is training-free, eliminating the need for costly and time-consuming data curation and labelling that was previously required for this registration task. This approach outperforms unsupervised learning-based registration methods tested, and has a performance comparable to weakly-supervised methods. Additional qualitative results are also presented to suggest that, for the first time, there is a potential correlation between language semantics and spatial correspondence, including the spatial invariance in language-prompted regions and the difference in language prompts between the obtained local and global correspondences. Code is available at https://github.com/yanwenCi/Tell2Reg.git . † † address: { 1 UCL Hawkes Institute; 2 Department of Medical Physics and Biomedical Engineering; 3 Cancer Institute; 4 Urology Department, UCL Hospital; 5 Centre for Medical Imaging, Division of Medicine; 6 Division of Surgery and Interventional Science}, University College London 7 Radiology Department, Imperial College Healthcare 1 Introduction Figure 1: A brief illustration of how Tell2Reg framework uses text prompts to generate corresponding ROIs from fixed and moving images. MR imaging has increasingly been used for diagnosing prostate cancer, planning targeted biopsy and other treatment procedures. Registering images from different patients is an interesting research topic, which may enable propagating procedural plans from reference images to new patients, constructing MR-based lower-pelvic atlases and other similar population studies [ 1 ] . Among the sequences that are useful for prostate cancer diagnosis, T2-weighted sequences contain the richest soft tissue contrast and are often used in these registration tasks, which is a focus of the previous work as well as this study 1 1 1 The potential need, albeit comparably minor, of inter-sequence registration has been a research topic of its own [ 2 ] and is not discussed further in this paper. . Registering inter-subject lower-pelvic MR images is challenging due to significant variability in intensity and morphology between subjects, unlike the intra-subject registration tasks [ 3 ] . This task usually requires large training datasets and benefits from segmentation labels and advanced training strategies [ 4 ] . Classical iterative algorithms has been shown inferior to learning-based approaches in this application [ 4 ] . Existing image registration methods often use deep learning models to learn pixel or feature correspondences between moving (to be aligned) and fixed (reference) images [ 5 , 6 , 7 ] . These models typically optimize a similarity metric that measures how well the warped moving image aligns with the fixed image, based on intensity differences, such as cross correlation and mutual information. Recent work also incorporated large language models for the same optimisation goal [ 8 , 9 ] , to predict a deformation field or a parametric transformation. Following the discussion in a previous work [ 10 ] that connects region-level correspondence and the ability to segment a wide range of objects, with or without anatomical significance, this paper describes a novel approach to establish spatial correspondence by multimodal approaches with language prompts. Notably, segment anything model (SAM) [ 11 ] has been developed together with language models, such as CLIP [ 12 ] and Bert [ 13 ] , to generate regions of interest (ROIs) from text prompts. GroundingDINO [ 14 ] models have merged the multi-level language model with object detection, generating ROIs from text prompts via contrastive learning. We hypothesize that the image registration task does not need to optimise pixel-level correspondence during model training; instead, it can be conceptualized as detecting the same regions from different images, therefore leveraging capabilities of segmentation models such as SAM without retraining. Further, by utilising pre-trained multimodal systems, these corresponding regions ought to be directly predicted by prompting identical text in the proposed “Tell2Reg” approach, illustrated in Fig. 1 . The proposed image registration does not require any training or finetuning, outperforming the tested state-of-the-art learning-based registration methods. 2 Method The above assertion - segmenting regions on both fixed and moving images using identical text prompts yields corresponding ROIs - represents an ideal scenario rather than a guaranteed outcome. With practical constraints such as the availability of medical-image-finetuned foundation models, selecting ROIs may require further prompt engineering and/or post-processing to ensure that the segmented ROIs are corresponding pairs. This section describes a specific algorithm, utilising pre-trained SAM and GroundingDINO, outlined in Fig. 2 . 2.1 Text to corresponding ROIs We start our discussion with a theorem [ 10 ] stating that the registration task can be framed as a correspondence learning problem between regions. Let the fixed and moving images be I^{\textit{fix}} and I^{\textit{mov}} , respectively. Establishing spatial correspondence between the two images can be achieved by identifying K pairs of corresponding ROIs, denoted as \{(R_{k}^{\textit{fix}},R_{k}^{\textit{mov}})\}_{k=1}^{K} . In the limiting case where each ROI reduces to a single pixel, the ROI-based correspondence becomes equivalent to a pixel-wise one, as governed by a dense displacement field (DDF). We use GroundingDINO [ 14 ] to generate bounding boxes {B}^{\textit{fix}}=f_{dino}(I^{\textit{fix}},p) and {B}^{\textit{mov}}=f_{dino}(I^{\textit{mov}},p) for fixed and moving images with same text prompt p , respectively, where p can be any text, examples are given in Table 2 . Ideally, bounding boxes generated from identical text prompts should correspond precisely between the fixed and moving images; however, due to the limits of the GroundingDINO model on prostate data, which could lead to false positive correspondence, by detecting regions that appear similar based on texture or intensity but are not corresponding regions of registration interest. Figure 2: Tell2Reg framework using pre-trained GroundingDINO and SAM. The ROI-to-dense transformation is an optional step for comparing with other methods. Before inputting BBox B=\{B^{fix},B^{mov}\} into the SAM model, we filter out those that are too small or too large, for reducing computation costs and potential outliers. We then use the remaining bounding boxes as box prompts to segment ROIs in both fixed and moving images, to obtain the segmented ROIs for fixed and moving: {R}^{\textit{fix}}=f_{sam}(I^{\textit{fix}},{B}^{\textit{fix}}) and {R}^{\textit{mov}}=f_{sam}(I^{\textit{mov}},{B}^{\textit{mov}}) resulting in \{M^{\textit{fix}}\}_{k=1}^{K^{\textit{fix}}} and \{M^{\textit{mov}}\}_{k=1}^{K^{\textit{mov}}} as the binary masks of the ROIs in the fixed and moving images, respectively. 2.2 ROI correspondence refinement To establish robust correspondence between these ROIs, we propose examining the similarity between ROI prototypes from the same or different text prompts. The ROI prototypes are generated by averaging the ROIs in the fixed and moving images. Given two sets of ROIs, \{{R}^{\textit{fix}}\}_{k=1}^{K^{\textit{fix}}} and \{{R}^{\textit{mov}}\}^{K^{\textit{mov}}}_{k=1} , where K^{\textit{mov}} may not be equal to K^{\textit{fix}} , the proposed matching process finds the correspondence ROIs by minimizing the L^{2} distance between the ROI prototypes, detailed