Towards Generalized Multimodal Homography Estimation
Jinkun You, Jiaxin Cheng, Jie Zhang, Yicong Zhou · 2026
Read Towards Generalized Multimodal Homography Estimation by Jinkun You, Jiaxin Cheng, Jie Zhang, Yicong Zhou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03956.txt.
Introduction
Jinkun You and others propose a training data synthesis method for generalized multimodal homography gauging. The new method creates unaligned pairs with known offsets from one input image, rendering varied textures and colors while keeping structural information. The team also designs a network fully leveraging cross-scale information and decoupling color information from feature pictures, improving gauging score.
Catalog record
| CREATOR | Jinkun You, Jiaxin Cheng, Jie Zhang, Yicong Zhou |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35471 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.03956
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Generalized Multimodal Homography Estimation Jinkun You, Jiaxin Cheng, Jie Zhang, Yicong Zhou Abstract Supervised and unsupervised homography estimation methods depend on image pairs tailored to specific modalities to achieve high accuracy. However, their performance deteriorates substantially when applied to unseen modalities. To address this issue, we propose a training data synthesis method that generates unaligned image pairs with ground-truth offsets from a single input image. Our approach renders the image pairs with diverse textures and colors while preserving their structural information. These synthetic data empower the trained model to achieve greater robustness and improved generalization across various domains. Additionally, we design a network to fully leverage cross-scale information and decouple color information from feature representations, thus improving estimation accuracy. Extensive experiments show that our training data synthesis method improves generalization performance. The results also confirm the effectiveness of the proposed network. 1 Introduction Homography estimation identifies a matrix that characterizes the projective transformation between two images of the same scene captured from different viewpoints. This transformation allows one image to be warped using the estimated matrix, enabling spatial alignment with the other image. Achieving this alignment is crucial for various applications, such as image stitching 25 ; 20 ; 4 , image fusion 46 ; 47 ; 33 , and guided super-resolution 32 ; 37 ; 39 . Traditional estimation methods depend on hand-crafted features to match points between image pairs for homography matrix calculation 22 ; 28 ; 16 . However, these approaches yield insufficient features in low-texture scenarios. To overcome this limitation, deep learning techniques have been introduced, leveraging their remarkable capabilities for feature extraction and representation across various domains 11 ; 40 ; 26 . The pioneering deep homography estimation method, inspired by supervised learning, extracts features from image pairs to predict the positional offsets of four corner points 12 . These offsets can then be used to compute the homography matrix through the direct linear transform algorithm 13 . During training, the model’s weights are adjusted by minimizing the difference between the predicted offsets and the ground truth. To further reduce estimation errors, recent supervised approaches have adopted the inverse compositional Lucas-Kanade framework to refine the predicted offsets iteratively 2 ; 45 ; 5 . These methods estimate offsets multiple times and progressively aggregate the results. Nevertheless, supervised methods require well-aligned image pairs or unaligned pairs with ground-truth offsets for training. The collection of aligned pairs poses a significant challenge, particularly for multimodal images captured by different sensors 19 . Additionally, obtaining ground-truth data in real-world scenarios is often difficult 24 . These challenges substantially impede the training of supervised methods. Unsupervised methods have gained significant attention recently because they do not require aligned images or ground truth for training. Early approaches focused on optimizing model parameters by maximizing visual similarity between image pairs 23 ; 42 ; 36 . To achieve this, offsets are predicted to align the images prior to calculating their similarity. However, these methods struggle with accuracy on multimodal image pairs due to substantial differences in appearance. Some strategies employ image translation techniques to standardize modalities while simultaneously learning homography estimation 1 ; 31 ; 34 . Nonetheless, these methods are limited to handling only small deformations, resulting in low accuracy. Self-supervised learning techniques have been introduced to address this issue. These methods generate pseudo-unaligned image pairs along with the corresponding ground truth, thereby enhancing homography estimation capabilities 43 ; 41 . The modality gap can be narrowed by enforcing consistency in image appearance or feature representation 30 ; 43 ; 41 . As a result, the estimation results are further improved. Supervised and unsupervised homography estimation methods have demonstrated promising accuracy. As illustrated in Fig. 1 , these models perform admirably when trained and tested on the same dataset. The training data helps adapt the models to the specific modalities involved. However, there are several limitations. First, the trained models exhibit limited generalization capability to other modalities. The estimation accuracy declines when image pairs demonstrate significant appearance differences across modalities. To improve performance, existing approaches require the collection of numerous image pairs from target modalities to learn modality-specific information. This need escalates both time expenditure and labor costs. Second, current methods tend to use features from different scales in isolation. While they effectively harness intra-scale information, they neglect complementary cross-scale information that is beneficial for establishing correspondences between the image pair 38 . Third, the integration of color information within the features can degrade the processing capabilities for multimodal images 27 . To address the above challenges, we introduce a training data synthesis method along with a homography estimation network. Our contributions are as follows: • We propose a training data synthesis method that enables zero-shot multimodal homography estimation. By generating synthetic data with various textures and colors, our approach enhances models’ ability to generalize across different modalities. This synthesis method can also be applied to existing datasets, leading to improved generalization performance. • We design a homography estimation network to achieve higher accuracy. It integrates cross-scale information in both top-to-bottom and bottom-to-top directions. Besides, the network decouples color information from feature representations, further enhancing the estimation. • We conduct comprehensive experiments to demonstrate the effectiveness of both the proposed training data synthesis method and the homography estimation network. 2 Related Work Figure 1 : Estimation results. The rows present the results of the supervised MCNet 49 and the unsupervised SSHNet 41 . The first two columns represent within-dataset results, while the remaining columns depict cross-dataset performance. Greater similarity between the green and red quadrilaterals indicates higher accuracy. Supervised Homography Estimation. DeTone et al. 12 found that parameterizing homography estimation as four-point offsets is more effective. They developed a VGG-style network to regress these offsets based on concatenated image pairs. The model is trained by minimizing the difference between the ground truth and the predicted offsets. Existing supervised methods adopt this training framework. However, regressing offsets just once limits accuracy. To address this, some approaches 14 ; 48 ; 18 cascade multiple networks for warping and processing input images multiple times. This strategy progressively refines the estimation result and reduces estimation errors, but it also introduces additional parameters. In contrast, the inverse compositional Lucas-Kanade (IC-LK) algorithm maintains model compactness while allowing iterative refinement of the estimation results. Chang et al. 7 introduced a differentiable yet untrainable IC-LK layer, enabling effective model training. Building on this, Le et al. 45 enhanced performance for multimodal image pairs through brightness-consistent feature extraction. Cao et al. 5 proposed a trainable iterative framework to fully leverage the data-driven capabilities of deep learning. They subsequently designed an attention mechanism to capture correspondences across various ranges 6 . Zhu et al. 49 integrated the iterative strategy into multiscale correlation searching, resulting in improved estimation accuracy and efficiency. Unsupervised Homography Estimation. Ground-truth offsets are often unavailable in real-world scenarios. This issue prompts increased interest in unsupervised methods since they do not require ground truth data for model training. Nguyen et al. 23 introduced the first unsupervised method. This method featured a differentiable warping layer and utilized photometric loss to minimize the visual discrepancies between image pairs. To enhance estimation accuracy, Zhang et al. 42 implemented attention maps to