MatRes: Zero-Shot Test-Time Model Adaptation for Simultaneous Matching and Restoration

Kanggeon Lee, Soochahn Lee, Kyoung Mu Lee · 2026

Read MatRes: Zero-Shot Test-Time Model Adaptation for Simultaneous Matching and Restoration by Kanggeon Lee, Soochahn Lee, Kyoung Mu Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10081.txt.

Introduction

Lee, Lee, and Lee treat image restoration and geometric matching as one joint problem with MatRes. Treated apart, the two tasks interfere when real photo pairs suffer both heavy damage and large viewpoint shifts. MatRes adapts at test time from a single low-quality and high-quality pair, updating only light modules while pretrained parts stay frozen. It needs no offline training or extra supervision. Tests across varied pairs show clear gains in both restoration and alignment over either kind of model alone.

Catalog record

CREATORKanggeon Lee, Soochahn Lee, Kyoung Mu Lee
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38314
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.10081

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MatRes: Zero-Shot Test-Time Model Adaptation for Simultaneous Matching and Restoration Kanggeon Lee, Soochahn Lee, Kyoung Mu Lee Abstract Real-world image pairs often exhibit both severe degradations and large viewpoint changes, making image restoration and geometric matching mutually interfering tasks when treated independently. In this work, we propose MatRes , a zero-shot test-time adaptation framework that jointly improves restoration quality and correspondence estimation using only a single low-quality and high-quality image pair. By enforcing conditional similarity at corresponding locations, MatRes updates only lightweight modules while keeping all pretrained components frozen, requiring no offline training or additional supervision. Extensive experiments across diverse combinations show that MatRes yields significant gains in both restoration and geometric alignment compared to using either restoration or matching models alone. MatRes offers a practical and widely applicable solution for real-world scenarios where users commonly capture multiple images of a scene with varying viewpoints and quality, effectively addressing the often-overlooked mutual interference between matching and restoration. 1 Introduction When photographing a scene, it is common to capture multiple images from different viewpoints and poses. In practice, however, some of these images often suffer from degradations such as blur, noise, or low light. Under such real-world conditions, the two fundamental tasks, restoring image content and establishing accurate correspondences, tend to hinder each other when performed independently. To fully exploit the available images, restoration and matching should ideally be performed jointly so that each task can benefit from the other. However, despite extensive research on image matching [ 16 , 63 , 73 ] and image restoration [ 14 , 62 , 72 , 53 , 13 , 68 , 23 ] , including recent all-in-one restoration approaches typically designed for single images [ 24 ] , few methods address their joint optimization. Some methods have proposed to perform image alignment and restoration jointly or iteratively, but for grouped images with very small baselines [ 69 , 4 , 28 , 15 ] or videos [ 48 , 25 , 41 , 58 ] . To the best of our knowledge, there has yet to be sufficient research on this approach for image pairs with wide baselines, as those in image matching benchmarks [ 2 , 35 , 74 , 32 , 10 ] . This lack of research is most likely attributed to the dilemma of data acquisition. While the complexity of the task warrants a deep learning approach, it is nearly impossible to acquire the training data in real-world situations suitable to achieve generalization and robust performance, including ground-truth (GT) clean images, GT transformation fields, and known degradation kernels. In this paper, we propose a novel framework to address the issue of simultaneous matching and restoration, which we call MatRes . The key to our framework is based on the realistic assumption that for the given image pair, one is the more corrupted, low-quality (LQ) image while the other is the relatively high-quality (HQ) image that can be used as reference. Then, pretrained models for matching and restoration can be adapted to the given images by enforcing conditional similarity between the LQ and HQ images at corresponding points. Our method eliminates the need for offline training, performing adaptation through an iterative optimization that functions directly at inference time. Our method comprises three sub-networks: a pretrained matching network, a pretrained restoration network, and a lightweight LoRA-based projection network [ 20 ] . Given an input LQ–HQ image pair, the matching network first extracts generative priors and estimates the geometric relationship between the two images. These priors are then projected into the feature space of the restoration network by the LoRA projection network, providing alignment-aware conditioning for restoration. The loss function comprises two terms; the sparsity of the similarity matrix of the feature maps at coinciding local neighborhoods, and the pixel differences between the restored images in overlapping positions. These loss terms are used to iteratively train the LoRA parameters. Finally, the restoration network generates the restored image, while the output of the matching network generates the image correspondences. We present extensive experimental results with different combinations of various matching and restoration networks. We demonstrate that this mutual guidance between matching and restoration network yields substantial improvements in restoration quality compared to applying the restoration network to the LQ image alone without any geometric guidance. We also demonstrate that the accuracy of point correspondences of our framework significantly outperforms the results of using the matching network alone. Overall, our work provides the following contributions: • The proposed method is practical and widely applicable for real-world scenarios, as it is very common for the user to take a group of two or more images of the scene with varying viewpoints and quality, and addresses the issue of mutual deterrence of matching and restoration, which has mostly been overlooked by previous works. • Our method does not require any additional training data other than the given reference image in the input image pair, and does not require any offline training. • Our method jointly improves both restoration quality and matching accuracy compared to existing methods that separately perform each task, for a pair of images. 2 Related works Joint matching and restoration methods The work of Zhang et al. [ 69 ] is a notable attempt at addressing the issue of simultaneous restoration and matching, prior to deep learning. A mathematical model is formulated on the assumed relation between image transforms and blur kernels, both assumed to be based on the motion between the images, and iteratively optimized to recover the restored image and the motion parameters. While the method provides a sophisticated approach, it is limited to images taken at very close intervals. More recent works employing deep learning have also leaned into this assumption, operating on burst mode photos [ 28 , 15 ] or video [ 48 , 25 , 41 , 58 ] . Our method differs with these works, as we assume image pairs where the degradations are uncorrelated with the geometric transforms between images. Moreover, the transforms can be formulated to be non-rigid, depending on the component matching model. Restoration and matching methods MatRes is compatible with any pretrained matching model where a gridwise feature map is computed, from which the geometric transform, most commonly in a deformation field format, in generated. This includes recent deep learning–based correspondence methods such as LoftUp [ 21 ] , ART [ 29 ] , DIFT [ 55 ] , RoMa [ 12 ] , and Mast3R [ 31 ] . Similarly, any restoration model equipped with an encoder that produces gridwise feature representations and a decoder that reconstructs the restored image is compatible with MatRes . This covers virtually all modern restoration architectures, including general-purpose models such as Restomer [ 66 ] and SwinIR [ 33 ] , as well as task-specific methods for super-resolution, denoising, and deblurring, such as EDSR [ 34 ] , N2N [ 30 ] , N2V [ 26 ] , DeblurGAN [ 27 ] , and SRN [ 56 ] . Figure 2 : Method Overview. Given an input pair (\mathcal{I}_{\texttt{LQ}},\mathcal{I}_{\texttt{HQ}}) , the pretrained generative prior model \mathcal{M}_{\theta} and the pretrained restoration network \mathcal{R}_{\phi} remain frozen during adaptation (gradient flows through both models but their

Read the complete text