Multiview Image-Based Localization

Cameron Fiore, Hongyi Fan, Benjamin Kimia · 2025

Read Multiview Image-Based Localization by Cameron Fiore, Hongyi Fan, Benjamin Kimia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23577.txt.

Introduction

Cameron Fiore, Hongyi Fan, and Benjamin Kimia locate a camera by matching its photo against a database, without storing any 3D map, in work from 2025. Retrieval-based localization is private, simple, and fast, but its position and angle estimates lag rival methods.

Their hybrid stores only image features, yet reasons with a hidden 3D reconstruction. Two ideas drive it: estimating the camera's center from relative translations alone, ignoring rotations, and computing the pose directly from multi-view matches instead of chained relative poses. On two standard datasets the method improved accuracy over state-of-the-art rivals. It also ran faster and used less memory, keeping the retrieval approach's practical advantages.

Catalog record

CREATORCameron Fiore, Hongyi Fan, Benjamin Kimia
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-23577

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multiview Image-Based Localization Cameron Fiore, Hongyi Fan, Benjamin Kimia Abstract The image retrieval (IR) approach to image localization has distinct advantages to the 3D and the deep learning (DNN) approaches: it is seen-agnostic, simpler to implement and use, has no privacy issues, and is computationally efficient. The main drawback of this approach is relatively poor localization in both position and orientation of the query camera when compared to the competing approaches. This paper represents a hybrid approach that stores only image features in the database like some IR methods, but relies on a latent 3D reconstruction, like 3D methods but without retaining a 3D scene reconstruction. The approach is based on two ideas: (i) a novel proposal where query camera center estimation relies only on relative translation estimates but not relative rotation estimates through a decoupling of the two, and (ii) a shift from computing optimal pose from estimated relative pose to computing optimal pose from multiview correspondences, thus cutting out the “middle-man”. Our approach shows improved performance on the 7-Scenes and Cambridge Landmarks datasets while also improving on timing and memory footprint as compared to state-of-the-art. Fig. -1 : An overview of our approach: the offline stage computes the NetVLAD [ 2 ] descriptors and local SuperPoint [ 10 ] features/descriptors for all database images. The online stage first computes the NetVLAD descriptors and SuperPoint features/descriptors for the query. The query NetVLAD descriptors are then used to retrieve the K=150 most similar images, called anchor images, by a nearest neighbor search in the descriptor space. SuperGlue [ 25 ] is used to detect 2D-2D feature matches between the query and each of the K anchor images. The 5-point algorithm [ 21 ] is used to compute relative pose between the query and each anchor image. A novel approach is used to find the optimal camera center and orientation, which are then used to find multiview feature correspondences. These multiview correspondences are the basis of a latent 3D reconstruction for highly accurate pose estimation. I Introduction Visual localization is a highly-precise, reliable alternative [ 43 , 9 , 4 ] to GPS localization and important to technologies such as self-driving cars, autonomous drones, and augmented reality devices [ 42 , 49 ] . Visual localization is based on a database representation of the environment, e.g. , 3D point clouds [ 27 , 32 ] , images with geo-tags [ 16 , 50 ] , etc , resulting in centimeter-level precision in location and fraction-of-degree precision in orientation in both indoor and outdoor scenes [ 1 , 7 , 38 , 47 ] . There are generally three canonical approaches to visual localization: (i) the image retrieval (IR) approach, (ii) the 3D approach, and (iii) the deep learning approach. Each is described below in turn. The Image Retrieval Approach: The scene is represented by a set of geo-tagged images. The top-K most similar images to the query image are retrieved. This is typically based on some global descriptors, such as NetVLAD [ 2 ] , using nearest neighbor search. In the extreme cases, the pose of the most similar image is used as an estimate of the query pose. In general, this is not accurate and the relative pose of the query images with respect to multiple anchor images is used to estimate the absolute pose of the query image [ 29 , 20 , 47 ] . This process is generally based on using local features with feature matching in a RANSAC scheme to acquire relative pose between the query image and anchor images followed by some form of translation/rotation averaging to find the optimal query pose. This approach is easily amenable to incremental addition of images and can span multiple scenes without additional effort, a clear advantage over scene-specific approaches. Despite its ease of implementation and use, this approach has had the distinct disadvantage of poor localization accuracy. The 3D Approach: In this approach the scene is reconstructed from image features using structure from Motion (SfM) and represented as a 3D point cloud, with each point endowed with feature descriptors [ 31 , 35 , 44 , 19 , 48 ] . Given a query image, its 2D features are matched to the 3D points and the query absolute pose is estimated. The accuracy of this approach is critically dependent on (i) the quality of the 3D point cloud, i.e. , whether there are a sufficient number of features, and (ii) the precision of the 3D point locations. While tremendous progress has been made in reconstructing 3D point clouds [ 35 , 44 ] , this reconstruction is by no means trivial, especially for large–scale scenes when captured by non-experts [ 31 ] : in fact [ 8 ] shows that for head-mounted cameras and indoor scenes, the state-of-the-art 3D reconstruction methods may fail when the scanning is not carefully done. Moreover, such reconstructions require significant computational resources for large-scale scenes so that incremental additions may require recomputing the scene, a challenging constraint for applications such a AR and environment exploration [ 3 ] , or applications with embedded systems [ 14 ] . Finally, 3D point cloud representations face privacy issues since the original image can be largely recovered by 3D points and the corresponding features [ 22 ] . There are variants that attempt to solve this issue, such as 3D lines replacing 3D points [ 36 ] , but in general these face the same challenges. The Deep Learning Approach: A neural net is trained to regress either the relative pose of the query with respect to retrieved images (RPR) or to regress absolute pose (APR). For RPR [ 47 , 1 ] a single image or multiple images similar to the query are retrieved and used to train a neural net that returns the relative pose of the query with respect to the similar images. Using this relative pose, the absolute pose can be calculated with epipolar geometry. Other “end-to-end” neural-net-based techniques learn to regress the absolute pose of the query [ 7 ] . These absolute pose regressors can learn from a variety of different inputs, and in the minimal case, they only need a database of images from the environment labeled with ground-truth poses. Neural-net based models have shown competitive accuracy in recent years, but training them can require huge amounts of time and training samples, and often the trained models cannot generalize well to new scenes [ 15 , 46 , 7 ] . More recent approaches that are scene agnostic do so at the expense of accuracy [ 26 ] . The proposed approach is a hybrid of these methods (see Figure 1). Like IR methods it identifies a series of most similar anchor images and optimally combines relative pose between the query image and anchor images to estimate the absolute pose. However, instead of simply using multiple relative poses to arrive at the optimal absolute pose, the relative pose estimates are removed as ”middle men” and instead, the proposed methods relies on the features directly, as in the 3D approach, but without explicitly retaining a 3D reconstruction. This hybrid approach presents two key novelties: First, it proposes a decoupling between

Read the complete text