Activating Self-Attention for Multi-Scene Absolute Pose Regression

Miso Lee, Jihwan Kim, Jae-Pil Heo · 2024

Read Activating Self-Attention for Multi-Scene Absolute Pose Regression by Miso Lee, Jihwan Kim, Jae-Pil Heo (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01443.txt.

Description

Abstract Multi-scene absolute pose regression addresses the demand for fast and memory-efficient camera pose estimation across various real-world environments. Nowadays, transformer-based model has been devised to regress the camera pose directly in multi-scenes. Despite its potential, transformer encoders are underutilized due to the collapsed self-attention map, having low representation capacity.

Catalog record

CREATORMiso Lee, Jihwan Kim, Jae-Pil Heo
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-01443

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Activating Self-Attention for Multi-Scene Absolute Pose Regression Miso Lee, Jihwan Kim, Jae-Pil Heo Abstract Multi-scene absolute pose regression addresses the demand for fast and memory-efficient camera pose estimation across various real-world environments. Nowadays, transformer-based model has been devised to regress the camera pose directly in multi-scenes. Despite its potential, transformer encoders are underutilized due to the collapsed self-attention map, having low representation capacity. This work highlights the problem and investigates it from a new perspective: distortion of query-key embedding space. Based on the statistical analysis, we reveal that queries and keys are mapped in completely different spaces while only a few keys are blended into the query region. This leads to the collapse of the self-attention map as all queries are considered similar to those few keys. Therefore, we propose simple but effective solutions to activate self-attention. Concretely, we present an auxiliary loss that aligns queries and keys, preventing the distortion of query-key space and encouraging the model to find global relations by self-attention. In addition, the fixed sinusoidal positional encoding is adopted instead of undertrained learnable one to reflect appropriate positional clues into the inputs of self-attention. As a result, our approach resolves the aforementioned problem effectively, thus outperforming existing methods in both outdoor and indoor scenes. 1 Introduction Camera pose estimation is a fundamental and essential computer vision task, adopted in numerous applications such as augmented reality and autonomous driving. Geometric pipelines [ 1 , 2 , 3 , 4 , 5 , 6 ] with 2D and 3D data have been a mainstream with high accuracy. After extracting features and matching 2D-3D correspondences, camera pose is approximated via Perspective-n-Points (PnP) algorithm and RANSAC [ 7 ] . However, there still remains several challenges for real-world applications, including high computational cost and a huge amount of 3D point cloud. Absolute Pose Regression (APR) tackles these issues by directly estimating the 6-DoF pose from a single RGB image in an end-to-end manner. First introduced by Kendall et al . [ 8 ] , subsequent APR methods [ 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 ] have been devised based on convolutional neural networks (CNN). However, they still demand multiple models and individual optimization to be applied in real-world multi-scene scenarios. In this regard, Multi-Scene Absolute Pose Regression (MS-APR) has emerged to satisfy the needs of speed and memory efficiency across multiple scenes [ 19 ] . MSTransformer [ 20 ] pioneers a streamlined one-stage MS-APR approach with transformer architecture. It leverages the transformer decoder not only to improve memory efficiency but also to enhance accuracy significantly. However, we point out that the learning capacity of transformer encoders is underutilized. As shown in Tab. 1 , MSTransformer’s encoder self-attention modules do not significantly improve or even degrade performance. Although we discovered low-rank, collapsed attention maps, which are known to cause gradient vanishing [ 21 ] or training instability [ 22 ] , general solutions [ 22 , 23 , 24 ] were not correct the problem in the case of MSTransformer. This challenge motivates us to elucidate the phenomenon from another perspective in the context of MS-APR. Therefore, this work proposes brand new analysis: distortion of query-key embedding space, where the attention score originates. We statistically substantiate that queries and keys are mapped in completely separated regions in the embedding space while only a few keys are blended into the query region. In this situation, attention maps inevitably collapse, as all queries are represented similarly to those few keys. Intuitively, it means that all image patches are just represented by only a few image patches, as illustrated in Fig. 1 . We conjecture that the task difficulty contributes to the issue; the model should extrapolate the camera pose from a single RGB image across multiple scenes. To support this, we empirically reveal that the model tends to avoid exploring self-relationships at the beginning of the training. These findings align with our additional observations that learnable positional embeddings used in self-attention are also undertrained. Nevertheless, we expect that image features incorporating global self-relations will be valuable to estimate the camera pose by capturing salient features such as long edges and corners. This work thus introduces simple but effective solutions that activate encoder self-attention modules for MS-APR. Concretely, we design an auxiliary loss which aligns the query region and the key region. By forcing all queries and keys to be mapped into the close and dense space, they are highly encouraged to interact with each other by self-attention—like putting boxers in the ring and blowing the whistle! Furthermore, we explore various positional encoding methods based on empirical evidence since current undertrained one confuses the model to estimate the camera pose with incorrect positions. Finally, fixed 2D sinusoidal encoding is adopted instead of learnable parameter-based methods [ 25 , 26 ] . This enables the model to consider appropriate positional clues of each image feature during exploring self-relationships from the beginning of the training. As such, the model can obtain rich global relations from activated self-attention as shown in Fig. 1 . These relations are incorporated into image features, acquiring more informative encoder output. Extensive experiments demonstrate that our solution recovers the self-attention successfully by preventing the distortion of query-key space and keeping high capacity of self-attention map [ 22 ] . As a result, our model outperforms existing MS-APR methods in both outdoor [ 8 ] and indoor [ 27 ] scenes without additional memory during inference, upholding the original purpose of MS-APR. Table 1 : Ablation on encoder self-attention modules in MSTransformer [ 20 ] . We report the average of median position and orientation errors for each experiment. Outdoor [ 8 ] Indoor [ 27 ] MST [ 20 ] 1.28\text{m}/$$ 0.18\text{m}/$$ MST w/o encoder SA 1.21\text{m}/$$ 0.18\text{m}/$$ Figure 1 : The figure shows query-key spaces, self-attention maps, and attended keys from the orientation transformer encoder of the baseline and ours, respectively. (a) In the case of the baseline, queries and keys are mapped in separate regions, while only a few keys are blended into the query region. Consequently, the self-attention map collapses and whole image features are represented by meaningless few keys, indicating waste of learning capacity of transformer encoder. (b) However, our solution makes queries and keys interact with each other, activating self-attention. This allows the model to obtain crucial global relations within image features, capturing salient global features. 2 Related Work Absolute Pose Regression (APR) is to estimate 6-DoF camera pose directly using only an RGB image as input. First proposed by Kendall et al . [ 8 ] , subsequent works [ 9 , 10 , 13 , 14 , 17 ] have introduced advanced architectures and training methods. However, Sattler et al . [ 28 ] demonstrated that these models have a tendency to memorize training data, resulting in poor generalization when presented with novel views. In response, IRPNet [ 16 ] attempted fast APR by adopting pre-trained image retrieval model as feature extractor. On the other hand, E-PoseNet [ 15 ] substituted vanilla CNN with Group-Equivalent CNN to extract geometric-aware image features. Recently, NeRF-based models [ 11 , 12 , 29 ]

Read the complete text