Muskie: Mu lti-view Ma sk ed I mag e Modeling for 3D Vision Pre-training

Wenyu Li, Sidun Liu, Peng Qiao, Yong DouTongrui HuNational University of Defense Technology · 2025

Read Muskie: Mu lti-view Ma sk ed I mag e Modeling for 3D Vision Pre-training by Wenyu Li, Sidun Liu, Peng Qiao, Yong DouTongrui HuNational University of Defense Technology (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-18115.txt.

Introduction

Wenyu Li, Sidun Liu, Peng Qiao, Yong Dou, and Tongrui Hu present Muskie, a vision backbone pre trained natively on multiple views at once for three dimensional tasks. Existing backbones process frames one by one and agree poorly across views, while three dimensional understanding needs consistency between viewpoints of the same scene.

Muskie learns by reconstructing heavily masked content in one view using geometric correspondences found in the other views, with an aggressive masking strategy. No three dimensional supervision is used, yet the model develops strong geometric understanding and beats frame wise backbones on correspondence accuracy. As a backbone it improves camera pose estimation and point map reconstruction.

Catalog record

CREATORWenyu Li, Sidun Liu, Peng Qiao, Yong DouTongrui HuNational University of Defense Technology
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-18115

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Muskie: Mu lti-view Ma sk ed I mag e Modeling for 3D Vision Pre-training Wenyu Li, Sidun Liu, Peng Qiao, Yong DouTongrui HuNational University of Defense Technology Abstract We present Muskie, a native multi-view vision backbone designed for 3D vision tasks. Unlike existing models, which are frame-wise and exhibit limited multi-view consistency, Muskie is designed to process multiple views simultaneously and introduce multi-view consistency in pre-training stage. Muskie is trained to reconstruct heavily masked content in one view by finding and utilizing geometric correspondences from other views. Through this pretext task and our proposed aggressive masking strategy, the model implicitly to learn view-invariant features and develop strong geometric understanding without any 3D supervision. Compared with state-of-the-art frame-wise backbones such as DINO, Muskie achieves higher multi-view correspondence accuracy. Furthermore, we demonstrate that using Muskie as a backbone consistently enhances performance on downstream 3D tasks, including camera pose estimation and pointmap reconstruction. Codes are publicly available at https://leo-frank.github.io/Muskie/ . Figure 1 : Muskie is a native multi-view visual backbone designed for 3D vision tasks. Through Multi-view Masked Image Modeling pre-training, it learns to jointly extract representations across multiple views in a single forward pass. This is in contrast to the conventional frame-wise paradigm, where a ViT independently encodes each view before features are fused. Muskie establishes stronger multi-view consistency, demonstrated by predicted tracks (red) that align closely with the ground truth (green). Using Muskie as a backbone also leads to superior performance in applications like pointmap estimation, where it produces coherent and geometrically complete 3D reconstructions. † † footnotetext: † Equal Contribution ∗ Corresponding Author 1 Introduction Reconstructing and understanding the 3D world from 2D images is one of the long-standing and fundamental goals in computer vision 37 ; 38 ; 28 ; 29 . A prevailing practice in this domain is to leverage Vision Foundation Models (VFMs), such as the DINO series 23 ; 32 , as powerful feature extractors for downstream 3D tasks like 3D reconstruction and camera pose estimation. These VFMs, pre-trained on massive-scale 2D image datasets, have achieved remarkable success and become the de-facto backbones for many modern approaches 40 ; 55 ; 44 ; 17 ; 9 . However, recent studies 15 ; 11 revealed that existing VFMs exhibit limited 3D understanding, notably a lack of multi-view consistency , which requires the feature representations to be consistent across different viewpoints, as shown in Fig. 1 . This capability is a core requirement for 3D tasks as it enables the accurate aggregation of information across views. While prior efforts have sought to instill 3D awareness into VFMs by leveraging supervisory signals from depth maps 15 or camera poses 41 , their reliance on 3D annotations limits scalability and generalizability. We posit that the key to achieving robust multi-view consistency lies in addressing it directly during the self-supervised pre-training phase. This insight stems from a core limitation in existing VFMs: their frame-wise pre-training paradigm. This paradigm operates only on single-view images, providing no incentive for the model to learn cross-view relationships. Consequently, when these models are applied to multi-view settings, they process each image independently and fail to establish consistency. To this end, we propose Muskie to instill multi-view consistency during pre-training by introducing the multi-view completion task, which extends Masked Image Modeling (MIM) 13 ; 47 to the multi-view domain. Given multi-view observations as input, the core idea of this pretext task is to reconstruct masked regions in one view by leveraging the visible content from other available views. This pretext task implicitly compels the model to discover geometric correspondences, resulting in view-consistent features without explicit 3D annotations. To make the pre-training scheme effective, we first need to prevent the model from taking shortcuts—that is, reconstructing masked regions using only intra-view cues like conventional MIM methods. We address this by proposing an aggressive masking strategy with high masking ratios and spatially concentrated mask blocks. By masking large and contiguous blocks, this method makes single-view reconstruction nearly impossible, thus compelling the model to seek correspondences across views. Furthermore, to process multi-view inputs, we incorporate Alternating Attention 40 , which facilitates efficient information exchange both intra-view and inter-view. We pre-train Muskie on large-scale, unlabeled multi-view image collections. The trained model demonstrates superior multi-view correspondence accuracy over state-of-the-art visual backbones, such as DINO family 23 ; 32 and MAE 13 . We further evaluate it as a backbone within end-to-end 3D reconstruction frameworks 44 for pointmap and camera pose estimation, observing consistent improvements across these downstream tasks. The results show that our multi-view pre-training effectively builds geometric understanding into visual features, enabling 3D-aware representations. In summary, our contributions are three-fold: • We introduce Muskie, a novel pre-training framework that learns 3D-aware representations from multi-view images without any 3D supervision. • We show that Muskie produces superior geometric consistency, demonstrating that our pre-training effectively equips the model with geometric reasoning capability. • We demonstrate that using Muskie as a feature extractor enhances the performance of downstream 3D tasks, outperforming state-of-the-art backbones that are restricted to single-image pre-training. Figure 2 : Overview of Muskie architecture. Multi-view images are divided into patches, and a portion of them is masked using various masking shapes (random, rectangular, or elliptical), replaced with learnable tokens. A subset of views is kept unmasked to serve as reference for others. These patches are jointly processed via stacked alternating-attention blocks 40 . A lightweight linear head reconstructs the masked patches along with confidence maps. For comparison, MAE 13 performs Masked Image Modeling (MIM) in a single-view setting, while CroCo 45 ; 46 extends MIM to dual views but still encodes each view independently during the encoding stage. 2 Related Works Self-supervised learning from 2D images. The rise of Vision Foundation Models (VFMs), trained via self-supervised learning on vast 2D image datasets without annotations, has marked a paradigm shift in computer vision. These models learn powerful general-purpose representations typically through one of two dominant approaches: contrastive learning 6 ; 23 ; 32 ; 14 ; 12 ; 7 , which learns by leveraging discriminative signals between images, or masked image modeling 13 ; 36 ; 47 , inspired by the masked token prediction task in BERT 10 . Both approaches excel at learning robust dense features, such as semantic object boundaries, leading to high performance on downstream tasks such as segmentation 1 ; 20 and depth estimation 49 ; 50 ; 42 . However, these methods are trained exclusively on unstructured 2D image collections, thus lacking knowledge of the underlying 3D structure. Incorporating 3D Awareness into VFMs. To enhance the 3D understanding of VFMs, a key research direction is to instill geometric awareness during self-supervised pre-training. Early approaches focused on incorporating explicit 3D priors into contrastive learning frameworks. For instance, Pri3D 15 enforced multiview consistency using geometric correlations from RGB-D scans, while other works leveraged publicly available CAD models 3 or known relative camera poses 41 . However, the reliance on such annotated 3D data limit the scalability and performance gains. A more recent line of work, CroCo 45 ; 46 , adapts the masked image modeling strategy to stereo data to learn 3D priors. However, this approach is limited by its pair-wise input structure, which constrains its applicability to stereo-only tasks.

Read the complete text