Multi-View 3D Reconstruction using Knowledge Distillation
Aditya Dutt* Ishikaa Lunawat* Manpreet Kaur* [email protected] [email protected] [email protected] Stanford University * Equal contribution. · 2024
Read Multi-View 3D Reconstruction using Knowledge Distillation by Aditya Dutt* Ishikaa Lunawat* Manpreet Kaur* [email protected] [email protected] [email protected] Stanford University * Equal contribution. (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-02039.txt.
Introduction
Aditya Dutt and co-authors build smaller models that learn to rebuild three-dimensional scenes from photos, in a paper from 2024. A large model called Dust3r can do this well from pairs of images. It is, though, slow and heavy to run, which limits tasks such as finding where a camera sits in a scene.
Their plan uses knowledge distillation: the large model acts as teacher and small student models learn from its outputs. The students are trained for one scene at a time on a known indoor data set. Two designs were tested, one based on classic image networks and one on a vision Transformer. Versions that started from pre-trained weights were compared with ones built from scratch. The vision Transformer design gave the best results, both by eye and by score.
Catalog record
| CREATOR | Aditya Dutt* Ishikaa Lunawat* Manpreet Kaur* [email protected] [email protected] [email protected] Stanford University * Equal contribution. |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-02039
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multi-View 3D Reconstruction using Knowledge Distillation Aditya Dutt* Ishikaa Lunawat* Manpreet Kaur* [email protected] [email protected] [email protected] Stanford University * Equal contribution. Abstract Large Foundation Models like Dust3r can produce high quality outputs such as pointmaps, camera intrinsics, and depth estimation, given stereo-image pairs as input. However, the application of these outputs on tasks like Visual Localization requires a large amount of inference time and compute resources. To address these limitations, in this paper, we propose the use of a knowledge distillation pipeline, where we aim to build a student-teacher model with Dust3r as the teacher and explore multiple architectures of student models that are trained using the 3D reconstructed points output by Dust3r. Our goal is to build student models that can learn scene-specific representations and output 3D points with replicable performance such as Dust3r. The data set we used to train our models is 12Scenes. We test two main architectures of models: a CNN-based architecture and a Vision Transformer-based architecture. For each architecture, we also compare the use of pre-trained models against models built from scratch. We qualitatively compare the reconstructed 3D points output by the student model against Dust3r’s and discuss the various features learned by the student model. We also perform ablation studies on the models through hyperparameter tuning. Overall, we observe that the Vision Transformer presents the best performance visually and quantitatively. 1 Introduction Obtaining multi-view 3D reconstruction from 2D images is a challenging task. Recently, this has been made easier through the use of large foundation models that can be trained on large sets of data. An example of a foundation model is DUSt3R [ 7 ] . DUSt3R aims to solve the multiview reconstruction problem for stereoimage pairs through direct scene coordinate regression without making any prior assumptions on camera intrinsics or extrinsic parameters. The strength of DUSt3R is observed in its applications towards downstream tasks like Visual Localization and 3D Reconstruction. For Visual Localization, the model first performs Pixel Correspondence matching and is then evaluated on two different datasets, performing comparably well on unseen images, despite having not been trained on visual localization tasks at all. However, a lot of processing time is required for these tasks, as the model only works with stereo-image pairs. Moreover, the 3D points are not output in the world reference frame. In order to address these issues, we plan to build a smaller neural network model that learns from the pre-trained foundation model through a knowledge distillation framework and outputs 3D points relative to a fixed world coordinate system. The smaller network will be trained to learn scene-specific knowledge and will be faster and more lightweight than DUSt3R. 2 Related Work 3D Dense Reconstruction Papers such as Accelerated Coordinate Encoding [ 1 ] propose a light-weight and ultra-fast neural network that predicts 3D coordinates for every pixel in an image. This paper establishes an important benchmark for neural network models that can be trained for a specific scene in under 5 mins and can predict a point cloud associated with an image in real-time. While the time to train a scene coordinate network is quite less, training such networks in real-time on mobile devices is often impractical due to lack of compute and high memory consumption. Hence, we look at an advent shift in paradigm brought on by foundation models. The paper introduced by [ 7 ] introduces a model that predicts 3D location corresponding to every pixel in the image without any scene-specific training, giving this method a remarkable ability to generalize to any scene. This paper then extends their sparse point cloud prediction model to solve downstream 3D computer vision tasks such as relative pose estimation, monocular depth estimation, etc and outperforms previous state of the art works. Knowledge-Distillation Knowledge distillation has been widely applied in the field of convolutional neural networks (CNNs) and vision models to enhance their efficiency and performance. In applications such as image classification, object detection, and semantic segmentation, distillation techniques enable the deployment of lightweight student models that maintain high accuracy while reducing computational load . Studies such as [ 3 ] have shown that knowledge distillation can improve the performance of compact models in real-time vision tasks, making it particularly valuable for resource-constrained environments like mobile and embedded devices . Advanced methods like attention transfer by [ 8 ] and intermediate layer guidance by [ 5 ] further refine the distillation process, ensuring that student models capture critical features and patterns from their teachers, thereby optimizing their performance in diverse vision applications 3 Problem Statement To address some of the shortcomings of the DUSt3R model, we plan to implement a smaller neural network that learns scene-specific information and provides generalized 3D reconstructed points expressed in a fixed world coordinate system, which will improve tasks such as Visual Localization. Details about the model architecture and our overall method setup is described in Section 4 . The dataset we will use is the 12Scenes dataset that stores rich scene-specific RGB-D data of 4 large scenes, containing 12 rooms. During training, we will evaluate our model using Mean-Square Loss (MSE) to increase the accuracy of the predicted 3D point locations. With the help of the knowledge distillation framework and data containing scene-specific information, we hope that our network predicts accurate world 3D points. 4 Approach 4.1 Knowledge-Distillation For our problem, we propose a knowledge distillation framework consisting of a teacher and student model. Our student model is designed as a convolutional neural network (CNN) and a standard vision transformer (ViT) that learns from our large teacher model, Dust3R [ 7 ] . The code for our approach is provided here, Github . Our approach is as follows: 1. Dataset Preparation : We use few scenes from the 12Scenes dataset [ 6 ] and create pairwise images with intersecting views to serve as input for the Dust3R model. This involves loading images and pairing them based on overlapping views of the scene. 2. Teacher Model Inference : The Dust3R model generates 3D coordinates for all pairs of images. These coordinates are obtained through an inference process and are used as the ground truth labels for training the Student model. Note that Dust3R provide 3D points in the frame of reference of the first image. 3. Global Alignment Step : To ensure consistency, we perform a global alignment process to align and transform the predicted 3D points to the same frame of reference. This involves fixing an origin point and aligning all predicted points accordingly, and is performed as a post-processing step. 4. Training Student Model : Using the aligned 3D points from the Dust3R model as labels, we train the Student model. The training objective is to minimize the MSE loss between the predicted 3D points from the Student model and the labels provided by the Dust3R Teacher model. As our preliminary test, we design the network as a six layer CNN, where each layer is followed by ReLU. 4.2 Student Models We primary explored two types of model architectures: a CNN-based model and a Vision Transformer based model. For the CNN architecture, we compared the use of a vanilla model structure against the use of an existing pre-trained model that encodes features and