Refine3DNet: Scaling Precision in 3D Object Reconstruction from Multi-View RGB Images using Attention Note: Produces the permission block, and copyright information
Ajith Balakrishnan, Dr Sreeja S, Dr. Linu Shine · 2024
Read Refine3DNet: Scaling Precision in 3D Object Reconstruction from Multi-View RGB Images using Attention Note: Produces the permission block, and copyright information by Ajith Balakrishnan, Dr Sreeja S, Dr. Linu Shine (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00731.txt.
Introduction
Ajith Balakrishnan, Sreeja S, and Linu Shine rebuild 3D objects from ordinary photos with an attention-driven network, in a 2024 paper. Turning flat images into volumes feeds virtual reality, robot vision, and interactive tools.
Their hybrid design runs image features through a self-attention auto-encoder, then a refiner network polishes the rough volume. A two-phase training scheme, which teaches the parts jointly but optimizes them separately, steadies the process. The system accepts one photo or many, from arbitrary angles. On the standard ShapeNet sets it posted the best overlap scores among rivals, beating them by over four points in single-view reconstruction. The authors credit attention for preserving fine shape detail.
Catalog record
| CREATOR | Ajith Balakrishnan, Dr Sreeja S, Dr. Linu Shine |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-00731
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Refine3DNet: Scaling Precision in 3D Object Reconstruction from Multi-View RGB Images using Attention Note: Produces the permission block, and copyright information Ajith Balakrishnan, Dr Sreeja S, Dr. Linu Shine Abstract . Generating 3D models from multi-view 2D RGB images has gained significant attention, extending the capabilities of technologies like Virtual Reality, Robotic Vision, and human-machine interaction. In this paper, we introduce a hybrid strategy combining CNNs and transformers, featuring a visual auto-encoder with self-attention mechanisms and a 3D refiner network, trained using a novel Joint Train Separate Optimization (JTSO) algorithm. Encoded features from unordered inputs are transformed into an enhanced feature map by the self-attention layer, decoded into an initial 3D volume, and further refined. Our network generates 3D voxels from single or multiple 2D images from arbitrary viewpoints. Performance evaluations using the ShapeNet datasets show that our approach, combined with JTSO, outperforms state-of-the-art techniques in single and multi-view 3D reconstruction, achieving the highest mean intersection over union (IOU) scores, surpassing other models by 4.2% in single-view reconstruction. Conference: Indian Conference on Computer Vision Graphics and Image Processing; December 13–15, 2024; Bengaluru, India Indian Conference on Computer Vision Graphics and Image Processing (ICVGIP 2024), December 13–15, 2024, Bengaluru, India DOI: 10.1145/3702250.3702292 ISBN: 979-8-4007-1075-9/24/12 CCS: 3D Model Reconstruction CCS: 3D Model Generative AI CCS: 3D Model Self-Attention 1. Introduction The field of research is becoming more and more interested in reconstructing 3D scenes from 2D images. This is because of the recent availability of large catalogs of 3D models that enable new possibilities for 3D reasoning from photographs. Reconstruction using 3D scans automatically enables quick development of 3D objects and significantly expands the capabilities of immersive technologies like Virtual Reality (VR), Ultrasound (US) ( 22 ) , Robotic Vision,medical research, military applications, Computed Tomography (CT) ( 23 ) , satellite and remote sensing ( 24 ) , Positron Imaging Tomography (PIT) ( 25 ) , Magnetic Resonance Imaging (MRI), and entertainment. In real-world applications, the 2D to 3D conversion techniques that calculate the depth map from 2D scenes for 3D reconstruction offer a viable way to reduce the expense of encoding, sending, and storing 3D visual material. 3D modeling software can be complex and time-consuming for those with little to no prior experience in 3D modeling. Conventional techniques, such as Structure from Motion (SFM) ( 15 ) and Visual Simultaneous Localization and Mapping (vSLAM) ( 14 ) , match image features across views. These methods fail if a large baseline separates the image viewpoints due to changes in local appearance or occlusions within the scene ( 28 ) . The reflecting behavior of the object surface also makes the output 3D model noisier. 3D voxel reconstruction methods, such as space carving, have become popular because they can reconstruct 3D structures even with a large baseline problem. These techniques rely on the assumption that the cameras are calibrated or that the objects are precisely separated from the backdrop, which is not always the case. Recent approaches are based on CNNs or transformers ( 32 ) , each with its advantages and limitations. CNN architectures are efficient in single-view 3D reconstruction, while transformers excel in multi-view reconstruction, feature detailing, and managing memory footprints effectively. CNN feature fusion with unordered input images is a challenging task ( 27 ) . Figure 1. Visualization of Refine3DNet for 3D Object Reconstruction from multiple images. The recent methods, such as 3D-R2N2 ( 3 ) , LSM ( 11 ) , Deep MVS ( 13 ) , and Pix2Vox ( 5 ) , estimate 3D shapes from multiple images with promising results. These methods utilize a shared encoder-decoder mechanism, known as an auto-encoder. 3D-R2N2 ( 3 ) and LSM ( 11 ) employ RNNs to fuse multiple 2D image features extracted by the shared encoder. However, RNN-based methods have three main drawbacks: permutation variance, long-term memory loss ( 31 ) , and lack of parallelization ( 30 ) , which hinder reliable 3D geometry estimation, full exploitation of input images, and increase inference time, respectively. Deep-MVS ( 13 ) uses max pooling to gather deep features but only captures first-order information, leading to a loss of significant image details. 5 ( 5 ) proposed a shared encoder-decoder architecture followed by an optimizer network based on 3D U-Net ( 29 ) and a VGG16-based encoder-decoder architecture. Nonetheless, both networks fail to process small image features. This work proposes a novel network architecture for 3D model reconstruction and a dedicated training algorithm to improve training efficiency. The proposed model architecture is based on a shared CNN encoder-decoder architecture followed by a 3D U-net for feature refinement and a dedicated algorithm named JTSO for training. As shown in Fig- 1 , the basic block diagram of the proposed CNN architecture reconstructs 3D models from 2D images. To address the feature fusion problem, an attention network inspired by the Transformer model ( 20 ) is used for transforming latent space features. The network employs a ResNet18-based autoencoder architecture for feature extraction and object generation. Refiner networks based on 3D U-Net ( 29 ) augment and optimize the 3D structures generated by decoders ( 16 , 5 ) . This network aims to produce highly accurate and scalable 3D representations. We evaluated and compared the network using several state-of-the-art techniques and datasets, demonstrating its efficacy with the specified training algorithm. The following is a summary of our approach’s primary contributions: • We introduce an innovative CNN architecture for 3D model reconstruction capable of generating 3D voxels from one or multiple 2D images captured from various viewpoints. • We introduce an innovative self-attention mechanism designed to effectively aggregate features from unordered images. • We develop a novel 3-stage training algorithm that decouples the network and efficiently updates parameters in distinct stages. • Our comprehensive evaluation using the ShapeNetCore dataset demonstrates that the methods proposed in this study surpass state-of-the-art results, particularly with a small number of input images. This is how the remainder of the paper is structured: A brief history of related work is given in Section 2. Our comprehensive network design is presented in Section 3. The suggested training algorithm and training set are covered in Section 4. At last, we showcase the outcomes of our methodology’s tests on the ShapeNet and online product datasets in Section 5. 2. Related Work Previous research in the field of 3D model reconstruction has explored various methodologies aimed at enhancing the accuracy, efficiency, and inference time of generated 3D model representations. These approaches range from single-view reconstruction to more complex multi-view methods. In this section, we delve into the developments in this area, focusing on the different methodologies and the challenges they address. 2.1. Multi-view 3D Model Reconstruction One classical computer-vision challenge is extracting three-dimensional information from images. Conventional techniques include visual Simultaneous Localization and Mapping (vSLAM) ( 14 ) and Structure from Motion (SfM) ( 15 ) to match manually crafted features across several views and estimate the camera pose for each image. When the input photos are separated by a long baseline, matching becomes very difficult, and these approaches only work well under favorable conditions. In response to the limitations of traditional methods, deep learning approaches have emerged, utilizing extensive datasets like ShapeNet