SUB-IMAGE RECAPTURE FOR MULTI-VIEW 3D RECONSTRUCTION

Yanwei Wang · 2025

Read SUB-IMAGE RECAPTURE FOR MULTI-VIEW 3D RECONSTRUCTION by Yanwei Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06818.txt.

Introduction

Yanwei Wang, writing alone in 2025, presents a generic trick for rebuilding 3D scenes from very large images without running out of memory. High-resolution inputs are the bottleneck: learning-based reconstruction methods beat older ones in quality, but they demand even more memory, which limits how far they scale to big scenes.

The approach, sub-image recapture, cuts each large image into smaller tiles and processes them one by one. Existing reconstruction algorithms can then run inside the framework unchanged. Memory needs drop sharply while scalability improves, the author reports. The method is framed as a wrapper, so many current and future reconstruction tools can adopt it directly.

Catalog record

CREATORYanwei Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE13 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06818

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SUB-IMAGE RECAPTURE FOR MULTI-VIEW 3D RECONSTRUCTION Yanwei Wang Abstract 3D reconstruction of high-resolution target remains a challenge task due to the large memory required from the large input image size. Recently developed learning based algorithms provide promising reconstruction performance than traditional ones, however, they generally require more memory than the traditional algorithms and facing scalability issue. In this paper, we developed a generic approach, sub-image recapture (SIR), to split large image into smaller sub-images and process them individually. As a result of this framework, the existing 3D reconstruction algorithms can be implemented based on sub-image recapture with significantly reduced memory and substantially improved scalability. † † address: Mixed Reality Microsoft Corporation 1 INTRODUCTION 3D object reconstruction from imagery has been an active research area for quite long time, with tremendous evolutions have been observed. Two main steps in a 3D reconstruction pipeline include a sparse modeling Structure-from-Motion (SfM) step and a dense modeling step Multi-View Stereo (MVS). The SfM undergoes a great deal of transformation and improvement over the years. Started from the early work [ 1 , 2 ] , increasingly large-scale reconstruction systems and variations have been developed [ 3 ] . Readers are referred to [ 4 ] for a survey and implementation of the current SfM algorithm. The MVS algorithm has also been advanced from early traditional MVS [ 5 , 6 , 7 , 8 ] to neural net and learning based algorithms including both stereo [ 9 , 10 , 11 , 12 , 13 , 14 , 15 ] as well as MVS [ 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 ] with improved accuracy and/or efficiency for various datasets. The learning based MVS algorithms hold the potential solution to hard problems such as variation of illumination, lack of texture or non-Lambertian surface, as well as occlusion [ 28 ] . However, these learning based MVS algorithms is known to have the scalability issue due to their high memory consumption. With today’s increasing camera sensor size as well as the super-resolution techniques [ 29 ] , the memory requirement increases geometrically. For example, a 10K by 10K resolution airborne image for 3D terrain reconstruction will use 1.2GB memory assuming floating point resolution for all RGB channels. For such scenario, even two-view stereo will has minimum memory requirement of 2.4GB, without considering the extra cost of any algorithm used. When the input image size is too large, down-sampling of the input image has to be performed to meet the memory requirement. Considerable efforts have been observed in the literature to address this issue [ 26 , 12 , 21 , 22 , 23 , 24 ] . All these works are trying to improve the performance from the algorithm point-of-view, with experimental maximum size between a few hundreds to a couple of thousands in width/height. The framework introduced in this paper address this scalability issue from a data point-of-view, independent of the algorithms used. This solves the existing learning based MVS algorithm’s scalability problem once for all, the existed learning MVS algorithms can be applied to any large image size without down-sampling. 2 SIR: AN UNIVERSAL STRATEGY We develop a sub-image recapture (SIR) approach which can be used to implement existing 3D reconstruction algorithms with reduced memory requirement and improved efficiency. In fact, it can be used for most 3D image processing algorithm which utilizes camera model. The idea of sub-image recapture starts from splitting the original image into a matrix of smaller images which is referred to as sub-images. Unlike down-sampling, each sub-image has a smaller image size than the native image while maintaining the same native spatial resolution of the native image such that there is no loss of details. Each sub-image is then treated as an independent image which is “recaptured” by a corresponding synthesized camera with its own distinct camera parameters. Through this process, for each target pixel or area of interest, we have more “focused” sub-images instead of the original full field of view images to calculate the depth information. The following advantages can be achieved by this approach: 1) significant memory reduction for each MVS step due to the use of sub-images; 2) under the fixed memory limit, in multi-view geometry stage, more images could be used for the same target area hence the improved depth estimation accuracy; 3) for some applications where there are only partial overlaps between adjacent images, sub-image approach can enforce only overlapping part of the sub-images enter the MVS step hence has more efficient use of memory, 4) improved parallel implementation of the MVS algorithm due to the fact that the original image is split into sub-images which can be processed in parallel. 3 SYNTHESIZED RECAPTURE CAMERA 3.1 Definition of Recapture Camera Recapture camera is best explained by dividing a native digital image into a matrix of sub-images. Each sub-image is associated with a distinct synthesized recapture camera which has synthesized intrinsic and extrinsic parameters mapped from a real camera associated with the native digital image. In particular, the synthesized intrinsic parameters for a sub-image’s camera include a principal point which corresponds to the native digital image’s principal point re-defined relative to the sub-image’s coordinate system origin. Re-defining the native principal point relative to the sub-image’s coordinate system origin spatially compensated all the location differences between the sub-image and its native parent, without any error involved. The lens distortion of the native image’s camera will also be carried over to all its sub-images, and hence the undistortion process. By associating each sub-image with a distinct synthesized recapture camera, each sub-image and associated synthesized recapture camera can be treated as a separate, full frame, digital image as if captured by a real camera in its entirety, only with a smaller field of view. Figure 1 (a) illustrates a camera model with 4 \times 4 sub-images. As an example, the synthesized camera of sub-image index (2,1) is shown in Figure 1 (a) and 1 (b). Here O(x_{o},y_{o}) denotes the location of the sub-image (2,1)’s origin, P(x_{p},y_{p}) denotes the native principal point. The synthesized camera for sub-image (2,1) has its principal point defined as P_{(2,1)}(x,y)=(x_{p}-x_{o},y_{p}-y_{o}) (1) Note that the synthesized camera principal point could have a negative location value. So for a native image size of S_{x}\times S_{y} with a sub-image recapture configuration of I\times J , the principal points for each sub-image (i,j) is given by P_{(i,j)}(x,y)=(x_{p}-i\cdot\frac{S_{x}}{I},y_{p}-j\cdot\frac{S_{y}}{J}) (2) where i and j denotes the index of sub-image at horizontal and vertical directions, respectively. The elegance of sub-image recapture is, except for the principal point change, all other intrinsic and extrinsic camera parameters of the synthesized recapture camera associated with each sub-image are kept the same as their corresponding native camera parameters. Note that depending on application, sub-image recapture may be used to “recapture” an arbitrary area and size of the native image, not necessarily on a uniform grid of same sized sub-images. (a) Camera with 4 \times 4 sub-images (b) principal point offset Figure 1: Cameras associated with both the original image and one of the corresponding 4 \times 4 sub-images. 3.2 SIR for SfM and MVS The SIR

Read the complete text