Image-Guided Shape-from-Template Using Mesh Inextensibility Constraints
Thuy Tran, Ruochen Chen, Shaifali Parashar · 2025
Read Image-Guided Shape-from-Template Using Mesh Inextensibility Constraints by Thuy Tran, Ruochen Chen, Shaifali Parashar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22699.txt.
Introduction
Thuy Tran, Ruochen Chen, and Shaifali Parashar present a fast method for rebuilding deforming shapes from images in this 2025 paper. Shape from Template starts with a known 3D template and bends it to match video frames. Older methods need point matches between image and template, which fail when parts are hidden. Newer methods need huge training data.
Their method uses only image cues such as color, edges, and silhouettes. A mesh constraint keeps the surface from stretching, as real cloth and skin resist stretch. The result runs about 400 times faster than the best prior unsupervised method. It also captures finer detail and handles heavy occlusion better. Code was released with the paper.
Catalog record
| CREATOR | Thuy Tran, Ruochen Chen, Shaifali Parashar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22699
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Image-Guided Shape-from-Template Using Mesh Inextensibility Constraints Thuy Tran, Ruochen Chen, Shaifali Parashar Abstract Shape-from-Template (SfT) refers to the class of methods that reconstruct the 3D shape of a deforming object from images/videos using a 3D template. Traditional SfT methods require point correspondences between images and the texture of the 3D template in order to reconstruct 3D shapes from images/videos in real time. Their performance severely degrades when encountered with severe occlusions in the images because of the unavailability of correspondences. In contrast, modern SfT methods use a correspondence-free approach by incorporating deep neural networks to reconstruct 3D objects, thus requiring huge amounts of data for supervision. Recent advances use a fully unsupervised or self-supervised approach by combining differentiable physics and graphics to deform 3D template to match input images. In this paper, we propose an unsupervised SfT which uses only image observations: color features, gradients and silhouettes along with a mesh inextensibility constraint to reconstruct at a 400 × 400\times faster pace than (best-performing) unsupervised SfT. Moreover, when it comes to generating finer details and severe occlusions, our method outperforms the existing methodologies by a large margin. Code is available at https://github.com/dvttran/nsft . 1 Introduction Figure 1 : Image-guided SfT vs SoTA. Our method recovers finer details, resolves self-occlusions and handles strong perspectivities better than existing methods. We evaluated traditional SfT [ 3 ] with correspondences obtained from Cotracker v3 [ 18 ] . Recovering 3D shapes of objects from monocular RGB images, obtained through a video sequence or wide-baseline viewpoints, is a key goal in 3D computer vision. Given a 3D template of an object, comprising a known geometric configuration of the object along with known texture details, [ 3 , 44 ] established that a unique 3D shape can be recovered from a single image of an object in a configuration different from the 3D template. It showed that combining projective geometry of the calibrated images with isometric (or geodesic-preserving) constraints on the 3D template deformation, the underlying 3D structure of any point in the image can be uniquely identified merely from the point-wise registration between the input image and the texture map associated with the 3D template. Template-based 3D reconstruction from images, a.k.a SfT, is therefore proven to be a well-posed problem with reliable solutions [ 8 ] that can perform real-time 3D reconstructions [ 9 , 32 , 47 , 50 ] . [ 31 ] extends the traditional SfT to adapt to handle occlusions by interpolating the non-corresponding points on images from the reconstructed 3D points of the relevant neighbors. However, all these methods fail when the deforming object causes large (self-)occlusions, sharp motion, and high perspectivities in the images, such as in the case of the deforming cloth shown in Figure 1 . This is because a unique solution of 3D shape is possible only if the point correspondences between the template texture map and the input images are known. Such occlusion artifacts appear quite often while capturing deforming objects; thus the inefficacy of traditional SfT to process them poses a severe limitation to their applicability. Sharp motions and strong perspectivities also significantly degrade the reliability of point correspondences which causes these methods to fail. To mitigate this issue, several modern SfT methods [ 10 , 11 , 37 , 12 , 48 , 25 ] use deep neural networks (DNN) that can jointly perform template-image registration and 3D reconstruction. These methods may require a large amount of data for supervision and/or several restrictions on the scene and camera. However, they fail to recover finer details and severe occlusions. Alternatively, \phi -SfT [ 17 ] proposed to take advantage of the physics-based simulation of thin-shell objects [ 22 , 30 ] to deform a 3D template and render using differentiable graphics to match input images in a complete unsupervised fashion. Although successful in handling fine details and severe occlusions to some extent, it incurs a huge computational cost, which makes it practically limited. Recently, PGSfT [ 49 ] used self-supervised learning of physics-based thin-shell simulations [ 45 , 6 ] to learn a neural cloth model which allowed a 400\times speedup over [ 17 ] , although with a degraded performance in handling severe occlusions and generating fine details. In this paper, like [ 17 ] , we present a correspondence-free, unsupervised SfT technique to reconstruct 3D shapes. However, instead of using physics-based simulation of thin shell objects, we rely on simple image-related measures only to guide the 3D template mesh deformations. Consequently, we achieve a 400\times computational speed up with a significantly improved performance than [ 17 ] , especially while recovering finer details and challenging motion. Moreover, we initialize the simulation of a given video frame with its previous one, allowing a simpler and faster prediction of the current shape. Furthermore, we enforce an inextensibility measure of the template mesh rather than a strict isometric deformation [ 3 ] to allow the template to change its geometry according to the deformations observed in the images. Such a non-strict measure allows us to deal with isometric objects such as a paper and elastic objects such as clothes equally well. Our experiments show that the proposed method outperforms traditional and modern methodologies by large margins, especially while reconstructing finer details and severe occlusions. 2 Related Works Traditional SfT methods. Given the registration between images and the texture map of the 3D template, these methods reconstruct surfaces as seen in input images assuming certain constraints on the deformations. [ 44 , 43 , 36 , 4 ] assume that the template is represented with a mesh and enforce inextensibility constraints on its deformations to reconstruct the surfaces. [ 3 , 8 ] assume a continuous representation of the template and enforce strict isometric constraints on its deformations. The reconstructions are obtained as a solution to a Partial Differential Equation. These methodologies have been extended to other deformation models such as conformality [ 3 ] , equiareality [ 5 , 34 ] , ARAP [ 33 ] , smoothness [ 55 ] and elasticity [ 27 , 28 , 26 , 14 , 1 ] in order to incorporate more realistic deformation scenarios. These methods are mostly computationally efficient and perform well in generic scenarios, but they are prone to high errors when encountered with strong deformations, severe occlusions, and lighting changes in the images. Under these conditions, the point correspondences obtained from the registration are highly inaccurate which contributes strongly to the degradation in their performance. [ 23 , 32 ] attempted to jointly estimate the shape and registration in order to overcome these issues; however, they could not achieve any significant performance improvement. DNN-based SfT methods. [ 37 , 12 ] consider the template to be a flat, regular mesh and estimate the displacement of 3D vertices to comply with observed images using an encoder-decoder network. These methods learn object-specific models that rely on both registration and reconstructions as ground truth for the supervision. Such data is difficult to obtain, and the learned models are limited to the images seen in training scenarios for which registration could be computed. Therefore, these methods do not generalize well and do not handle complex deformation scenarios. [