Unsupervised 2D-3D lifting of non-rigid objects using local constraints
Shalini Maiti, Lourdes AgapitoBenjamin Graham · 2025
Read Unsupervised 2D-3D lifting of non-rigid objects using local constraints by Shalini Maiti, Lourdes AgapitoBenjamin Graham (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-19227.txt.
Introduction
Shalini Maiti, Lourdes Agapito, and Benjamin Graham reconstruct the three-dimensional shapes of deformable objects from two-dimensional keypoints, without supervision, in work from 2025. The task is ill-posed: flat observations hide depth, and viewpoint changes mix with shape changes. Past models relied on low-rank constraints across the whole shape and canonical alignment steps that are hard to train.
Their approach trains a high-capacity model with an unsupervised loss and applies the low-rank constraint only to local patches of the shape. Local constraints tame the model without forcing a global alignment first. On a standard benchmark of deformable objects, the method cut reconstruction error by over seventy percent against the prior state of the art. The result suggests locality is the key to lifting flexible objects into 3D.
Catalog record
| CREATOR | Shalini Maiti, Lourdes AgapitoBenjamin Graham |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-19227
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unsupervised 2D-3D lifting of non-rigid objects using local constraints Shalini Maiti, Lourdes AgapitoBenjamin Graham Abstract For non-rigid objects, predicting the 3D shape from 2D keypoint observations is ill-posed due to occlusions, and the need to disentangle changes in viewpoint and changes in shape. This challenge has often been addressed by embedding low-rank constraints into specialized models. These models can be hard to train, as they depend on finding a canonical way of aligning observations, before they can learn detailed geometry. These constraints have limited the reconstruction quality. We show that generic, high capacity models, trained with an unsupervised loss, allow for more accurate predicted shapes. In particular, applying low-rank constraints to localized subsets of the full shape allows the high capacity to be suitably constrained. We reduce the state-of-the-art reconstruction error on the S-Up3D dataset by over 70%. Figure 1 : 3D Reconstruction for partially-occluded 2D semantic keypoints. With an orthographic camera model, for visible points ( v=1 ) we know the screen coordinates (x,y) and must predict only the depth z . For occluded points ( v=0 ), we must predict all three (x,y,z) -coordinates. At test time, a trained deep network predicts the unknown coordinates. The network is generic, without any kind of low-rank constraints built-in; the geometric intuition is built into the losses during training, see Figure 2 . For a perspective camera model, the same process can be applied along the camera rays, rather than directly using the x -, y - and z -axes. $*$ $*$ footnotetext: Work done while working at Meta. 1 Introduction Unsupervised 2D-3D lifting of non-rigid objects is the problem of inferring the 3D shape of a deformable object such as animals solely from 2D observations. The solution to this open-ended problem could assist many applications in human-computer interaction, virtual/augmented reality and creative media generation. Early NRSfM techniques often made unrealistic assumptions about the data, for example, that they were restricted to sets of keypoints without occlusions, or to small numbers of keypoints. They were also often forced to use orthographic or weak perspective camera models because of the non-linearity introduced by perspective effects. NRSfM is a fundamentally ill-posed problem and therefore requires the use of various mathematical priors to avoid simply predicting uniform depth. One such prior is the low-rank shape prior [ 4 ] , which assumes that the various shapes can be described using linear combinations of a small set of basis shapes. For temporal sequences, a popular prior is minimizing the change in shape between consecutive frames [ 22 , 15 ] . Figure 2 : Unsupervised losses Starting from a randomly-initialized generic deep network, we iteratively learn from batches of partially-occluded 2D-annotated training samples. Our training use two unsupervised, batch-wise losses. The subset loss, see Section 3.3.1 , acts on the batch of noisy 3D reconstructions. It selects a subset of nearby keypoints, aligns the sub-shapes by rotation and translation, and finally measures the size of the non-rigid motion using the log-product of the singular value decomposition of the residual error matrix, i.e. the log Gramian determinant. This encourages the model to predict body parts that are as consistent as possible. The occlusion loss, see Section 3.3.2 , encourages a weak negative correlation between the binary keypoint-visibility annotations and the predicted depths. This uses the fact that visible keypoints often hide other keypoints because they are closer to the camera. DeepNRSFM [ 29 ] uses block sparsity to restrict the set of linear combinations to lie in a union of low-rank subspaces. C3DPO [ 20 ] assumes that each shape can be associated with a canonical pose, with a low-rank shape basis constraining the range of motion within the canonical frame of reference. A network to predict the canonical pose has to be learnt alongside the shape basis regression network. We take a different approach, using a deep network that lifts 2D observations into the 3D space directly, by ‘inpainting’ information missing due to occlusions and the camera projection. In particular, we find that parameter efficient MLP-Mixer models [ 23 , 19 ] can be trained to generalize well to new observations. To train these models, we introduce a novel unsupervised loss that minimizes the variation in shape of local neighborhoods of the object, after allowing for rotation and translation. This is suitable for objects tracking at least a semi-dense set of keypoints, e.g. around 50-100 keypoints per object. To resolve flip ambiguity in the depth direction, we also use occlusion as an additional learning signal. Due to the nature of self-occlusion, being visible tends to be weakly negatively correlated with depth; points nearer to the camera tend to occlude points that are further away. We apply our method both to isolated observations from a dataset of semantic keypoints, and to videos where arbitrary keypoints are tracked over the length of the video. We find that our method can be applied robustly across a range of different types of datasets, shapes and camera settings. In summary, our contributions are: • We introduce a novel combination of unsupervised loss functions for the 2D-3D lifting of deformable shapes. • We find that MLP-Mixers [ 23 , 19 ] are a parameter efficient alternative to the large MLPs that have been used in previous work on 2D-3D lifting using neural networks. • We achieve state of the art result on the S-Up3D dataset, reducing the reconstruction error by over 70% compared to prior works [ 20 , 28 , 29 , 5 ] . • We show that our method can be also be applied in a one-shot fashion to lift arbitrary videos of tracked keypoints into 3D, with superior performance to baselines [ 20 , 28 , 29 ] . 2 Related Work This section reviews the approaches for the 2D-to-3D lifting problem including classical baselines, parts based methods, and deep-learning based methods trained using unsupervised learning. Classical NRSfM Tomasi and Kanade’s [ 24 ] approached the problem by fixing the rank of the rigid motion as three. Bregler et al. [ 4 ] built on this low-rank assumption and formulated the problem of NRSfM as a linear combination of shape bases. Other works [ 1 , 2 , 7 ] have leveraged the low-rank prior, inspired from Bregler’s factorization. Kumar [ 15 ] argues that properly utilizing smoothness priors can result in competitive performance for video sequences. Low-rank constraints have informed many of the recent works applying deep-learning to NRSfM. Piecewise non-rigid 2D-3D lifting. The idea of piecewise 3D reconstruction has been explored before. [ 17 ] and its extension [ 6 ] treat 3D reconstruction as a consensus sampling problem, by first reconstructing subparts of the object, generating ”weaker” hypotheses followed by optimisation over them to generate the final 3D reconstruction. [ 10 ] is another classical method that divides the surface into overlapping patches, individually reconstructs each patch using a quadratic deformation model and registers them globally by imposing the constraint that points shared by patches must correspond to the same 3D points in space. Our training loss relies on the same intuition as the methods above, that parts of an object can be simpler to model than the whole. However, rather than learning pieces