Deep Non-rigid Structure-from-Motion Revisited: Canonicalization and Sequence Modeling -Appendix-
Hui Deng, Jiawei Shi, Zhen Qin, Yiran Zhong, Yuchao Dai · 2024
Read Deep Non-rigid Structure-from-Motion Revisited: Canonicalization and Sequence Modeling -Appendix- by Hui Deng, Jiawei Shi, Zhen Qin, Yiran Zhong, Yuchao Dai (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-07230.txt.
Introduction
Hui Deng and colleagues rebuilt deep reconstruction of deforming three-dimensional shapes in a 2024 paper. Non-rigid structure from motion takes a two-dimensional video and infers the three-dimensional sequence behind it, a classic problem made harder by motion ambiguity: many shapes can project to the same frames. Deep methods advanced the field but handled sequences awkwardly and canonicalized each dataset as a whole, baking in its biases.
Their fix canonicalizes each sequence on its own, a simple change with wide effect, and pairs it with sequence modeling that respects time order and a subspace constraint on the deformation. Tested on common benchmarks, the pipeline with its regularization modules beat prior pipelines. Per-sequence treatment respects that every performance deforms in its own way.
Catalog record
| CREATOR | Hui Deng, Jiawei Shi, Zhen Qin, Yiran Zhong, Yuchao Dai |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 12 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-07230
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Deep Non-rigid Structure-from-Motion Revisited: Canonicalization and Sequence Modeling -Appendix- Hui Deng, Jiawei Shi, Zhen Qin, Yiran Zhong, Yuchao Dai Abstract In this supplementary material, we provide additional discussions on the use of the context layer designed in the main text and the experiment result in the main text. Furthermore, we add more visualization of results in the form of figures and video to the experiments. Temporal information of sequence In traditional Non-Rigid Structure-from-Motion (NRSfM) methods, the 2D observation sequence is generally taken as a whole to solve for the corresponding 3D shape sequence. In the deep NRSfM methods, however, there is few such sequence-to-sequence modeling approach, and a single-frame lifting paradigm ( i.e . , solving for each 3D shape individually from a 2D frame) is usually used: \displaystyle\mathbf{S}_{i}=f_{s}(f(\mathbf{W}_{i})),\mathbf{S}_{i}\in\mathbb{R}^{3\times P},\mathbf{W}_{i}\in\mathbb{R}^{3\times P}, (1) \displaystyle\mathbf{R}=f_{R}(f(\mathbf{W}_{i})),\mathbf{R}\in\text{SO}(3), where \mathbf{S}_{i} is a 3D shape in the sequence, and similarly \mathbf{W}_{i} is the corresponding 2D observation frame. Under the above 3D reconstruction paradigm, the following loss functions are also used as self-supervised signal sources: \displaystyle\mathcal{L}=f_{data}(\mathbf{R},\mathbf{S},\mathbf{W})+g(\mathbf{S}), (2) where f_{data}(\cdot) is a data term, it constrains the output results that should be consistent with the modeling with the input data to produce direct supervision. The most widely data term is the reprojection error \|\mathbf{W}-\mathbf{RS}\| . To prevent the data terms from guiding the optimization in the wrong direction, extra regularization terms such as g(\cdot) are added to further narrow down the solution space, for example, the caononicalization loss used by C3dpo ( Novotny et al. 2019 ) . Modeling a sequence of shapes is not the same as modeling a collection of shapes. Specifically, a sequence can be viewed as a set with direction and length, so both set and temporal information (direction and length) should be taken into account when constructing the constraints. The current methods that take sequences as input objects rarely consider this issue. These methods ( Park, Lee, and Kwak 2020 ; Zeng et al. 2021 ) usually take the sequence as input and subsequently construct a loss function between the members of the sequence using constraints such as smoothing or alignment. However, these constraints cannot distinguish whether the sequence has direction and distance, and the temporal information is not used. Similarly, in ( Deng et al. 2022 ) , even though the sequence order is not modeled specifically, the temporal information is added as a bias. However, such bias doesn’t work too well for injecting temporal information. This is because without using downstream constraints to train such a context layer, the network is unable to recognize what exactly the bias is doing. On the other hand, this is also determined from the computational process of the context layer. First of all, the calculation before output of the context layer can be simply described as follows: \displaystyle\mathbf{X}_{o}=(\mathbf{CX}+\mathbf{\tau})\mathbf{W}_{v}=\mathbf{C(X+\Gamma)W}_{v}. (3) \mathbf{C} is a weight matrix used to reconstruct the sequence, which has a relationship with the sequence as well as the temporal encoding. \mathbf{\tau} is a bias that contains the temporal encoding with learnable parameters and \mathbf{W}_{v} is an output linear layer. We found that the sequence reconstruction task can actually be done by the first term only, where the bias term would be of no direct use for the representation of sequence information. The bias term contains the sequence temporal encoding and learnable parameters. The loss function puts a constraint on the output of the sum of the two terms, but not specifically on the second term. This makes it uncertain what effect the second term can actually have, and it is not possible to determine whether this approach has a positive or negative effect on the outcome, making the computational process unpredictable in terms of results. In contrast, this bias term is absent in the calculation process of this paper, and all temporal information is only included in the weight matrix, avoiding this problem without adding an additional loss function. Figure 1: We have given a detailed description of the network modules in the main text, here we give a more intuitive diagram of the network modules to help the reader better understand our network structure. Network architecture The architecture of our network modules used in main text. As shown in 1 , the encoder f(\cdot) contains six res-layers. The Single-Frame Predictor and the Rotation Layer utilize the decomposition network structure proposed in C3dpo ( Novotny et al. 2019 ) . More precisely, shape predictor comprises of two phases: the first one estimates the weights necessary for making linear combinations, while the second one computes the shapes from the weights. To ensure a computation process similar to that of the traditional factorization framework - which is linear, the structure employs purely linear layers for forming these two stage. For the same reasons, rotation layer employs a linear layer that produces a rotation vector and then uses the Rodriguez formula to convert it into a rotation matrix. The feature space mapping network module g is a one-layer linear layer, the input dimension is 3\times P , the output dimension is 128. The shape space remapping network g_{s} contains a gated linear unit ( Shazeer 2020 ) and a four-layer MLP. \mathcal{H} contains a gated Toeplitz unit (Gtu) as ( Qin et al. 2023 ) . C3dpo DNRSfM MHR Seq2Seq Ours Kinect 0.191 0.286 1.15 0.101 0.166 Rug 0.317 0.351 0.306 - 0.193 Table 1: Results on the short sequences dataset. We report the e_{3D} metric on this dataset. Due to the small size of this dataset, which is not well suited for data-driven deep learning methods to fully demonstrate the advantages of data-driven, the results of traditional methods are not added to the table, and are only compared in several relevant Deep NRSfM. Seq2Seq doesn’t work on Rug, we mark it as ’-’. Discussion Seq2Seq Seq2Seq with GPA Ours CMU-S23-All 0.137 0.209 0.168 CMU-S34-All 0.142 0.232 0.201 H36M 79.8 75.6 66.1 Table 2: We show the results of the ablation experiments, where Seq2Seq with GPA represents the results of removing the canonical loss in the GPA Layer based on Seq2Seq. This shows the limitation of GPA Layer on datasets with shorter sequence length. Table 3: Performance on H36M of different pipeline structures. We only show the best result for other methods and our original strategy. Weight 0.1 0.4 1 1.2 1.5 1.8 MPJPE \downarrow MPJPE \downarrow MPJPE \downarrow MPJPE \downarrow MPJPE \downarrow MPJPE \downarrow C3dpo - - 95.6 - - - DNRSfM - - 109.9 - - - PAUL - - 88.3 - - - Independent Loss 97.1 119.4 112.5 89.8 94.8 94.5 Frozen Predictor - - 91.3 - - - Origin - - 72.5 - - - The difference in results between Table 1 and Table 2 of main text highlights the advantages of our GPA layer on longer sequence data and the limitations on slightly shorter sequence data. In this section, we discuss the limits and scope of different alignment regularization terms. We begin this discussion by attempting to answer why Table 1 and Table 2 of main text do not conform over comparing methods. The canonicalization loss used by ( Novotny et al. 2019 ) is an implicit regularization term that obtains alignment results over the entire dataset. This approach is more