Twist and Compute: The Cost of Pose in 3D Generative Diffusion
Kyle Fogarty, Jack Foster, Boqiao Zhang, Jing Yang, Cengiz Öztireli · 2025
Read Twist and Compute: The Cost of Pose in 3D Generative Diffusion by Kyle Fogarty, Jack Foster, Boqiao Zhang, Jing Yang, Cengiz Öztireli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08203.txt.
Introduction
Kyle Fogarty, Jack Foster, Boqiao Zhang, and Jing Yang show that image to three dimensional generative models carry a strong bias toward one standard viewing angle. These models turn a single picture into a three dimensional object, but their hidden preferences are poorly understood. Rotating the simple two dimensional input in controlled tests exposes how fragile viewpoint handling can be.
Performance of the leading Hunyuan3D model degraded noticeably under rotated inputs. A small convolutional network that detects and corrects input orientation restored results without touching the generative backbone. The authors ask whether sheer scale is enough for such models, or whether modular, symmetry aware designs are the better path.
Catalog record
| CREATOR | Kyle Fogarty, Jack Foster, Boqiao Zhang, Jing Yang, Cengiz Öztireli |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-08203
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Twist and Compute: The Cost of Pose in 3D Generative Diffusion Kyle Fogarty, Jack Foster, Boqiao Zhang, Jing Yang, Cengiz Öztireli Abstract Despite their impressive results, large-scale image-to-3D generative models remain opaque in their inductive biases. We identify a significant limitation in image-conditioned 3D generative models: a strong canonical view bias. Through controlled experiments using simple 2D rotations, we show that the state-of-the-art Hunyuan3D 2.0 model can struggle to generalize across viewpoints, with performance degrading under rotated inputs. We show that this failure can be mitigated by a lightweight CNN that detects and corrects input orientation, restoring model performance without modifying the generative backbone. Our findings raise an important open question: Is scale enough, or should we pursue modular, symmetry-aware designs? 1 Introduction Figure 1: Equivariance between world and observation spaces preserves identity. The expressivity of generative models, particularly diffusion architectures, has led to unprecedented success in 3D content creation from single images. These models promise to learn the complex distribution from a mixture of curated 3D training data and image foundation models. However, this success raises a foundational question: What inductive biases are actually learned during training, and how do they impact generalization? In the physical world, objects retain their identity despite changes in position and orientation. However, when these objects are projected into an observation space, such as an image, transformations result in structured variations in appearance. A model that genuinely understands 3D structure should therefore maintain object identity across these transformations, exhibiting equivariance between the 3D world space and its representation in the observation space. Established works argue that symmetry should be encoded rather than merely discovered: group-equivariant CNNs enforce rotation/reflection structure in 2D 2 , and SE(3)-equivariant networks extend this to 3D representations 11 ; 3 . In contrast, popular image-to-3D pipelines often inherit dataset view biases (e.g., canonical front views), which can encourage shortcut solutions over true geometry 1 ; 6 . Recent efforts try to counter this bias through two main strategies: fine-tuning models on orientation-aligned data to directly produce canonical outputs 8 , or normalizing the input pose before reconstruction. The viability of the latter is supported by foundational work showing that simple networks can effectively predict 2D image rotation 4 . Figure 2: Rotating the input image breaks the image-to-3D pipeline (canonical-view bias), while a lightweight CNN predicts the rotation and applies its inverse to re-canonicalize the image before 3D generation, restoring performance. Note that the degraded 3D mesh is rotated to the canonical frame for better visualization. We position our study within this landscape, using in-plane rotations of the input image as a controlled probe of whether the Hunyuan3D 2.0 image-to-3D pipeline 15 , a current state-of-the-art model, exhibits genuine equivariance or instead relies upon on canonical views. A model that internalizes 3D structure should be equivariant to such transformations, maintaining object identity as the observation rotates. However, the prevalence of canonical orientations in large-scale image corpora may incentivize shortcut learning, yielding a preference for canonical views rather than robust 3D geometry. We refer to this as canonical-view bias , which limits generalization to arbitrarily oriented inputs. Contributions This work investigates the impact of viewpoint biases in image-to-3D generation. (i) We empirically identify a strong canonical view bias in the Hunyuan3D generative model, particularly for object categories with well-defined orientations, such as airplanes, chairs, and cars. (ii) We demonstrate that this bias can significantly degrade 3D generation quality when input images are rotated away from their canonical poses. (iii) We show that a lightweight CNN-based pre-processing module, trained to detect and correct image orientation, can effectively restore generation quality, without requiring any changes to the generative model itself. 2 Methodology 2.1 Image-to-3D Generative Model Our investigation focuses on Hunyuan3D 15 , a state-of-the-art flow-matching architecture for single-image 3D generation. The model adopts a decoupled pipeline, separating 3D representation learning from the generative process. To obtain the 3D representation, high-level semantic features are extracted from the input image using a frozen DINOv2-based encoder 9 . These features are projected into a latent space as vector sets 14 , which serve as implicit representations of complex 3D shapes. The generative component is a flow-matching diffusion transformer trained in this latent space to predict object token sequences from the input image. These token sequences are decoded into Signed Distance Functions (SDFs), which are subsequently converted into triangle meshes via iso-surfacing 7 . This architecture implies that the model’s synthesis quality is bottlenecked by the feature fidelity of its 2D encoder. If the DINOv2 encoder exhibits a bias toward canonical object views, it may produce distorted feature representations when processing rotated inputs. 2.2 Probing for Canonical View Bias Curated Dataset. To investigate the model’s sensitivity to input orientation, we constructed a targeted evaluation dataset comprising three object categories with well-established canonical poses: airplanes, chairs, and cars. Images were sourced from publicly available repositories and manually curated to ensure the presence of a single, dominant object in each image, minimal occlusion, and sufficient diversity across object instances. Rotational Transformations. To systematically evaluate robustness to viewpoint variation, we apply a set of in-plane 2D rotations to each image in the dataset. Specifically, each source image I is rotated by angles \theta\in\{0^{\circ},90^{\circ},180^{\circ},270^{\circ}\} , producing a set of transformed inputs \{I^{\prime}_{0},I^{\prime}_{90},I^{\prime}_{180},I^{\prime}_{270}\} . Here, 0^{\circ} denotes the original, canonical orientation. Evaluation Metric: Cross-Modal Similarity (ULIP). To quantitatively assess the semantic fidelity of the generated 3D shapes, we employ the ULIP (Unified Language-Image Pre-training) score 13 , which enables direct comparison between modalities by embedding 2D images and 3D shapes into a shared semantic space. In our setup, each generated mesh is converted into a point cloud of 8,192 points, and 3D features are extracted using a pretrained Point-BERT variant integrated within the ULIP framework. The ULIP score is computed as the cosine similarity between the image and point cloud embeddings, with higher scores indicating stronger semantic alignment between the input image and the generated 3D shape. Evaluation Procedure. For each rotated image I^{\prime}_{\theta} , we use the Hunyuan3D model to generate a corresponding 3D shape M^{\prime}_{\theta} , and compute the associated ULIP score: \mathcal{S}_{\theta}=\text{ULIP}(I^{\prime}_{\theta},M^{\prime}_{\theta}). By comparing scores across rotation angles, we assess the model’s robustness to input orientation and identify any performance degradation when the input deviates from the canonical view. 2.3 A Lightweight CNN-based Orientation Corrector To mitigate orientation bias, we introduce a lightweight pre-processing module: a compact orientation classifier based on EfficientNetV2, equipped with a 4-way softmax output. We provide more information on the orientator in Appendix A . Given an input image I , the classifier produces logits \mathbf{z}(I)\in\mathbb{R}^{4} , corresponding to discrete rotation angles \{0^{\circ},90^{\circ},180^{\circ},270^{\circ}\} . The predicted rotation r^{\star} is obtained by r^{\star}=\arg\max_{k}z_{k}(I), after which the image is rotated back to a canonical orientation. We employ publicly available pretrained weights for the classifier without any additional fine-tuning. With only 20.3M parameters, negligible compared to the 2.8B parameters of the generative model, the classifier achieves 96.2% accuracy in our evaluations. 3 Results and Analysis 3.1 Canonical-View Bias in Hunyuan3D 2.0 We observe a consistent decline in ULIP similarity scores as the input image is rotated away from the canonical 0^{\circ} orientation across all evaluated categories: airplanes , chairs , and cars . In each case, the