RayZer: A Self-supervised Large View Synthesis Model
Hanwen Jiang Hao Tan Peng Wang Haian Jin Yue Zhao Sai Bi, Kai Zhang Fujun Luan Kalyan Sunkavalli Qixing Huang Georgios Pavlakos · 2025
Read RayZer: A Self-supervised Large View Synthesis Model by Hanwen Jiang Hao Tan Peng Wang Haian Jin Yue Zhao Sai Bi, Kai Zhang Fujun Luan Kalyan Sunkavalli Qixing Huang Georgios Pavlakos (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-00702.txt.
Introduction
Hanwen Jiang and co-authors teach a model 3D awareness from plain photos alone in this 2025 paper. View synthesis tools usually need known camera positions and scene geometry. Their model, RayZer, gets neither. It takes unposed, uncalibrated images, guesses the cameras itself, builds a scene code, and renders new viewpoints.
Training leans only on those self-guessed poses, so ordinary photo sets suffice. Two design choices drive the 3D sense: an encoding that splits camera from scene, and a transformer whose only built-in geometry is the ray linking them. RayZer matched or beat rival methods that cheat with true poses. The authors release it as a step toward 3D learning without labels.
Catalog record
| CREATOR | Hanwen Jiang Hao Tan Peng Wang Haian Jin Yue Zhao Sai Bi, Kai Zhang Fujun Luan Kalyan Sunkavalli Qixing Huang Georgios Pavlakos |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-00702
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RayZer: A Self-supervised Large View Synthesis Model Hanwen Jiang Hao Tan Peng Wang Haian Jin Yue Zhao Sai Bi, Kai Zhang Fujun Luan Kalyan Sunkavalli Qixing Huang Georgios Pavlakos Abstract We present RayZer, a self-supervised multi-view 3D Vision model trained without any 3D supervision, i.e., camera poses and scene geometry, while exhibiting emerging 3D awareness. Concretely, RayZer takes unposed and uncalibrated images as input, recovers camera parameters, reconstructs a scene representation, and synthesizes novel views. During training, RayZer relies solely on its self-predicted camera poses to render target views, eliminating the need for any ground-truth camera annotations and allowing RayZer to be trained with 2D image supervision. The emerging 3D awareness of RayZer is attributed to two key factors. First, we design a self-supervised framework, which achieves 3D-aware auto-encoding of input images by disentangling camera and scene representations. Second, we design a transformer-based model in which the only 3D prior is the ray structure, connecting camera, pixel, and scene simultaneously. RayZer demonstrates comparable or even superior novel view synthesis performance than “oracle” methods that rely on pose annotations in both training and testing. Project: https://hwjiang1510.github.io/RayZer/ Figure 1 : We propose RayZer , a self-supervised multi-view 3D Vision model trained on unlabeled data without any annotations, e.g . , camera pose labels. At inference, RayZer supports feed-forward novel view synthesis from unposed & uncalibrated images. RayZer achieves novel view synthesis performance comparable to that of supervised “oracle” methods (GS-LRM and LVSM), which require camera labels in both training and inference, and even outperforms them when they rely on (potentially noisy) COLMAP camera annotations. We show two examples on the right, where COLMAP camera annotations lead to consistent failures of GS-LRM and LVSM during inference. 1 Introduction Self-supervised learning has driven the rise of foundation models, enabling training on vast amounts of unlabeled data and fueled by the scaling law [ 38 ] . This paradigm has proven highly effective for LLMs [ 60 ] , VLMs [ 2 ] , and visual generation [ 56 ] . In contrast, 3D Vision models still rely heavily on ground-truth 3D geometry and camera pose labels [ 77 , 28 ] , which are usually estimated from time-consuming optimization methods, e.g . , COLMAP [ 65 ] , and are not always perfect. This reliance limits both learning scalability and effectiveness. To break free from this constraint, we move beyond the supervised paradigm and ask: how far can we push a 3D Vision model without any 3D supervision ? In this paper, we present RayZer , a large multi-view 3D model trained with self-supervision and exhibiting emerging 3D awareness . The input of RayZer is unposed and uncalibrated multi-view images, sampled from continuous video frames or unordered multi-view captures. RayZer first recovers the camera parameters, then reconstructs the scene representation, and finally renders novel views. The key insight of our self-supervised training is to use the camera poses predicted by RayZer itself to render views that provide photometric supervision , rather than following the standard protocol of using ground truth poses for rendering [ 74 , 29 , 92 ] . Thus, RayZer can be trained with zero 3D supervision , i.e . , no 3D geometry or camera pose supervision. During inference, RayZer predicts camera and scene representations in a feed-forward manner, without requiring per-scene optimization. We show inference results in Fig. 1 . As RayZer uses the camera poses predicted by itself for training, this self-supervised task can be interpreted as 3D-aware image auto-encoding [ 61 , 95 , 41 ] . Initially, RayZer disentangles input images into camera parameters and scene representations (reconstruction). It then re-entangles these predicted representations back into images (rendering). To facilitate this disentanglement, we control the information flow . As shown in Fig. 2 , we divide all images into two parts : one set predicts the scene representation (input views), while the other offers photometric self-supervision (target views). This is achieved by using estimated poses of the second set to render the scene representation predicted from the first set, thereby preventing trivial solutions that are not 3D-aware. To facilitate self-supervised learning, RayZer is built only with transformers – no 3D representation, hand-crafted rendering equation, or 3D-informed architectures. This design is motivated by self-supervised large models in other modalities [ 6 , 2 , 56 ] , enabling RayZer to flexibly and effectively learn domain-specific knowledge. The only 3D prior incorporated in RayZer is the ray structure , which simultaneously models the relationship between camera, pixels (image), and scene. Concretely, RayZer first predicts camera poses, which are then converted into pixel-aligned Plücker ray maps [ 57 ] to guide the scene reconstruction that follows. This ray-based representation serves as a strong prior for addressing the chicken-and-egg problem of structure and motion [ 68 ] , effectively allowing the camera and scene representations to regularize each other during training. We evaluate RayZer on three datasets, including both scene-level and object-level data with different camera configurations. We observe that RayZer demonstrates comparable or even better novel view synthesis performance than “oracle” methods [ 91 , 33 ] that use pose labels in both training and testing. Interestingly, we identify that potentially noisy pose annotations from COLMAP can limit the performance of “oracle” models. The results not only demonstrate the effectiveness of RayZer, but also shows the potential of 3D Vision models to break free from supervised learning. 2 Related Work Large-scale 3D Vision Models . 3D Vision models learn 3D representations and priors from data [ 15 , 58 , 23 , 39 , 59 , 71 , 72 , 94 , 93 ] . Recently, researchers have developed large-scale models to acquire general 3D knowledge. One research direction focuses on designing improved model architectures that incorporate the inductive biases of multi-view stereo [ 86 , 75 , 10 , 14 ] and epipolar geometry [ 25 , 19 , 9 , 13 ] . Another line of work leverages full transformer models that intentionally omit architectural 3D inductive biases [ 62 , 54 , 29 ] . For example, LEAP [ 29 ] , LRMs [ 28 , 74 , 91 , 78 , 98 ] , and DUSt3R [ 77 , 43 , 20 , 83 , 76 ] are the first works employing transformers to convert 2D images into 3D representations. SRT [ 62 ] and LVSM [ 33 ] further replace 3D representations and physical rendering equations with latent representations and learned rendering functions, improving performance and scalability. However, they still require ground-truth camera poses for supervised training and/or accurate camera annotations during inference. To achieve scalable supervised learning, MegaSynth [ 32 ] and Stereo4D [ 34 ] leverage synthetic data and stereo videos to expand the data scale, however, curating data for different tasks can be laborious. In contrast, RayZer explores self-supervised training to break free from supervised learning. Figure 2 : Our proposed self-supervised training framework . This is an abstract design that we later operationalize with our RayZer model (illustrated in Fig. 3 and Sec. 4 ). We divide the input images into two sets \mathcal{I}_{\mathcal{A}} and \mathcal{I}_{\mathcal{B}} . We predict the scene representation from \mathcal{I}_{\mathcal{A}} , and use the predicted cameras