Scaling View Synthesis Transformers

Evan KimMIT [email protected], Hyunwoo RyuMIT [email protected] Thomas W. MitchelAdobe [email protected] Vincent SitzmannMIT [email protected] · 2026

Read Scaling View Synthesis Transformers by Evan KimMIT [email protected], Hyunwoo RyuMIT [email protected] Thomas W. MitchelAdobe [email protected] Vincent SitzmannMIT [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-21341.txt.

Introduction

Geometry-free view synthesis transformers now lead novel view synthesis, but their scaling behavior with compute is unclear. Evan Kim, Hyunwoo Ryu, Thomas Mitchel, and Vincent Sitzmann map those scaling laws in this 2026 arXiv paper, deriving design principles for compute-optimal models. Contrary to prior findings, encoder-decoder architectures can be compute-optimal; earlier negative results trace to suboptimal choices and unequal compute comparisons.

Across compute levels, their Scalable View Synthesis Model scales as effectively as decoder-only models. It reaches a superior performance-compute frontier and beats the previous state of the art on real-world benchmarks, with substantially reduced training compute. Practitioners get scaling guidance plus an architecture that spends compute where it matters for view synthesis.

Catalog record

CREATOREvan KimMIT [email protected], Hyunwoo RyuMIT [email protected] Thomas W. MitchelAdobe [email protected] Vincent SitzmannMIT [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-21341

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scaling View Synthesis Transformers Evan KimMIT [email protected], Hyunwoo RyuMIT [email protected] Thomas W. MitchelAdobe [email protected] Vincent SitzmannMIT [email protected] Abstract Geometry-free view synthesis transformers have recently achieved state-of-the-art performance in Novel View Synthesis (NVS), outperforming traditional approaches that rely on explicit geometry modeling. Yet the factors governing their scaling with compute remain unclear. We present a systematic study of scaling laws for view synthesis transformers and derive design principles for training compute-optimal NVS models. Contrary to prior findings, we show that encoder–decoder architectures can be compute-optimal; we trace earlier negative results to suboptimal architectural choices and comparisons across unequal training compute budgets. Across several compute levels, we demonstrate that our encoder–decoder architecture, which we call the Scalable View Synthesis Model (SVSM), scales as effectively as decoder-only models, achieves a superior performance–compute Pareto frontier, and surpasses the previous state-of-the-art on real-world NVS benchmarks with substantially reduced training compute. https://www.evn.kim/research/svsm 1 Introduction Figure 1 : Scaling Laws for View Synthesis Transformers. Evaluated on RealEstate10K ( 34 ) , our SVSM exhibits a 3\times more compute-optimal Pareto frontier than LVSM while retaining the same scaling behavior (similar slope and curvature everywhere). Given a set of images of a scene with known camera poses, the goal of Novel View Synthesis (NVS) is to render novel views of the scene from arbitrary viewpoints. Single scene approaches such as NeRF ( 16 ) and Gaussian Splatting ( 12 ) have achieved impressive fidelity by explicitly modeling 3D geometry and rendering. Feed-forward extensions of these frameworks train neural networks to reconstruct the 3D representation, achieving promising results ( 29 ; 2 ; 3 ; 31 ) . However, their formulation inherits handcrafted 3D structure, constraining their ability to scale and handle more complex artifacts such as reflections or transparency. Typified by Large View Synthesis Model (LVSM) 10 , a new class of view synthesis models have emerged which achieve state-of-the-art rendering quality using pure transformer architectures with fewer (if any) geometric inductive biases 10 ; 9 ; 17 ; 22 . However, this class of models is relatively new, and their design space remains unexplored. In particular, training a NVS model involves making design choices over a large number of variables, including the number of context and target views, camera pose parameterizations, attention mechanisms, etc., and there does not yet exist a rigorous investigation of how these choices affect the performance, training efficiency, and inference throughput. To this point, while there exist extensive scaling analyses in language modeling and 2D vision 7 ; 8 ; 11 ; 19 ; 30 , there exists no analogue for 3D vision. Thus, the goal of this work is to provide such an analysis in addition to a compute-optimal training recipe for view synthesis transformers in terms of both architecture and training strategy. In particular, we first challenge the necessity of the decoder-only architecture proposed in LVSM ( 10 ) . While powerful, it requires passing all context images through the entire transformer each time a single target image is decoded. It is a bidirectional model where both the target view tokens and context view tokens are updated in each layer of the network. While this allows the model to consider only information in the context images that is relevant to the target view, it incurs substantial computational cost due to repeated processing of context views. Instead, we advocate for an encoder-decoder design which produces an intermediate scene latent representation. This approach is potentially far more efficient: the computational cost of constructing the scene representation is amortized through repeated calls to the decoder, which efficiently extracts information from the representation via unidirectional cross attention from scene to target. However, the scene representation also represents an information bottleneck. Without a proper training strategy that maximally leverages the efficiency of the encoder-decoder design, it can be challenging for encoder-decoder models to outperform decoder-only models. Here, we identify that the key for unlocking the potential of encoder-decoder models lies in the way target views ( i.e. the reconstruction targets) are utilized during training. The implicit standard practice employed by prior work ( 2 ; 10 ; 22 ; 31 ) has been to reconstruct multiple different target views from a single scene during training. However, the consequences of this approach have never been fully analyzed. To this point, we propose and empirically validate the effective batch hypothesis , which argues that reconstructing multiple target views per scene effectively multiplies the batch size. These insights yield a principled transformer view synthesis model, which we call the Scalable View Synthesis Model (SVSM), that fully capitalizes on the rendering efficiency of a unidirectional encoder-decoder architecture, maximizing training throughput without compromising performance or scalability. We demonstrate that our unidirectional model scales as efficiently as bidirectional models, which aligns with the scalability of causal, unidirectional attention in large language models ( 11 ) . As part of this analysis, we also reveal scaling relationships within view synthesis that parallel those observed in the Chinchilla language model family ( 8 ) . Finally, we demonstrate that SVSM achieves state-of-the-art results in real-world NVS tasks with significantly reduced compute, challenging the previous understanding that bidirectional attention is critical to high-fidelity view synthesis ( 10 ) . Key Contributions: • We provide the first rigorous scaling analysis for novel view synthesis transformers. • We propose and empirically confirm the effective batch size hypothesis that unlocks compute-optimal training. • We show that bidirectional decoding is not critical for scalable view synthesis, contrasting recent work 10 . • Based on this analysis, we present a compute-optimal model that achieves a new state-of-the-art in real-world NVS tasks with substantially reduced training compute. 2 Related Work and Preliminaries Generalizable novel view synthesis. In generalizable novel view synthesis, we are given a set of V_{C} context images paired with known camera poses and intrinsics \mathfrak{C}=\left\{(I_{i},\,g_{i},\,K_{i})\mid i=1,\,\ldots,\,V_{C}\right\} . The typical objective is to synthesize an unseen view of the same scene given a target camera configuration g_{T},K_{T} : \tilde{I}_{T}=\mathrm{Render}\big[\mathfrak{C},g_{T},K_{T}\big]. (1) One line of work attempts to solve this problem with neural network architectures that explicitly model aspects of 3D image formation, for instance, via differentiable rendering or using epipolar line constraints 29 ; 24 ; 2 ; 26 ; 25 . In contrast, geometry-free methods avoid explicit geometric modeling in favor of flexibility and generality 6 ; 23 ; 22 ; 20 ; 21 . Here, we are primarily interested in a recently proposed subclass of these models which achieve state-of-the-art results with pure transformer architectures 10 ; 9 ; 22 ; 17 . In particular, we seek to study the “Large View Synthesis Model” (LVSM) 10 , which achieves state-of-the-art NVS performance and serves as the prototypical instance of the view synthesis transformer. LVSM can be implemented in two ways: as either an encoder-decoder model or decoder-only model. The authors’ proposed decoder-only variant is far more performant, so we primarily consider this architecture (pictured in Fig. 2 , left) in our analysis. Figure 2 : Architectures of the current SOTA, the decoder-only LVSM ( 10 ) (a) and SVSM (ours, b). Our cross-attention based decoder enables parallel rendering of multiple target views after a single scene encoding. Each target view is decoded independently given the shared scene representation, but

Read the complete text