MORPHOS: Autoregressive 4D Generation with Temporal Structured Latents Thanks: : Corresponding author Minkyung Kwon, Jinhyeok Choi, Youngjin Shin, Jaeyeong Kim, JongMin Lee Abstract We present MORPHOS, a novel autoregressive framework that generates dynamic 3D assets from videos across diverse representations, including meshes, 3D Gaussians, and radiance fields. Existing methods are typically limited to a single representation, struggle to model topological changes, or fail to maintain temporal consistency over long videos. To address these limitations, we introduce the Temporal Structured Latents (T-SL at ), a unified 4D representation that jointly encodes geometry and appearance along the temporal dimension. Leveraging T-SL at , MORPHOS autoregressively generates dynamic 3D assets via causal attention, conditioning each frame on its preceding history to ensure temporal consistency while handling evolving topologies. We also propose a temporal-structural augmentation to mitigate error accumulation in autoregressive generation. MORPHOS achieves state-of-the-art performance in appearance and competitive results in geometry across multiple benchmarks, demonstrating superior generalization across various representations and robustness in long-horizon generation. Figure 1 : Teaser. Given video inputs, MORPHOS autoregressively generates unified dynamic 3D representations (meshes, 3D Gaussians, and radiance fields). 1 Introduction The rapid evolution of 3D generative models has significantly advanced the frontiers of digital content creation, enabling the high-fidelity synthesis of 3D assets from image or text prompts 25 ; 52 ; 24 ; 43 ; 23 ; 50 . These frameworks utilize diverse representations, ranging from meshes for geometry modeling to 3D Gaussians 22 or Radiance Fields 30 for volumetric rendering. Building upon these foundations, 4D generative models extend synthesis to the temporal dimension, leveraging video inputs to generate dynamic, animated 3D content 37 ; 5 ; 36 ; 49 ; 32 ; 45 ; 46 ; 20 . Figure 2 : Autoregressive streaming. Method Representation Long-horizon Mesh Gaussian RF Generation Motion324 5 ✔ ✗ ✗ ✔ ActionMesh 37 ✔ ✗ ✗ ✗ Mesh4D 20 ✔ ✗ ✗ ✗ L4GM 36 ✗ ✔ ✗ ✔ GVFD 49 ✗ ✔ ✗ ✗ MORPHOS (Ours) ✔ ✔ ✔ ✔ Table 1 : Baseline comparison. MORPHOS generates diverse 3D representations, while maintaining temporal consistency in long-horizon generation. Despite their promise, current 4D generation methods suffer from two primary limitations: representation fragmentation and inability to handle topological changes . Existing frameworks typically specialize in a single, incompatible 3D format like meshes 37 ; 45 ; 20 ; 5 or 3D Gaussians 36 ; 49 . This specialization restricts model generalization across diverse modalities. Furthermore, they typically rely on deformation-based modeling to maintain temporal consistency 5 ; 37 ; 20 ; 49 . While effective for rigid motions, these methods struggle to accommodate topological changes or significant structural shifts over time. To address these challenges, we introduce MORPHOS , an autoregressive 4D generative model designed for unified, animated 3D representations. We first propose T-SL at (Temporal Structured Latents), a 4D representation which extends 3D structured latent 43 to the temporal domain. This latent space facilitates the joint encoding of geometry and appearance along the temporal dimension, while capturing complex structural transformations. Our proposed model, MORPHOS, consists of two autoregressive flow models 28 ; 43 : the first generates a sparse structure ( i.e., voxel) and the second generates T-SL at conditioned on the sparse structure. In both models, we adopt a causal attention architecture that conditions each frame on its preceding history, enabling long-horizon 4D generation from videos with large motion and evolving topologies (see Figure 2 ). Furthermore, this causal architecture enables key-value (KV) caching 12 ; 27 ; 47 ; 17 , which significantly reduces redundant computations and accelerates inference. To further ensure temporal consistency during the long-horizon generation, we introduce a temporal-structural augmentation strategy during training. Specifically, it addresses the accumulation of errors in the preceding history frames, as well as the imperfect sparse structure used in T-SL at generation. The temporal augmentation applies independent noise levels across frames during training, exposing the model to histories of varying quality and making it robust to imperfect past context at inference time. In addition, structural augmentation perturbs the sparse voxel structures conditioned on the generation of T-SL at , ensuring robustness to structural inaccuracies accumulated during the two-stage generation process. Experimental evaluations across multiple 4D benchmarks 37 ; 5 ; 19 demonstrate that MORPHOS achieves state-of-the-art performance in appearance and competitive results in geometry. Crucially, the framework effectively models both geometry and appearance even under significant topological transitions during long-horizon generation (see Table 1 ). Our main contributions are as follows: 1. We introduce an autoregressive 4D generative framework for unified and dynamic 3D representations, with complex motion and topological changes. 2. We propose a training strategy with temporal-structural augmentation to enhance the stability and robustness of long-horizon 4D generation. 3. MORPHOS demonstrates state-of-the-art performance across multiple 4D benchmarks, achieving superior geometric fidelity and appearance with minimal error accumulation. 2 Related Work 2.1 3D Asset Generation A growing line of works 48 ; 43 ; 50 ; 23 ; 52 ; 24 ; 25 ; 6 generate 3D assets from images or text inputs, by utilizing a diffusion process on learned, scalable 3D latent space 48 ; 43 . 3DShape2VecSet 48 introduces unordered, diffusible latent vector sets for encoding meshes and point clouds. TRELLIS 43 proposes voxel-based structured latents (SL at ), which can be decoded into diverse 3D representations — including meshes, 3D Gaussians 22 , and Radiance Fields 30 . Follow-up works extend this paradigm in various directions 7 ; 14 ; 42 ; 16 ; 18 . Our approach is built upon SL at for robust and unified 4D generation. 2.2 4D Asset Generation Recent 4D generation methods 36 ; 49 ; 45 ; 46 ; 37 ; 5 ; 20 extend 3D asset generation to dynamic 3D assets given a video input. Some 3D Gaussian-based methods reconstruct per-frame 3D Gaussians 36 or predict the deformation field of 3D Gaussians 49 , but operate entirely in Gaussian space without explicit geometric representations. On the other hand, mesh-based methods 45 ; 46 ; 20 typically fine-tune a pretrained 3D generator with temporal attention to produce mesh sequences. Some works utilize a deformation VAE decoder 37 ; 20 , or leverage a feed-forward transformer to predict the deformation of the first frame’s mesh 5 , producing topology-consistent mesh output. However, deformation-based approaches struggle to model structural transitions. In contrast, we generate frame-wise 3D representations with temporally consistent motion. 2.3 Autoregressive Video Diffusion Early video diffusion models 41 ; 2 ; 44 ; 3 ; 15 typically employ bidirectional attention to denoise frames simultaneously. Conversely, recent works 13 ; 47 ; 4 ; 17 ; 12 ; 21 ; 27 adopt autoregressive frameworks for improved scalability and long-horizon generation. Some methods 12 ; 27 ; 47 ; 17 employ sliding temporal windows and causal attention architecture to enable efficient inference via key-value (KV) caching. Diffusion Forcing 4 ; 38 utilizes independent per-frame noise levels to mitigate the training-inference gap, where the model is trained on ground truth history but conditioned on self-generated frames with error. While we adopt this formulation, the temporal augmentation alone is insufficient for our framework, as structural errors accumulate during the later stages of the diffusion process, highlighting the necessity of the structural augmentation. 3 Preliminary: Structured Latents for 3D Generation Structured latents (SL at ) 43 is a unified latent space that can be decoded into meshes, 3D Gaussians 22 , and Radiance Fields 30 . Specifically, a 3D asset \mathcal{O} is encoded to a SL at \mathbf{z}=\{(z_{i},x_{i})\}^{L}_{i=1} , where x_{i}\in\{1,...,N\}^{3} is a coordinate in the 3D voxel grid, and z_{i}\in\mathbb{R}^{C} is a visual feature attached on the voxel at x_{i} . Here, L\ll N^{3} denotes the number of occupied voxels. Specifically, to encode \mathcal{O} , the asset is first voxelized to obtain a sparse structure \mathbf{x}=\{x_{i}\}^{L}_{i=1} . Multi-view images are rendered around the asset to extract the visual features from a vision encoder 31 , which are then projected to the voxels \{x_{i}\}^{L}_{i=1} . This gives us a voxelized feature \mathbf{f}=\{(f_{i},x_{i})\}_{i=1}^{L} , where f_{i}\in\mathbb{R}^{D} , is the feature vector at voxel x_{i} . This sparse representation is then encoded by a sparse VAE 43 into the SL at \mathbf{z}=\{(z_{i},x_{i})\}_{i=1}^{L} , which can be decoded into diverse 3D formats by different decoders. To generate 3D assets, a two-stage rectified flow process 28 is applied to synthesize sparse structure and SL at sequentially. Sparse structure generation. The model first generates a sparse structure \mathbf{x} given an input image to capture an object’s coarse shape. The sparse structure is converted to a 3D occupancy binary grid \mathbf{o}\in\mathbb{\{}0,1\}^{N\times N\times N} and then encoded by VAE 43 with 3D convolutions to a downsampled feature \mathbf{s}\in\mathbb{R}^{D\times D\times D\times C_{s}} . The rectified flow 28 transformer \mathcal{G}_{\text{S}} 43 predicts the velocity field that transports a noise to the feature \mathbf{s} via flow matching. The generated feature \mathbf{s} can be decoded into the occupancy grid \mathbf{o} , from which the sparse structure \mathbf{x} is obtained. Structured latent generation. Given the active voxels in the generated sparse structure, another rectified flow transformer \mathcal{G}_{\text{L}} 43 generates the SL at \mathbf{z}=\{(z_{i},x_{i})\}_{i=1}^{L} from a structured noise, for fine-grained geometry and appearance from the input image. Finally, this generated SL at is decoded into meshes, 3D Gaussians, and Radiance Fields with the dedicated decoders 43 . 4 Method Given an input video \mathbf{I}^{1:T}=\{\mathbf{I}^{t}\}_{t=1}^{T} , where \mathbf{I}^{t}\in\mathbb{R}^{H\times W\times 3} denotes an RGB image at time t , our goal is to generate dynamic 3D assets including meshes, 3D Gaussians 22 , and Radiance Fields 30 . For a unified modeling of both geometry and appearance, we leverage a pre-trained image-to-3D generative model with structured 3D latents 43 . Building upon this, we design three key components: (i) T-SL at , which jointly encodes geometry and appearance along the temporal dimension ( Section 4.1 ); (ii) autoregressive generation with causal architecture, enabling efficient inference ( Section 4.2 ); (iii) temporal-structural augmentation training strategy to mitigate error accumulation for robust long-horizon generation ( Section 4.3 ). 4.1 Temporal Structured Latents for 4D Generation Figure 3 : Sparse structure generation. We introduce T-SL at (Temporal Structured Latents) by extending SL at into the temporal domain, which can be decoded into versatile dynamic 3D representations. We define an animated 3D asset, \mathcal{O}^{1:T}=\{\mathcal{O}^{t}\}_{t=1}^{T} , where \mathcal{O}^{t}=\langle\mathcal{V}^{t},\mathcal{F}^{t},\mathcal{T}^{t}\rangle is a mesh at time t with vertices \mathcal{V}^{t} , faces \mathcal{F}^{t} , and texture \mathcal{T}^{t} . We then encode this mesh sequence into the T-SL at \mathbf{z}^{1:T}=\{\mathbf{z}^{t}\}_{t=1}^{T} , where \mathbf{z}^{t}=\{(z_{i}^{t},x_{i}^{t})\}_{i=1}^{L_{t}} is SL at at time t . To preserve the temporal coherence of the animated 3D asset, we employ a global spatial normalization before the T-SL at encoding. We compute a global union Axis-Aligned Bounding Box (AABB) across the entire sequence of length T . Specifically, given the animated 3D asset \mathcal{O}^{1:T} , we determine the global spatial extrema across all frames. We define the sequence-level bounding box by extracting the minimum and maximum coordinates along each spatial axis j\in\{x,y,z\} : v_{\min}^{(j)}=\min_{1\leq t\leq T}\Big(\min_{v\in\mathcal{V}^{t}}v^{(j)}\Big),\quad v_{\max}^{(j)}=\max_{1\leq t\leq T}\Big(\max_{v\in\mathcal{V}^{t}}v^{(j)}\Big) (1) where v_{\min},v_{\max}\in\mathbb{R}^{3} . From these bounds, we derive a single scene center c=\frac{1}{2}(v_{\min}+v_{\max}) and a scale factor s=1/\max_{j}(v_{\max}^{(j)}-v_{\min}^{(j)}) . For every frame t , the mesh vertices are transformed via \bar{\mathcal{V}}^{t}=s(\mathcal{V}^{t}-c) , confining the entire motion sequence within a normalized canonical space. 4.2 Autoregressive 4D generation Figure 4 : T-SL at generation. T-SL at provides a unified representation for dynamic 3D generation. To enable AR generation, we employ a causal attention architecture over frames. The temporal augmentation applies different noise timesteps across frames for robust long-horizon modeling, while spatial augmentation uses voxel dropout to improve robustness against structural errors. To synthesize dynamic 3D assets from input videos, we propose a two-stage autoregressive generative framework within the T-SL at space. This approach decomposes 4D generation into the sequential synthesis of sparse structure features \mathbf{s}^{1:T}=\{\mathbf{s}^{t}\}_{t=1}^{T} and temporal structured latents \mathbf{z}^{1:T} . Specifically, we factorize the joint distribution of the T-SL at as a Markovian process: p(\mathbf{z}^{1:T}\mid\mathbf{I}^{1:T})=\prod_{t=1}^{T}p(\mathbf{z}^{t}\mid\mathbf{z}^{