Learning on the Manifold: Unlocking Standard Diffusion Transformers with Representation Encoders Amandeep Kumar, Vishal M. Patel Abstract Leveraging representation encoders for generative modeling offers a path for efficient, high-fidelity synthesis. However, standard diffusion transformers fail to converge on these representations directly. While recent work attributes this to a capacity bottleneck—proposing computationally expensive “width scaling” of diffusion transformers—we demonstrate that the failure is fundamentally geometric. We identify Geometric Interference as the root cause: standard Euclidean flow matching forces probability paths through the low-density interior of the hyperspherical feature space of representation encoders, rather than following the manifold surface. To resolve this, we propose Riemannian Flow Matching with Jacobi Regularization (RJF) . By constraining the generative process to the manifold geodesics and correcting for curvature-induced error propagation, RJF enables standard Diffusion Transformer architectures to converge without width scaling. Our method RJF enables the standard DiT-B architecture (131M parameters) to converge effectively, achieving an FID of 3.37 where prior methods fail to converge. Code: https://github.com/amandpkr/RJF Figure 1 : Bridging the Geometric Gap. We demonstrate that respecting the intrinsic geometry of pre-trained representations encoders enables the use of standard Diffusion Transformers without any architectural modification such as Width Scaling ( Zheng et al. 2025 ) . Our method, Riemannian Flow Matching with Jacobi Regularization ( +DiNO+RJF ), achieves an FID of 4.95 using standard LightingDiT-B ( Yao et al. 2025 ) architecture without guidance , significantly outperforming the VAE-based LightingDiT-B (FID 15.83 ). In contrast, applying standard Flow Matching to DINOv2-B features ( +DiNO ) fails to converge (FID 21.64 ) due to Geometric Interference . Even restricting the noise to the hypersphere to strictly learn the angular component ( +DiNO+SN ) yields only marginal improvement (FID 19.07 ), as the Euclidean linear paths still traverse the low-probability interior of the feature manifold. 1 Introduction Flow Matching ( Lipman et al. 2022 ; Esser et al. 2024 ; Liu et al. 2022 ; Albergo & Vanden-Eijnden 2022 ) and Diffusion models ( Ma et al. 2024 ; Rombach et al. 2022 ; Ho et al. 2020 ; Song et al. 2020 ) have revolutionized generative modeling, enabling high-fidelity synthesis across modalities. While initial approaches operated in pixel space, the paradigm has shifted toward Latent Diffusion Models (LDMs) ( Rombach et al. 2022 ; Vahdat et al. 2021 ) that leverage compressed representations of VAE ( Kingma & Welling 2013 ) . However, because VAEs are optimized for reconstruction, they predominantly capture low-level texture; this forces the diffusion model to learn high-level semantics from scratch, leading to slow convergence. To overcome this, recent works enhance the VAE latent space with semantic priors from strong representation encoders like DINOv2 ( Oquab et al. 2023 ) and SigLIP ( Radford et al. 2021 ) . These approaches typically require explicitly aligning semantic representations within the VAE latent space ( Yao et al. 2025 ) or the diffusion intermediate features ( Leng et al. 2025 ; Yu et al. 2024 ) . Consequently, these methods often necessitate complex auxiliary losses and additional training stages. Recent work challenges this complexity by proposing Representation Autoencoders (RAE) ( Zheng et al. 2025 ) , which discard the VAE entirely in favor of diffusing directly within the feature space of frozen representation encoders. RAEs demonstrate that these high-dimensional semantic representations can support high-fidelity generation without the need for auxiliary alignment losses or complex training stages. However we also have the similar observation like RAE ( Zheng et al. 2025 ) that the standard diffusion recipe fails to converge effectively on these high-dimensional latents, even in a simplified single-image overfitting regime. While RAE attributes this failure to a capacity bottleneck, proposing to scale the transformer width to match the latent dimensionality—we identify a more fundamental cause rooted in the intrinsic geometry of the latent space. We argue that the optimization difficulty arises not from insufficient parameter count, but from a “Geometry Gap”: a structural conflict where the Euclidean probability paths assumed by standard flow matching ( Lipman et al. 2022 ) violate the hyperspherical manifold of representation space. To understand this failure, we analyze the intrinsic geometry of the feature space. We observe that DINOv2 representations do not populate the ambient Euclidean space but are strictly confined to a hypersphere, creating a hard shell geometry, and all the information are encoded in angular vectors. We identify the root cause of convergence failure for standard diffusion transfomer as Geometric Interference : the standard linear probability path used in flow matching cuts through the low-density interior (off-manifold) of hypersphere (forming a chord), rather than following the manifold’s surface ( Rozen et al. 2021 ; Mathieu & Nickel 2020 ) . This forces the model to learn a velocity field in regions where the representation space is undefined as shown in Figure 2 . Crucially, we challenge the prevailing hypothesis that this requires scaling model width ( Zheng et al. 2025 ) . Our experiments reveal that the model has sufficient capacity to learn the semantics, but under the standard objective, it wastes its capacity, minimizing radial error (learning feature magnitude which is fixed as the radius of hypersphere) and learning trajectories through the off-manifold area induced by this geometric mismatch. Figure 2 : Geometric Trajectories on the Hypersphere. Visualization of flow matching paths on the manifold \mathcal{S}^{d-1} . Standard Euclidean Flow Matching constructs linear paths that ignore the manifold geometry. Whether targeting standard Gaussian noise \epsilon ( orange ) or projecting noise onto the sphere \epsilon_{s} ( purple ) to strictly learn the angular component, the linear interpolation forms a chord that cuts through the low-density interior. This forces the model to learn a velocity field in undefined regions regardless of the endpoint. In contrast, Riemannian Flow Matching follows the geodesic (blue curve) , ensuring the intermediate state x_{t} remains strictly on the manifold surface. The resulting velocity field u_{t}^{M}(x_{t}) is correctly defined within the tangent space ( pink plane ), naturally respecting the geometry of the representations. Motivated from this insight, we propose Riemannian Flow Matching with Jacobi Regularization (RJF). First, we address the trajectory mismatch by adopting Riemannian Flow Matching ( Chen & Lipman 2023 ) , which replaces the Euclidean linear path with Spherical Linear Interpolation (SLERP). This ensures the generative process follows the geodesic(shortest path along the curve between two points), staying strictly on the manifold surface. Second, we recognize that simply fixing the path is insufficient because the flow matching objective remains geometrically unaware; it treats errors uniformly. On a positively curved hypersphere, velocity errors propagate non-linearly due to the focusing of geodesics (similar to how parallel longitude lines eventually meet at the poles). To correct this, we introduce a Jacobi Regularization derived from Jacobi fields ( Zaghen et al. 2025 ) , which reweights the loss to account for curvature-induced distortion. This geometric alignment enables standard DiT architectures to converge efficiently without width scaling. Our contributions are summarized as follows: • Geometric Analysis of Convergence Failure: We identify Geometric Interference as fundamental bottleneck preventing standard diffusion transformers from learning on high-dimesion representations. We demonstrate that failure arises not from a capacity deficit, but from the Euclidean objective forcing the model to minimize radial errors and learning trajectories through the low-density interior of the feature manifold. • Riemannian Flow Matching with Jacobi Regularization: We propose a geometrical framework that defines the generative process directly on the hyperspherical manifold. By combining Riemannian Flow Matching (to correct the trajectory) with Jacobi Regularization (to account for geodesic focusing), we ensure the optimization is consistent with both the topology and curvature of the latent space. • Efficient Generative Modeling: We achieve state-of-the-art performance using standard DiT architectures without the need for computationally expensive width scaling. On the 131M-parameter DiT-B, along with RJF and DINOv2-B achieves an FID of 3.37 with guidance and FID of 4.95 without guidance in 200 epochs as shown in Figure 1 , whereas the standard flow matching fails to converge. These gains persist at scale: on DiT-XL, we attain an FID of 3.62 in 80 epochs without guidance, outperforming both the standard flow matching (FID 4.28) and the VAE-based DiT trained with alignment losses (FID 4.29). 2 Geometrical Analysis Following RAE ( Zheng et al. 2025 ) , we investigate the feasibility of directly using pretrained representation encoders within the Diffusion Transformer framework and had similar observation that standard diffusion recipe fail to converge effectively, even in a simplified single-image overfitting. Rather than seeking marginal architectural improvements to address this failure, we aim to answer a more fundamental question: Why are these high-dimensional, semantically rich representations resistant to the standard Diffusion recipe? To answer this, we first analyze the intrinsic geometry of the feature space produced by these encoders. 2.1 The Geometry Gap We analyze the distribution of the final feature vectors z\in\mathbb{R}^{d} extracted from the DINOv2-B encoder. Decomposing these features into radial and angular components reveals a rigid geometric constraint: the features do not populate the ambient Euclidean space but are explicitly projected onto a hypersphere \mathcal{S}^{d-1} of fixed radius \sqrt{d} : z=r\cdot\hat{z},\quad\text{where }r\approx\sqrt{d}\text{ and }\hat{z}\in\mathcal{S}^{d-1}. (1) As illustrated in Figure 3 , the radial component r exhibits near-zero variance due to the ubiquitous application of LayerNorm. This creates a hard shell geometry where all semantic information is encoded exclusively in the angular component \hat{z} . This stands in sharp contrast to the standard Gaussian prior used in diffusion models, which assumes a probability mass concentrated in a diffuse shell. Figure 3 : The Geometry Gap . A comparison of radial feature norms ( r=\|z\|_{2} ) between DINOv2-B representations and a standard Gaussian prior in \mathbb{R}^{768} . While the Gaussian prior (blue) is distributed across a diffuse shell, DINOv2-B features (orange) are rigidly constrained to a hypersphere with near-zero radial variance. This extreme geometric mismatch prevents standard diffusion models from converging effectively. Figure 4 : Geometric Interference vs. Capacity. We train DiT-S models of varying widths on DINOv2 tokens ( d=768 ). Top Row: When minimizing Euclidean MSE, narrower models ( d<768 ) suffer from collapse; the Angular Loss (semantics) gets stuck. Bottom Row: When the radial loss is ignored, even narrow models ( d=384 ) converge perfectly on the angular component. This proves the bottleneck is not the dimensionality of the data, but the geometric conflict in the objective. This hyperspherical geometry reveals why the standard flow matching become suboptimal. The standard algorithm constructs a conditional probability path p_{t}(x) via linear interpolation between the source distribution (Gaussian noise \epsilon ) and the target data ( x ): x_{t}=(1-t)x+t\epsilon. (2) In Euclidean space, this linear trajectory is optimal. However, on a hyperspherical manifold, this creates a critical distribution shift ( Rozen et al. 2021 ; Mathieu & Nickel 2020 ) . Since \epsilon and x are high-dimensional vectors, they are approximately orthogonal( \epsilon\cdot x\approx 0 ). Consequently, the squared norm of the intermediate state x_{t} follows: \|x_{t}\|^{2}\approx(1-t)^{2}\|x\|^{2}+t^{2}\|\epsilon\|^{2}. (3) At t=0.5 , the norm collapses to \|x_{0.5}\|\approx\frac{1}{\sqrt{2}}\sqrt{d}\approx 0.7\sqrt{d} . This implies that the linear flow trajectory x_{t} does not stay on the manifold \mathcal{S}^{d-1} but rather cuts through the interior of the hypersphere (a chord). This forces the network to learn a velocity field v_{t} in regions of the feature space that are strictly off manifold for the pretrained representation encoder. The model must essentially hallucinate valid semantic gradients in a region where the representation space is undefined, leading to the convergence failure. 2.2 Revisiting the Capacity Hypothesis: Geometric Interference This convergence failure is also identified in RAE ( Zheng et al. 2025 ) , to resolve that they proposed a width scaling solution: increasing the Diffusion Transformer’s width ( d_{model} ) to match atleast the token dimension ( n ). Crucially, they demonstrate that this is not just a capacity issue—simply adding layers (depth) fails to improve convergence. They hypothesize that the bottleneck is strictly dimensional: because the added Gaussian noise is full-rank, a model with width d_{model}