Beyond Gaussian Bottlenecks: Topologically Aligned Encoding of Vision-Transformer Feature Spaces
Andrew Bond, Ilkin Umut Melanlioglu, Erkut Erdem, Aykut Erdem · 2026
Read Beyond Gaussian Bottlenecks: Topologically Aligned Encoding of Vision-Transformer Feature Spaces by Andrew Bond, Ilkin Umut Melanlioglu, Erkut Erdem, Aykut Erdem (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-28122.txt.
Introduction
Bond and colleagues argue that visual world models fail on geometry not only from limited capacity but from latent codes that ignore shape. Their S2VAE framework compresses a scene's latent three-dimensional state, including camera motion, depth and point structure, using hyperspherical distributions that preserve direction and geometry under strong compression. Across depth, camera pose and point-cloud tasks it beats standard Gaussian bottlenecks, especially when compression is severe.
Catalog record
| CREATOR | Andrew Bond, Ilkin Umut Melanlioglu, Erkut Erdem, Aykut Erdem |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54827 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.28122
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Gaussian Bottlenecks: Topologically Aligned Encoding of Vision-Transformer Feature Spaces Andrew Bond, Ilkin Umut Melanlioglu, Erkut Erdem, Aykut Erdem Abstract Modern visual world modeling systems increasingly rely on high-capacity architectures and large-scale data to produce plausible motion, yet they often fail to preserve underlying 3D geometry or physically consistent camera dynamics. A key limitation lies not only in model capacity, but in the latent representations used to encode geometric structure. We propose S 2 VAE, a geometry-first latent learning framework that focuses on compressing and representing the latent 3D state of a scene, including camera motion, depth, and point-level structure, rather than modeling appearance alone. Building on representations from a Visual Geometry Grounded Transformer (VGGT), we introduce a novel type of variational autoencoder using a product of Power Spherical latent distributions, explicitly enforcing hyperspherical structure in the bottleneck to preserve directional and geometric semantics under strong compression. Across depth estimation, camera pose recovery, and point cloud reconstruction, we show that geometry-aligned hyperspherical latents consistently outperform conventional Gaussian bottlenecks, particularly in high-compression regimes. Our results highlight latent geometry as a first-class design choice for physically grounded visual and world models. 1 Introduction Modern computer vision has increasingly shifted from pixel-level representations to high-dimensional feature spaces produced by large-scale Vision Transformers (ViTs) ( 5 ) . Recent architectures further enrich these representations by embedding explicit geometric information, such as camera motion, depth, and 3D structure, directly into the Transformer’s hidden states. While these features are highly expressive, their dimensionality makes them impractical, necessitating aggressive compression into a compact latent space. A central but often implicit assumption in this process is that standard latent variable models, particularly Gaussian VAEs, are well matched to the geometry of transformer features. In practice, this assumption breaks down. Due to the extensive use of Layer Normalization and dot-product attention, ViT features concentrate on approximately hyperspherical manifolds, where semantic identity is encoded primarily through angular relationships rather than Euclidean magnitude. Compressing such features with a Gaussian VAE introduces a fundamental geometric mismatch: the model is forced to map data that lies on a thin spherical shell into a dense Euclidean volume. This mismatch manifests in several failure modes, including posterior collapse under strong compression, radial noise that perturbs angular semantics, and gradual semantic drift in reconstructed features. These effects are mild at low compression ratios but become pronounced precisely in the regimes that matter most, i.e. high compression and downstream geometry-sensitive tasks such as depth estimation, camera pose regression, and point cloud reconstruction. Despite these issues, Gaussian latents remain the default choice, largely due to their mathematical convenience rather than empirical suitability. This raises a key question: Is the observed degradation under compression an unavoidable consequence of information loss, or is it amplified by a latent geometry that is fundamentally misaligned with Transformer features? To address this question, we introduce S 2 VAE , a geometry-aligned variational autoencoder that uses a product of hyperspherical latent distributions to respect the directional geometry of ViT feature representations under compression. Instead of a multivariate Gaussian bottleneck, we employ a product of Power Spherical distributions , enforcing hyperspherical structure directly in the latent space. By decomposing the latent into multiple lower-dimensional spheres, the model maintains numerical stability while introducing a compositional inductive bias that facilitates the separation of distinct geometric attributes. Crucially, this alignment removes the need for the model to implicitly learn spherical constraints, allowing compression to focus on preserving semantically meaningful directions. We show that this geometry-aligned bottleneck substantially reduces the failure modes observed with Gaussian VAEs. The resulting latent representation supports strong reconstruction fidelity even at high compression ratios and integrates naturally with task-specific geometric losses. As a result, downstream performance is preserved not merely because the model is expressive, but because the latent geometry is matched to the structure of the features it compresses. Our contributions are threefold: 1. We provide an empirical diagnosis of the mismatch between Gaussian latent assumptions and the intrinsic geometry of ViT features, and show that this mismatch leads to measurable degradation under compression. 2. We propose a geometry-aligned latent bottleneck based on a product of hyperspherical distributions, enabling stable and differentiable compression of transformer features while preserving directional semantics. 3. We demonstrate that latent alignment matters: compared to Gaussian VAEs, our approach consistently yields superior reconstruction quality and downstream geometric performance, particularly in high-compression regimes. 2 Related Work 2.1 Latent Variable Models for Non-Euclidean Data While standard VAEs assume Euclidean space and data, there have been much work into similar models for other types of data. The foundational work on this field is ( 3 ) , which introduced the original Hyperspherical VAE formulation, based on the von Mises-Fisher (vMF) distribution. Meanwhile, ( 4 ) introduced the power spherical distribution for VAEs, which helped resolve the instability issues and inefficient sampling of the vMF distribution. Additionally, ( 8 ) explored other types of manifold-valued latent variables, focusing on the case of VAEs for Lie groups. While focused on simple experiments, they show that choosing the correct manifold for the data can strongly affect the latent representation. Finally, there is also some work which has explored the phenomenon of posterior collapse in VAEs, a common problem in training these models ( 16 ) . 2.2 Geometry of Transformer Features Much of the analysis of the geometry of transformer features has come from analysis of LLMs. ( 12 ) showed that certain pretrained LLMs always induce non-smooth anisotropic semantic spaces, emphasizing the importance of focusing on certain directions. Meanwhile, ( 7 ) showed something similar, that in certain layers of LLMs, the anisotropy can be so bad that the cosine similarity between embeddings of two random words can be almost perfect. ( 17 ) also argued that directional similarity is more important, and proposed a method to learn text embeddings in spherical space. 2.3 Visual Geometry and World Models Prior work has explored reconstructing visual signals from Vision Transformer representations, primarily with the goal of recovering pixel-level appearance or enabling generative modeling ( 27 ) . In contrast, our work focuses on representations that are explicitly grounded in 3D geometry rather than appearance. We build on geometry-centric foundation models such as VGGT ( 24 ) , which are trained to encode camera motion, depth, and point-level structure directly in their hidden states. Recent extensions of this line of work have begun to address dynamic scenes, for example by disentangling pose and geometry across time ( 29 ) . While these approaches demonstrate the expressiveness of geometry-aware transformers, they do not address how such high-dimensional geometric representations can be efficiently compressed or structured in a latent space. Our work complements these efforts by focusing on geometry-aligned latent representations that preserve directional and structural information under compression. 3 Preliminaries 3.1 Visual Geometry Grounded Transformer (VGGT) VGGT is a geometry-first foundation model. Unlike standard ViTs which are trained on semantic tasks, VGGT is explicitly trained to understand the 3D structure of the world, by explicitly requiring the model to reconstruct a wide range of geometric primitives, such as camera poses, depth maps, and point clouds. Through this training, the model learns to produce tokenized representations that are grounded in physical space. These features are high-dimensional ( 2048