Self-supervised Latent Space Optimization with Nebula Variational Coding
Yida Wang, David Joseph Tan, Nassir Navab, and Federico Tombari · 2025
Read Self-supervised Latent Space Optimization with Nebula Variational Coding by Yida Wang, David Joseph Tan, Nassir Navab, and Federico Tombari (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01414.txt.
Introduction
Yida Wang and co-authors present a 2025 paper on shaping the hidden space inside deep learning models so that it clusters by meaning. Features in that latent space usually scatter without order. Their method plants guide points, called nebula anchors, that pull matching features into clusters during training.
A variational rule keeps each cluster bell-shaped and stops the anchors from collapsing together. Because every feature sits nearest to some anchor, self-supervised metric learning can then push the clusters further apart. The coder works across classification, segmentation, completion, and rebuilding tasks. Experiments show the clustered space adapts to the labels hidden in the data. Cleaner latent order can make downstream models simpler and steadier.
Catalog record
| CREATOR | Yida Wang, David Joseph Tan, Nassir Navab, and Federico Tombari |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01414
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Self-supervised Latent Space Optimization with Nebula Variational Coding Yida Wang, David Joseph Tan, Nassir Navab, and Federico Tombari Abstract Deep learning approaches process data in a layer-by-layer way with intermediate (or latent) features. We aim at designing a general solution to optimize the latent manifolds to improve the performance on classification, segmentation, completion and/or reconstruction through probabilistic models. This paper proposes a variational inference model which leads to a clustered embedding. We introduce additional variables in the latent space, called nebula anchors , that guide the latent variables to form clusters during training. To prevent the anchors from clustering among themselves, we employ the variational constraint that enforces the latent features within an anchor to form a Gaussian distribution, resulting in a generative model we refer as Nebula Variational Coding (NVC). Since each latent feature can be labeled with the closest anchor, we also propose to apply metric learning in a self-supervised way to make the separation between clusters more explicit. As a consequence, the latent variables of our variational coder form clusters which adapt to the generated semantic of the training data, e.g. the categorical labels of each sample. We demonstrate experimentally that it can be used within different architectures designed to solve different problems including text sequence, images, 3D point clouds and volumetric data, validating the advantage of our proposed method. I Introduction In machine learning, we aim at learning relationships of different kinds between the input and the output signals. For instance, solutions in language translation [ 1 , 2 ] and 3D understanding [ 3 , 4 , 5 ] aim at completely transforming the input data to a different form as the probabilities in classification and segmentation. They rely on compressing the input to its latent representations while identifying its discriminative information. On the other hand, solutions in image processing [ 6 , 7 , 8 , 9 ] and 3D understanding [ 10 , 11 , 12 , 13 , 14 , 8 ] aim at preserving the information from the input data as part of the output, formulating additional skip connection [ 9 , 8 , 10 , 12 ] and fusion [ 11 ] between them. Although there are traditional methods that can reduce the dimensionality of the input in an unsupervised way, such as principal component analysis (PCA) [ 15 ] and independent component analysis (ICA) [ 16 ] , as well as in a supervised way, e.g. linear discriminant analysis (LDA) [ 17 ] , they can hardly be applied when modeling complex data. Moving towards deep learning, such compression can also be modeled with an encoder-decoder architecture. Auto-encoders (AE) [ 18 , 19 ] are among the simplest, yet effective, architectures able to extract the latent features in an unsupervised fashion. For these reasons, they are commonly employed in a variety of tasks such as image denoising [ 20 , 21 ] and retrieval [ 22 ] . Moreover, more complicated tasks such as pose estimation [ 23 , 24 , 25 ] or 3D completion [ 5 , 10 ] utilize a more generalized encoder-decoder architecture. Fig. 1 : Example applications of the proposed nebula variational coder (NVC) on different architectures that are designed to solve problems in (a) planar reconstruction, (b) semantic scene completion, (c) point cloud segmentation and (d) hand pose estimation. Note that (a) and (b) take real images captured from real scenes as input while (c) and (d) take synthetic images rendered from realistic scenes. Looking at the bigger picture, there are three ways to improve the architecture, namely, encoder design [ 26 ] , decoder design [ 27 ] and latent feature optimization [ 26 ] . In this work, we focus on proposing a novel latent feature optimization approach. One of the popular approaches to optimize the latent feature is through the variational inference [ 28 , 29 ] . It matches the likelihood of the encoder with the true posterior of the decoder by setting a distribution constraint on the latent features such as Gaussian or Bernoulli. Since they impose a distribution in the latent space, it becomes feasible to generate reasonable output, with a trained decoder, directly from random sampling of the latent feature, transforming the deep architecture to a generative model such as variational auto-encoder (VAE) [ 28 , 29 ] . However, there are two main problems in assuming a simple distribution like the Gaussian in VAEs. First, the latent space often includes unused regions which are accountable for generating samples that do not belong to the real distribution. Specifically for VAEs, the Gaussian assumption with the fixed expected density does not always match the likelihood of the encoder with the true posterior of the decoder, leading, e.g., to blurry reconstructed contours in perceptual VAE [ 30 ] . Such problem is mostly influenced by the decoder. With respect to the encoder, another problem is the uncertainty described in [ 31 ] , where the latent variables are not expected to be continuous in some situations. To solve these issues, conditional information [ 29 ] and additional parameters [ 5 , 10 ] out of the encoder-decoder inference model are used to improve their results. Nevertheless, these solutions carry their own disadvantages. In the former, the conditional information is still required during inference; while, in the latter, the additional architecture makes training less efficient. We propose a new variational inference model which can be applied to different architectures, and used in a variety of supervised and unsupervised training including classification, regression and recurrent prediction. Our approach relies on the idea of having additional latent variables used during training, called nebula anchors , that are explicitly embedded in training the model so to help the formation of hyper-clusters in the latent space. One of its main advantages is the ability to exploit the semantic meaning such as the categorical information in the latent space without any human supervision. As an example on the MNIST dataset [ 32 ] , we illustrate that our generative model automatically forms 10 clusters, each representing a different digit (see Fig. 4 (b)). The limitation of our anchor-based solution is the need to pre-define one key hyper-parameter which is the number of anchors. To ease this, we also introduce a self-supervised metric learning derived from the Siamese [ 33 ] and Triplet [ 34 , 35 , 36 ] losses, tailored for the nebula anchors. We apply the metric learning efficiently using the labels produced by the nebula anchors based on the nearest neighbour classifier to separate the clusters. Notably, unlike other generative models such as CVAE [ 29 ] that use labels as conditional variables to improve inference, our method exploits the additional information only during training, which implies that no additional information other than the input samples is required during inference. By design, our nebula anchors can be applied on the latent space of not just the VAEs, but also other deep networks such as Convolutional Neural Networks (CNN) [ 37 , 38 ] , Recurrent Neural Networks (RNNs) [ 39 , 40 ] as well as adversarial generative models such as GANs [ 41 , 42 , 5 ] . Motivated