Hyperbolic Residual Quantization: Discrete Representations for Data with Latent Hierarchies

Piotr Piękos, Subhradeep Kayal, Alexandros Karatzoglou · 2025

Read Hyperbolic Residual Quantization: Discrete Representations for Data with Latent Hierarchies by Piotr Piękos, Subhradeep Kayal, Alexandros Karatzoglou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-12404.txt.

Introduction

Piotr Piękos, Subhradeep Kayal, and Alexandros Karatzoglou published a paper in 2025 on turning layered data into compact codes. Many data sets hide a tree-like order, from word meanings to company charts. A common coding method squeezes data step by step in flat space, which fights against that branching shape.

Their method performs the same stepwise coding inside a curved space shaped like a saddle, which fits trees far better. They test it two ways: learning a known word hierarchy, and finding hidden order the model was never told about. In both tests the curved codes served later tasks better, with gains up to twenty percent on the hierarchy test. Geometry, chosen well, becomes a form of knowledge.

Catalog record

CREATORPiotr Piękos, Subhradeep Kayal, Alexandros Karatzoglou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-12404

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hyperbolic Residual Quantization: Discrete Representations for Data with Latent Hierarchies Piotr Piękos, Subhradeep Kayal, Alexandros Karatzoglou Abstract Hierarchical data arise in countless domains, from biological taxonomies and organizational charts to legal codes and knowledge graphs. Residual Quantization (RQ) is widely used to generate discrete, multitoken representations for such data by iteratively quantizing residuals in a multilevel codebook. However, its reliance on Euclidean geometry can introduce fundamental mismatches that hinder modeling of hierarchical branching, necessary for faithful representation of hierarchical data. In this work, we propose Hyperbolic Residual Quantization (HRQ), which embeds data natively in a hyperbolic manifold and performs residual quantization using hyperbolic operations and distance metrics. By adapting the embedding network, residual computation, and distance metric to hyperbolic geometry, HRQ imparts an inductive bias that aligns naturally with hierarchical branching. We claim that HRQ in comparison to RQ can generate more useful for downstream tasks discrete hierarchical representations for data with latent hierarchies. We evaluate HRQ on two tasks: supervised hierarchy modeling using WordNet hypernym trees, where the model is supervised to learn the latent hierarchy - and hierarchy discovery, where, while latent hierarchy exists in the data, the model is not directly trained or evaluated on a task related to the hierarchy. Across both scenarios, HRQ hierarchical tokens yield better performance on downstream tasks compared to Euclidean RQ with gains of up to 20 % 20\% for the hierarchy modeling task. Our results demonstrate that integrating hyperbolic geometry into discrete representation learning substantially enhances the ability to capture latent hierarchies. 1 Introduction Hierarchical structures appear throughout human knowledge and information organization, serving as essential frameworks for understanding complex relationships between entities. These structures can be found in biological classifications of living organisms [ 36 ] , business organizational structures [ 13 ] , and computer file systems [ 38 ] . Studies show that when children learn, they organize their knowledge in hierarchies [ 27 ] . This pattern extends to numerous other domains as well: from taxonomic categorization in libraries and archives to the nested organization of legal codes and regulations. Government systems typically follow hierarchical arrangements, with federal, state, and local levels each containing their own internal hierarchies. Similarly, academic disciplines are organized into fields, subfields, and specialized areas. The presence of hierarchical structures across such diverse domains reflects their importance in how humans conceptualize and organize information. In the current state of machine learning modeling, most often continuous vectors are used to represent entities [ 5 , 39 , 7 ] . However, it can sometimes be beneficial to use discrete representations rather than continuous vector embeddings. Discrete tokens function effectively as labels because in the discrete domain, generation is equivalent to prediction. This equivalence allows models to avoid complicated generation methods like GANs [ 19 ] or diffusion models [ 25 ] , and instead rely on more straightforward prediction tasks. Discrete representations also tend to be more interpretable as each token can correspond to a specific concept or attribute in the hierarchy [ 46 ] . Figure 1: Recall@10 of the hypernym generation based on tokens generated by HRQ vs tokens generated by RQ. HRQ consistently outperforms RQ. Furthermore, HRQ sustains consistent scores across different dimensionalities of the embedding. Residual Quantization Variational Autoencoders (RQ-VAE) [ 32 , 60 ] leverage these benefits by creating semantic hierarchical discrete representations through a multilevel quantization process [ 53 ] . At each step of residual quantization, the model encodes increasingly fine-grained details, with earlier levels capturing broader structural elements and later levels representing more specific attributes. The result is a list of tokens that together create an identifier of the entity. We will refer to this hierarchical discrete representation as Multitoken(MT) . By learning discrete tokens at multiple levels of abstraction, RQ-VAE provides a framework for modeling hierarchical relations directly from dense embedding. However, RQ-VAE operates within Euclidean space, which imposes fundamental limitations on its ability to capture hierarchical relationships. Euclidean geometry struggles to efficiently represent tree-like structures [ 22 ] , as the volume of space grows polynomially with distance from the origin, while the number of nodes in a hierarchy typically grows exponentially with depth. This geometric mismatch means that Euclidean-based models like RQ-VAE inevitably lose important hierarchical information during encoding. In contrast, hyperbolic space [ 22 ] - a Riemannian manifold with constant negative curvature - has been shown to model hierarchies remarkably well [ 42 , 43 ] . The hyperbolic space can approximately isometrically embed any tree already in two dimensions [ 22 ] , whereas the same cannot be said for the Euclidean space of any dimension. The volume in the hyperbolic space grows exponentially with distance from the origin, aligning well with the growth of number of nodes in hierarchy. The ability to encode trees by hyperbolic geometry has inspired numerous advances in machine learning. Hyperbolic neural networks have also been extensively leveraged in continuous embedding models to exploit latent hierarchies in a variety of domains. Poincaré embeddings [ 42 ] , learn continuous hierarchies by mapping symbolic data into an n - dimensional Poincaré ball. The authors showed that these embeddings outperform the Euclidean ones on tree - structured data in terms of both representation capacity and generalization ability. Hyperbolic embeddings found immediate use in data domains rich in latent hierarchies, such as knowledge - graph representation [ 3 , 12 , 34 ] and recommender systems [ 51 , 10 , 41 ] , and other [ 56 , 18 , 33 ] . Despite these advances in continuous embedding models, the application of hyperbolic geometry to discrete representation learning has remained mostly underexplored. HyperVQ [ 20 ] proposes to perform vector quantization in a hyperbolic space by phrasing it as a hyperbolic multinomial logistic regression. In this paper, we use hyperbolic distance to find the nearest codebook vector and focus on the hyperbolic version of Residual Quantization. We introduce Hyperbolic Residual Quantization (HRQ), which performs residual quantization (RQ) in a hyperbolic space with an adapted process of residual quantization to accommodate the hyperbolic structure. We claim that for data with latent hierarchies residual quantization benefits from hierarchical inductive bias induced by hyperbolic space. We implement this approach through several key adaptations: first, we employ hyperbolic neural networks for the embedding process, ensuring that data representations reside natively in hyperbolic space. Second, we utilize hyperbolic operations to calculate the residuals between quantization levels, preserving the geometric properties of the space throughout the quantization process. Finally, we incorporate hyperbolic distance metrics in the clustering algorithm, allowing the model to properly capture the hierarchical relationships between data points. These modifications enable HRQ-VAE to make use of the natural advantages of hyperbolic geometry to represent hierarchical structures while maintaining the benefits of discrete token-based representations. We evaluate the quality of the multitokens created by HRQ in two scenarios. First, we test its ability to model hierarchies with supervision on the hierarchy. (H)RQ creates multitokens of nouns [ 40 ] based on their hypernymy relation. Then, we test which representation is more useful in generating the hypernym for a given noun. We show that multitokens learned with Hyperbolic Residual Quantization significantly outperform tokens learned with Residual Quantization. Furthermore, we test the

Read the complete text