Hierarchical Mamba Meets Hyperbolic Geometry: A New Paradigm for Structured Language Embeddings
Sarang Patil [email protected], Ashish Parmanand Pandey [email protected], Ioannis Koutis [email protected], Mengjia Xu [email protected] · 2025
Read Hierarchical Mamba Meets Hyperbolic Geometry: A New Paradigm for Structured Language Embeddings by Sarang Patil [email protected], Ashish Parmanand Pandey [email protected], Ioannis Koutis [email protected], Mengjia Xu [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18973.txt.
Introduction
Sarang Patil and co-authors present a language model that stores word meanings in curved space, in a 2025 paper. Most models place meanings in flat space, which cramps tree-like structure. Language is full of such trees, from grammar to topic kinds. Their model pairs a fast sequence engine with curved geometry that has room for branching. The curve's bend is learned from data, not set by hand.
Meanings are cast into one of two curved homes: a ball form and a smoother sheet form. A loss shaped for curved space trains the whole net end to end. On four data sets from language and medicine, both forms beat flat rivals at tasks that need multi-step links across a tree. The ball form drew finer splits between close kinds. The sheet form stayed more compact and stable. Code is shared in public.
Catalog record
| CREATOR | Sarang Patil [email protected], Ashish Parmanand Pandey [email protected], Ioannis Koutis [email protected], Mengjia Xu [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18973
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Hierarchical Mamba Meets Hyperbolic Geometry: A New Paradigm for Structured Language Embeddings Sarang Patil [email protected], Ashish Parmanand Pandey [email protected], Ioannis Koutis [email protected], Mengjia Xu [email protected] Abstract Selective state-space models excel at long-sequence modeling, but their capacity for language representation – in complex hierarchical reasoning – remains underexplored. Most large language models rely on flat Euclidean embeddings, limiting their ability to capture latent hierarchies. To address this, we propose Hierarchical Mamba (HiM) , integrating efficient Mamba2 with hyperbolic geometry to learn hierarchy-aware language embeddings for deeper linguistic understanding. Mamba2-processed sequences are projected to the Poincaré ball or Lorentzian manifold with ‘‘learnable’’ curvature, optimized with a hyperbolic loss. Our HiM model facilitates the capture of relational distances across varying hierarchical levels, enabling effective long-range reasoning for tasks like mixed-hop prediction and multi-hop inference in hierarchical classification. Experimental results show both HiM variants effectively capture hierarchical relationships across four linguistic and medical datasets, surpassing Euclidean baselines, with HiM-Poincaré providing fine-grained distinctions with higher h-norms, while HiM-Lorentz offers more stable, compact, and hierarchy-preserving embeddings-favoring robustness 1 1 1 The source code is publicly available at https://github.com/BerryByte/HiM . 1 Introduction Large language models (LLMs), such as Transformers ( Vaswani, 2017 ) and BERT ( Devlin et al., 2019 ) , typically encode input sequences into a flat Euclidean space. However, they struggle to capture the hierarchical and tree-like structures inherent in natural language ( Chomsky, 2014 ) , often leading to distortions at different levels of abstraction and specificity ( Nickel & Kiela, 2017 ; Ganea et al., 2018 ) . Moreover, transformer-based encoders face significant computational overhead due to the quadratic complexity of the attention mechanism ( Vaswani, 2017 ) . This limitation becomes particularly evident when dealing with hierarchical data (e.g., text ontologies, brain connectome ( Ramirez et al., 2025 ; Baker et al., 2024 ) ) with exponentially expanding structure. State-space models, starting with the Structured State Space (S4) model ( Gu et al., 2021 ) , have shown exceptional scalability for long-sequence modeling. Mamba’s selective mechanism ( Gu & Dao, 2023 ) dynamically prioritizes relevant information, achieving state-of-the-art performance in tasks with long-range dependencies. Mamba2 refines the original Mamba model for long-range sequence tasks by introducing a duality between state-space computations and attention-like operations, enabling the model to function as either an SSM or a structured, “mask-free” form of attention ( Dao & Gu, 2024 ) . Recently, leveraging hyperbolic geometry as the latent representation space in machine learning models has shown great promise for learning meaningful hierarchical structures ( Nickel & Kiela, 2018 ; Peng et al., 2021 ; Petrovski, 2024 ) . The Poincaré disk and Lorentz model are two prevalent representations of hyperbolic space. The Poincaré disk model is often favored for its conceptual simplicity (bounded in a unit ball). However, the Lorentz model (with unbounded infinite space) offers a closed-form distance function, but requires careful handling of numerical functions dealing with space-like dimensions and a time-like dimension using exponential mapping and logarithmic mapping ( Peng et al., 2021 ) . These numerical considerations are critical because the improper handling of the time-like coordinate in Lorentz models can lead to manifold violations, requiring specialized projection techniques ( Fan et al., 2024 ; Liang et al., 2024 ) . Existing hyperbolic LLM architectures ( He et al., 2024b ; Peng et al., 2021 ) often rely on Transformer blocks and apply a simple Poincaré disk model, leading to O(L^{2}) complexity that becomes prohibitive for long sequences typical in deep hierarchies. A key challenge in implementing hyperbolic models is tuning the curvature to maintain numerical stability, particularly in Lorentz parameterization. In this paper, we introduce hyperbolic mamba with the Lorentz model, and compare it with its counterparts – Poincaré model and Euclidean model . To address the potential numerical instability in the Lorentz model ( Mishne et al., 2023 ) , we explicitly bound the embedding norms and employ curvature-constrained Maclaurin approximations for hyperbolic operations. HiM aims to achieve high-performance hierarchical classification by preserving relational hierarchies. It demonstrates scalability for processing long sequences without compromising on accuracy or computational efficiency. HiM’s novelty lies in integrating a state-space model (Mamba2) with hyperbolic geometry, leveraging Mamba2’s O(L) complexity for efficient sequence modeling while preserving hyperbolic properties for hierarchical representation. Additionally, our HiM incorporates task-specific hyperbolic losses that explicitly enforce parent-child distance constraints in hyperbolic space, enabling end-to-end hierarchy learning without Euclidean biases and achieving significant F1 gains on multi-hop inference tasks. To support HiM’s framework, we introduce SentenceMamba-16M , a compact, Mamba2-based large language model with 16 million parameters designed to generate high-quality sentence embeddings. 2 Related Works Hyperbolic geometry has demonstrated strong potential in modeling hierarchical structures in both shallow and deep neural networks. Foundational works, such as Poincaré embeddings ( Nickel & Kiela, 2017 ) and hyperbolic entailment cones ( Ganea et al., 2018 ) , showed their effectiveness in capturing hierarchical relationships in taxonomies with shallow neural networks. Moreover, hyperbolic manifolds have also been applied to encode hierarchies in graph-structured data ( Liu et al., 2019 ; Chami et al., 2019 ) . More recent efforts have extended hyperbolic representations to multimodal computer vision tasks, including visual and audio modalities ( Yang et al., 2024c ; Mandica et al., 2024 ) , further demonstrating their strength in capturing both hierarchical structure and uncertainty. However, hyperbolic approaches in language modeling remain limited. As an early approach to hyperbolic word embeddings, Dhingra et al. (2018) provided an important early step by reparameterizing Poincaré embeddings for GRU-based sequence modeling. This approach eliminated projection steps and supported both shallow and parametric encoders, but it was ultimately limited by its shallow representations, which restricted its expressive power and ability to capture long-range dependencies. More recent research has extended these concepts to transformers and their variants ( He et al., 2024b ; Chen et al., 2021 ; Chen et al., 2024 ) . These approaches enable effective prediction of subsumption relations and transitive inferences across hierarchy levels using hyperbolic embeddings, providing a principled framework for encoding syntactic dependencies through geodesic distances. However, Hyperbolic BERT exhibits high computational cost than standard BERT due to the complexity of hyperbolic operations ( Chen et al., 2024 ) . To improve efficiency, recent works have explored fine-tuning LLMs directly in hyperbolic space with the Low-Rank Adaptation (LoRA) technique ( Hu et al., 2022 ) . For example, HoRA ( Yang et al., 2024a ) and HypLoRA ( Yang et al., 2024b ) apply LoRA to the hyperbolic manifold, allowing parameter-efficient fine-tuning while capturing complex hierarchies. These methods show strong gains—up to 17.3% over Euclidean LoRA. However, these models usually assume a constant curvature, which may not be optimal for all data, and can suffer from numerical instability due to the exponential and logarithmic mappings required to transition between Euclidean and hyperbolic spaces ( López & Strube, 2020 ; Patil et al., 2025 ) . Limitations in Current Approaches and Our Contribution: Despite significant progress, most existing methods either exploit only partial hyperbolic representations (e.g., using adapters or static embeddings) or rely heavily on attention-based architectures that scale poorly with long sequences and deep hierarchies. For instance, Poincaré GloVe ( Tifrea et al., 2018 ) is limited to word embeddings,