Fast and Geometrically Grounded Lorentz Neural Networks
Robert van der Klis, Ricardo Chávez Torres, Max van Spengler, Yuhui Ding, Thomas Hofmann, Pascal Mettes · 2026
Read Fast and Geometrically Grounded Lorentz Neural Networks by Robert van der Klis, Ricardo Chávez Torres, Max van Spengler, Yuhui Ding, Thomas Hofmann, Pascal Mettes (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21529.txt.
Introduction
Robert van der Klis, Ricardo Chávez Torres, Max van Spengler, Yuhui Ding, Thomas Hofmann, and Pascal Mettes build fast neural networks that live in hyperbolic space. That curved space suits data with tree-like structure, such as categories and family trees. A sound math base for networks in this space has been slow to arrive.
The authors give a formulation that is both efficient and faithful to the geometry. Their networks use the Lorentz model, prized for numerical stability. Fast, correct hyperbolic layers could lift tasks from language hierarchies to recommendation, where tree shapes are everywhere but flat models flatten them away.
Catalog record
| CREATOR | Robert van der Klis, Ricardo Chávez Torres, Max van Spengler, Yuhui Ding, Thomas Hofmann, Pascal Mettes |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-21529
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Fast and Geometrically Grounded Lorentz Neural Networks Robert van der Klis, Ricardo Chávez Torres, Max van Spengler, Yuhui Ding, Thomas Hofmann, Pascal Mettes Abstract Hyperbolic space is quickly gaining traction as a promising geometry for hierarchical and robust representation learning. A core open challenge is the development of a mathematical formulation of hyperbolic neural networks that is both efficient and captures the key properties of hyperbolic space. The Lorentz model of hyperbolic space has been shown to enable both fast forward and backward propagation. However, we prove that, with the current formulation of Lorentz linear layers, the hyperbolic norms of the outputs scale logarithmically with the number of gradient descent steps, nullifying the key advantage of hyperbolic geometry. We propose a new Lorentz linear layer grounded in the well-known “distance-to-hyperplane” formulation. We prove that our formulation results in the usual linear scaling of output hyperbolic norms with respect to the number of gradient descent steps. Our new formulation, together with further algorithmic efficiencies through Lorentzian activation functions and a new caching strategy results in neural networks fully abiding by hyperbolic geometry while simultaneously bridging the computation gap to Euclidean neural networks. Code available at: https://github.com/robertdvdk/hyperbolic-fully-connected . † † affiliationnotice: Equal contribution (a) Our Lorentz linear layer. (b) 6 ’s ( 6 ) Lorentz linear layer. Figure 1 : Visualizing decision boundaries in Lorentz model, with level curves projected on the Poincaré disk. The black line denotes a class boundary. The blue and red colors denote positive and negative distances to the decision boundary. (a) : We propose a formulation of a linear layer in the Lorentz model which ensures that the decision boundary forms a proper hyperbolic hyperplane. (b) : The current leading formulation of the Lorentzian linear layer, where intersections do not form valid hyperbolic hyperplanes in general, leading to unwanted configurations such as in the upper right and lower examples, as well as a lack of exponential growth of distances. 1 Introduction Hyperbolic geometry has garnered significant interest in recent years in deep learning. Amongst others, hyperbolic manifolds hold promise for embedding hierarchical structures with arbitrarily low distortion ( 26 ) . Hierarchical structures are ubiquitous: from natural language to computer vision and graph tasks, all either have inherent hierarchical organizations or benefit from hierarchical knowledge. In semantics, the relations between word semantics are hierarchical, which is important for word embeddings ( 22 ; 28 ) and for language models ( 23 ; 12 ; 32 ) . In computer vision, hyperbolic learning improves hierarchical visual recognition ( 15 ; 16 ; 31 ) , robustness ( 30 ; 2 ) , few-shot learning ( 18 ; 21 ; 10 ) , and more ( 19 ; 13 ) . In graph tasks, “hierarchy is a central organizing principle of complex networks” ( 8 ) , with hyperbolic graph networks as powerful and broadly applicable tools ( 4 ; 17 ; 7 ) . However, building fully hyperbolic neural networks is not straightforward: much of deep learning relies on the definition of the fully connected layer, which serves as the basic building block for more complex components such as convolutions, attention mechanisms, or graph message-passing operations. In past years, seminal works by 9 and 27 formulated a hyperbolic linear layer in the Poincaré model. While theoretically sound, operating in the Poincaré model is computationally inefficient ( 6 ; 20 ) . As a consequence, several recent works started operating in the Lorentz model of hyperbolic space ( 6 ; 1 ; 33 ) as a computationally much more efficient alternative. In this work, however, we find that the current formulation of hyperbolic layers in the Lorentz model is inconsistent with the geometric interpretation of Euclidean layers, interestingly resulting in a loss of precisely the efficient hierarchical embedding property of hyperbolic geometry that we want to encapsulate in our neural networks, see Figure 1 . This motivates us to propose a new formulation of the foundational building block of deep learning in the Lorentz model of hyperbolic space. We propose a new Lorentz linear layer. We strive to get the best out of the theoretical soundness of the Poincaré model formulation and the computational efficiency of the Lorentz model. We make the following contributions: 1. Proof of the pathology: We prove that the hyperbolic norms of embeddings generated by current formulations of Lorentz linear layers scale logarithmically rather than linearly with the number of gradient steps. We demonstrate this behaviour on a toy experiment. 2. Geometric unification: We formulate a new Lorentz linear layer based on the “distance-to-hyperplane” formulation used in Poincaré models, and we prove that this new layer resolves the pathology. 3. Algorithmic efficiency: We introduce Lorentzian activations, a family of activation functions that simplify the computation of our layer. We also introduce a way to cache our parameters, yielding inference speeds of our Lorentz linear layer that are 2.9x as fast as the current state-of-the-art Lorentz neural networks, and 8.3x as fast as current state-of-the-art Poincaré neural networks. 4. FGG-LNN: We use all these building blocks to create Fast and Geometrically Grounded Lorentz Neural Networks (FGG-LNN). We show that this new formulation, together with a well-optimized implementation, speeds up training times of a ResNet-18 over state-of-the-art Lorentz neural networks by 3.5x, and over state-of-the-art Poincaré neural networks by 7.5x, closing the gap towards Euclidean networks. 2 Preliminaries A Riemannian manifold is a tuple (M,g) , where M is a smooth manifold and g is a Riemannian metric, defining a smooth inner product g_{x}(\cdot,\cdot)=\langle\cdot,\cdot\rangle_{x} on the tangent space T_{x}M at each point x\in M . The metric induces a distance function d_{M}(x,y) , which is the length of the shortest path (geodesic) connecting x,y\in M . We use the Lorentz model (or hyperboloid model) of hyperbolic geometry. Let \mathbb{R}^{D+1} be the ambient space (Minkowski space). For any vector \mathbf{x}\in\mathbb{R}^{D+1} , we separate the time component and the space components and denote them as \mathbf{x}=(x_{1},\overline{\mathbf{x}}) , where x_{1}\in\mathbb{R} and \overline{\mathbf{x}}\in\mathbb{R}^{D} . This space is equipped with the Lorentz inner product denoted \circ , with Minkowski metric \mathbf{I}_{1,D}=\text{diag}(-1,1,\dots,1) : \mathbf{x}\circ\mathbf{y}=-x_{1}y_{1}+\overline{\mathbf{x}}^{\top}\overline{\mathbf{y}}=\mathbf{x}^{\top}\mathbf{I}_{1,D}\mathbf{y}. (1) The hyperbolic space \mathbb{L}_{\kappa}^{D} with constant negative curvature -\kappa ( \kappa>0 ) is defined as the upper sheet of a hyperboloid: \mathbb{L}_{\kappa}^{D}=\left\{\mathbf{z}\in\mathbb{R}^{D+1}\mid\mathbf{z}\circ\mathbf{z}=-1/\kappa,\,z_{1}>0\right\}. (2) The geodesic distance between any two points in \mathbb{L}_{\kappa}^{D} is: d(\mathbf{x},\mathbf{y})=\frac{1}{\sqrt{\kappa}}\text{arccosh}(-\kappa(\mathbf{x}\circ\mathbf{y})). (3) The tangent space T_{\mathbf{z}}\mathbb{L}_{\kappa}^{D} at a point \mathbf{z} is the set of vectors Minkowski-orthogonal to \mathbf{z} : T_{\mathbf{z}}\mathbb{L}_{\kappa}^{D}=\{\mathbf{v}\in\mathbb{R}^{D+1}\mid\mathbf{z}\circ\mathbf{v}=0\}. (4) We define the origin of the manifold as \mathbf{o}=(\frac{1}{\sqrt{\kappa}},0,\dots,0) . For learning on manifolds, we require maps to transition between the manifold and its tangent spaces, as well as a way to move vectors across the manifold, specifically the exponential map from tangent space to manifold ( \text{Exp}_{\mathbf{x}}:T_{\mathbf{x}}\mathbb{L}_{\kappa}^{D}\to\mathbb{L}_{\kappa}^{D} ), the logarithmic map for the reverse ( \text{Log}_{\mathbf{x}}:\mathbb{L}_{\kappa}^{D}\to T_{\mathbf{x}}\mathbb{L}_{\kappa}^{D} ), and the parallel transport which transports a tangent vector \mathbf{v} along the geodesic from \mathbf{x} to \mathbf{y} while preserving the inner product ( \Gamma_{\mathbf{x}\to\mathbf{y}}:T_{\mathbf{x}}\mathbb{L}_{\kappa}^{D}\to T_{\mathbf{y}}\mathbb{L}_{\kappa}^{D} ). We refer to Appendix A for full definitions. There are several different (pseudo)norms we will use in this paper. The first is the Euclidean norm: we will denote this by \|\mathbf{v}\|_{E}^{2}=\mathbf{v}\cdot\mathbf{v} . Analogous to the Euclidean case, we define the pseudonorm \|\mathbf{v}\|_{\mathcal{L}}^{2}=\mathbf{v}\circ\mathbf{v} (not positive definite). When \mathbf{v}\in T_{\mathbf{z}}\mathbb{L}_{\kappa}^{D} ,