HAM: A Hyperbolic Step to Regulate Implicit Bias
Tom Jacobs, Advait Gadhikar, Celia Rubio-Madrigal, Rebekka Burkholz · 2025
Read HAM: A Hyperbolic Step to Regulate Implicit Bias by Tom Jacobs, Advait Gadhikar, Celia Rubio-Madrigal, Rebekka Burkholz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02630.txt.
Introduction
Jacobs, Gadhikar, Rubio-Madrigal, and Burkholz propose Hyperbolic Aware Minimization, a training method that pushes neural networks toward sparse, leaner models. The head uses this title, which differs from the catalog title. Sparse models can be faster and easier to read, but an older geometry trick that favors sparsity also slows learning near zero. Their paper was published in 2025.
The new method swaps a standard training step with a light mirror step that keeps the good geometry. This swap costs less memory and computing than the older approach. Tests on standard image tasks show steady gains, even when training itself stays dense. Gains grow further when the method is paired with tools that prune models.
Catalog record
| CREATOR | Tom Jacobs, Advait Gadhikar, Celia Rubio-Madrigal, Rebekka Burkholz |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-02630
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HAM: A Hyperbolic Step to Regulate Implicit Bias Tom Jacobs, Advait Gadhikar, Celia Rubio-Madrigal, Rebekka Burkholz Abstract Understanding the implicit bias of optimization algorithms has become central to explaining the generalization behavior of deep learning models. For instance, the hyperbolic implicit bias induced by the overparameterization m ⊙ w m\odot w —though effective in promoting sparsity—can result in a small effective learning rate, which slows down convergence. To overcome this obstacle, we propose HAM (Hyperbolic Aware Minimization), which alternates between an optimizer step and a new hyperbolic mirror step. We derive the Riemannian gradient flow for its combination with gradient descent, leading to improved convergence and a similar beneficial hyperbolic geometry as m ⊙ w m\odot w for feature learning. We provide an interpretation of the the algorithm by relating it to natural gradient descent, and an exact characterization of its implicit bias for underdetermined linear regression. HAM’s implicit bias consistently boosts performance —even of dense training, as we demonstrate in experiments across diverse tasks, including vision, graph and node classification, and large language model fine-tuning. HAM is especially effective in combination with different sparsification methods, improving upon the state of the art. The hyperbolic step requires minimal computational and memory overhead, it succeeds even with small batch sizes, and its implementation integrates smoothly with existing optimizers. 1 Introduction The success of modern deep learning approaches relies on the highly overparameterized regime, which has led to a computationally demanding trend to increase the size of models, and thus the amount of trainable parameters by orders of magnitude ( hoffmann2022training ; kaplan2020scaling ) . A common explanation for this phenomenon are implicit biases that originate from a combination of the optimizer and the overparameterization ( pesme2021implicit ; gunasekar2017implicitregularizationmatrixfactorization ; woodworth2020kernel ) , which regularize the training dynamics and thus boost generalization performance. Training sparse models instead leads to suboptimal performance ( li2017pruning ; Frankle2018TheLT ) . This fact has limited pruning at initialization (PaI) approaches ( synflow ; snip ; random-pruning-vita ) that aim to reduce the heavy computational and memory demands by masking the network before training the remaining parameters. In contrast, state-of-the-art sparsification methods utilize overparameterization in some capacity, as they either iteratively prune parameters in Dense-to-Sparse (DtS) training ( peste2021acdc ; kuznedelev2024cap ; kusupati20 ; jacobs2024maskmirrorimplicitsparsification ; kolb2025deep ) or dynamically explore multiple sparse masks to find high-performing sparse networks with Dynamic Sparse Training (DST) ( evci-rigl ; struct-dst ; chen2021chasing ) . Key observations regarding these algorithms are that a) mild amounts of sparsity and b) longer training with standard optimizers can improve generalization performance significantly. The latter indicates that sparse models are difficult to train and take longer to converge ( Kuznedelev2023AccurateNN ) . Accordingly, standard optimization stands to benefit from overparameterization not only with respect to its generalization performance, but also its convergence speed. Figure 1: The Riemannian metric magnitude \|\mathcal{I}^{-1}(x)\| of HAM is above that of gradient descent (GD), while the overparameterization m\odot w is below it. Therefore, HAM fixes the vanishing artificial learning rate. Note the hyperbolic geometric structure of HAM compared to the flatness of GD. A recent development to improve sparse training is the pointwise overparameterization proposed in PILoT ( jacobs2024maskmirrorimplicitsparsification ) and Sign-In ( gadhikar2025signinlotteryreparameterizingsparse ) . All parameter weights x\in\mathbb{R}^{n} are replaced by a pointwise product of parameters m\odot w , with both m,w\in\mathbb{R}^{n} . This changes the implicit bias of the optimization process and leads to substantial generalization benefits for sparse training. In PILoT, a continuous sparsification method, the overparameterization is used to jointly learn the mask m . Meanwhile, the PaI method Sign-In uses it on the remaining network parameters to increase their plasticity and facilitate sign flips, which has been shown to be a necessary condition for sparse training ( gadhikar2024masks ) . Both methods are state-of-the-art in their respective categories. However, on their own, they fall short of the general baseline methods for sparsity, such as Ac/Dc ( peste2021acdc ) and RiGL ( evci-rigl ) . To understand this gap, we must investigate their training dynamics. The resulting dynamics of the overparameterization m~\odot~w can be derived within the mirror flow ( Li2022ImplicitBO ) or time-dependent mirror flow framework ( jacobs2025mirrormirrorflowdoes ) . It is associated with the hyperbolic mirror map ( woodworth2020kernel ) and, depending on initialization, learning rate, and regularization, it changes from an implicit L_{2} to L_{1} bias during training. The mirror flow induced by m\odot w can also be characterized by a Riemannian gradient flow with an associated metric ( Li2022ImplicitBO ) . This insight allows to establish a connection with natural gradient descent ( Martens2014NewIA ) , which uses either the Fisher or empirical Fisher information as the Riemannian metric. Comparing these metrics highlights a problem: m\odot w suffers from a small artificial learning rate \mathcal{I}^{-1}(x) ( jacobs2024maskmirrorimplicitsparsification ) (see Fig. 1 ), impeding convergence. As an additional consequence, parameters can become stuck at 0 , preventing the ability to effectively sign-flip. Sign-In ( gadhikar2025signinlotteryreparameterizingsparse ) partially mitigates this issue by iteratively re-initializing m and w . However, the remedy is unstable and introduces a hard perturbation to the training dynamics, limiting its positive effects. In this work, we propose to capture the essential structure of these two methods, the implicit bias of pointwise overparameterization, which provably aids in finding generalizable sparse solutions. At the same time, we are able to avoid their drawbacks: their slower convergence and their need for explicit overparameterization that negatively impacts memory and compute. We do this by deriving a plug-and-play hyperbolic optimization step, which we alternate with gradient descent or any other first-order optimizer. Our alternating method is called HAM: Hyperbolic Aware Minimization (§ 3 , 5 ). HAM mitigates the small artificial learning rate and keeps a similar but fully controllable implicit bias than m\odot w , as shown in § 4 for its combination with gradient descent. We evaluate HAM across diverse benchmarks—including vision tasks, graph and node classification, and LLM finetuning—and find it consistently improves generalization, especially in sparse training (§ 6 ). Remarkably, HAM tends to enhance generalization complementary to sharpness aware minimization (SAM) ( foret2021sharpnessaware ) , yet incurs only negligible computational overhead. These improvements can be explained by two major mechanisms: a) It accelerates training around 0 , thus improving sign learning. This is facilitated by its geometry and a larger artificial learning rate. b) The implicit bias towards mild sparsity regularizes training. Both mechanisms boost generalization of sparsification techniques, such as Ac/Dc ( peste2021acdc ) and RiGL ( evci-rigl ) , and even of dense model training. In summary, our contributions are: • HAM step: A plug-and-play optimization update, compatible with first- or higher-order optimizers without incurring significant computational or memory overhead. • We provide a characterization of the training dynamics using natural gradient descent and Riemanian gradient flow. • HAM inherits the geometric benefits of the pointwise m\odot w parameterization in recent sparsity methods, while mitigating their vanishing artificial learning rate problem (see Figure 1 ). • HAM improves strong baseline sparsity methods, including Ac/Dc, RiGL, and STR. • Moreover, it boosts dense training, as shown for vision, graph and node classification, and LLM fine tuning. 2 Related work Sparsification Sparse training methods can be categorized into