On the Duality between Gradient Transformations and Adapters
Lucas Torroba-Hennigen, Hunter Lang, Han Guo, Yoon Kim · 2025
Read On the Duality between Gradient Transformations and Adapters by Lucas Torroba-Hennigen, Hunter Lang, Han Guo, Yoon Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13811.txt.
Description
Lucas Torroba-Hennigen, Hunter Lang, Han Guo, and Yoon Kim study ways to train neural networks with less memory. Their 2025 paper deals with methods that shrink gradients into a smaller space before the update step. That shrink saves the memory used to store gradients and optimizer state during training.
The authors prove that training in the shrunk space equals a second route: add a simple linear adapter to the model and train only the adapter. When the shrink uses a Kronecker form, the link ties two known methods, GaLore and one-sided LoRA, into one picture. The result unifies past tricks and suggests fresh ways to cut memory in large model training.
Catalog record
| CREATOR | Lucas Torroba-Hennigen, Hunter Lang, Han Guo, Yoon Kim |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-13811