On the Duality between Gradient Transformations and Adapters

Lucas Torroba-Hennigen, Hunter Lang, Han Guo, Yoon Kim · 2025

Read On the Duality between Gradient Transformations and Adapters by Lucas Torroba-Hennigen, Hunter Lang, Han Guo, Yoon Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13811.txt.

Description

Lucas Torroba-Hennigen, Hunter Lang, Han Guo, and Yoon Kim study ways to train neural networks with less memory. Their 2025 paper deals with methods that shrink gradients into a smaller space before the update step. That shrink saves the memory used to store gradients and optimizer state during training.

The authors prove that training in the shrunk space equals a second route: add a simple linear adapter to the model and train only the adapter. When the shrink uses a Kronecker form, the link ties two known methods, GaLore and one-sided LoRA, into one picture. The result unifies past tricks and suggests fresh ways to cut memory in large model training.

Catalog record

CREATORLucas Torroba-Hennigen, Hunter Lang, Han Guo, Yoon Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-13811

Browse the full library · About the Tower of Babel