Study of Training Dynamics for Memory-Constrained Fine-Tuning
Aël Quélennec, Nour Hezbri, Pavlo Mozharovskyi, Van-Tam Nguyen, Enzo Tartaglione · 2025
Read Study of Training Dynamics for Memory-Constrained Fine-Tuning by Aël Quélennec, Nour Hezbri, Pavlo Mozharovskyi, Van-Tam Nguyen, Enzo Tartaglione (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19675.txt.
Description
Aël Quélennec and four co-authors propose TraDy, a fine-tuning scheme for training large neural networks under strict memory limits. As models grow, the memory needed to train them outruns many deployment environments. TraDy rests on two observations: which layers deserve updates depends on the architecture and can be decided in advance, and gradient estimates improve when channels are chosen dynamically rather than fixed.
Within the preselected layers, channels are resampled stochastically between epochs. Across tasks and architectures the method reached state-of-the-art accuracy while keeping almost all activations inactive, most weight derivatives unused, and the computing cost of weight derivatives cut by about ninety-seven percent. It offers a practical route to adapting big models on modest hardware.
Catalog record
| CREATOR | Aël Quélennec, Nour Hezbri, Pavlo Mozharovskyi, Van-Tam Nguyen, Enzo Tartaglione |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-19675