AdaFRUGAL: Adaptive Memory-Efficient Training with Dynamic Control

Quang-Hung Bui, Anh Son Ta · 2026

Read AdaFRUGAL: Adaptive Memory-Efficient Training with Dynamic Control by Quang-Hung Bui, Anh Son Ta (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11568.txt.

Introduction

Quang-Hung Bui and Anh Son Ta cut the memory cost of training LLMs. Their 2026 preprint is named AdaFRUGAL, on adaptive training. Models like GPT-4, LLaMA, and BLOOM reshaped language tech. Their immense scale brings a heavy training bill with it.

Memory for training is the exorbitant part named here. Weights, gradients, and optimizer states all pile up. AdaFRUGAL controls memory use dynamically as training runs. Adaptive control can spend memory where it helps most. Frugal steps may let smaller labs train bigger models. The work joins a push for efficient large model training. Lower memory walls widen who can build frontier systems.

Catalog record

CREATORQuang-Hung Bui, Anh Son Ta
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-11568

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AdaFRUGAL: Adaptive Memory-Efficient Training with Dynamic Control Quang-Hung Bui, Anh Son Ta Abstract 1 Introduction The proliferation of Large Language Models (LLMs) such as GPT-4 [ 1 ] , LLaMA [ 2 ] , and BLOOM [ 17 ] has revolutionized Natural Language Processing (NLP). However, their immense scale brings a significant challenge: the exorbitant memory requirements for training. A substantial portion of this memory is not consumed by the model parameters themselves, but by optimizer states, such as the momentum and variance vectors required by popular optimizers like Adam [ 11 ] and AdamW [ 3 ] . For large models, these states can demand upwards of 64GB of GPU memory, creating a significant barrier to entry [ 4 , 12 ] . To address this bottleneck, the FRUGAL framework [ 5 ] recently emerged as a highly effective solution. It introduces a novel "gradient splitting" mechanism that partitions parameters into a small, critical state-full subspace (updated with AdamW) and a large state-free subspace (updated with a memoryless optimizer like SignSGD [ 13 ] ). This hybrid approach enables full-parameter training with performance competitive to AdamW at a fraction of the memory cost. Despite its effectiveness, the original FRUGAL framework relies on two static hyperparameters: the state-full subspace ratio ( \rho ) and its update frequency ( T ). A fixed value for these may be suboptimal across the different phases of training. In this paper, we propose AdaFRUGAL, an adaptive extension that introduces dynamic control mechanisms to automate and optimize resource management. Instead of static values, AdaFRUGAL intelligently adjusts \rho and T based on training progress. Our contributions are threefold: • We introduce a dynamic decay policy for \rho , allowing the model to start with a larger state-full subspace for robust initial learning and progressively reduce it to conserve memory. • We propose a loss-aware adaptive schedule for T , reducing the frequency of expensive subspace updates as the model’s convergence slows, thereby saving training time. 2 Related Work and Background 2.1 Memory-Efficient Optimization Strategies Research on reducing optimizer memory overhead has yielded several distinct strategies. One prominent family is Parameter-Efficient Fine-Tuning (PEFT), which includes methods like LoRA [ 6 ] , Adapter Tuning [ 14 ] , and Prefix-Tuning [ 15 ] . These techniques freeze most model parameters and only train a small number of auxiliary parameters, which is highly efficient but not suitable for full-parameter training. Another line of work enables full-parameter training while minimizing memory. GaLore [ 4 ] projects gradients into a low-rank subspace to store smaller optimizer states. Methods based on Block Coordinate Descent (BCD) [ 16 ] , such as BAdam [ 7 ] , update only a subset of parameters at a time. System-level optimizations like ZeRO [ 20 ] partition optimizer states across multiple GPUs. Orthogonal to these are techniques that directly modify the optimizer, such as 8-bit optimizers [ 19 ] and AdaFactor [ 18 ] , which can often be combined with other frameworks. 2.2 The FRUGAL Framework The FRUGAL framework [ 5 ] , the foundation of our work, enables full-rank, full-parameter updates with minimal memory by employing gradient splitting . At each step, a parameter’s gradient is split into two components. A state-full component, projected onto a low-dimensional subspace, is updated via a stateful optimizer like AdamW. The remaining state-free component is updated via a memoryless optimizer like SignSGD [ 13 ] . This subspace is periodically redefined based on two static hyperparameters: its size, determined by the ratio \rho , and its update frequency, T . FRUGAL’s mechanism of resetting optimizer states upon subspace change avoids the staleness issue seen in prior work, ensuring training stability. 3 The AdaFRUGAL Framework While the original FRUGAL framework provides a robust baseline for memory-efficient training, its reliance on static hyperparameters \rho and T limits its adaptability. The optimal values for these parameters can vary significantly not only between different tasks and models but also across different phases of a single training run. To address this limitation, we introduce AdaFRUGAL, which incorporates two dynamic control mechanisms to automate resource management and enhance training efficiency. 3.1 Dynamic State-Full Ratio Control (Dynamic \rho ) Motivation: In the early stages of training, a model undergoes rapid and significant weight updates as it learns fundamental features from the data. During this period, a larger state-full subspace (higher \rho ) is beneficial, as it allows the powerful AdamW optimizer to stabilize and accelerate convergence. However, as training progresses and the model converges, updates become smaller and more focused on fine-tuning. In this later phase, maintaining a large state-full subspace becomes memory-inefficient, as the simpler SignSGD optimizer is often sufficient to handle these smaller adjustments. A static \rho fails to capture this dynamic, leading to either insufficient adaptation early on or wasted memory later. Mechanism: To address this, we introduce a dynamic control policy for \rho based on a simple yet effective linear decay schedule. The state-full ratio at a given training step k , denoted as \rho(k) , is calculated as follows: \rho(k)=\max\left(\rho_{\text{end}},\rho_{\text{start}}-(\rho_{\text{start}}-\rho_{\text{end}})\cdot\frac{k}{K_{\text{total\_steps}}}\right) (1) where \rho_{\text{start}} is the initial ratio, \rho_{\text{end}} is the final target ratio, k is the current training step, and K_{\text{total\_steps}} is the total number of training steps. This mechanism allows AdaFRUGAL to begin with a large, robust state-full subspace and gradually reduce its size, continuously optimizing the trade-off between performance and memory usage throughout the training process. 3.2 Adaptive Update Frequency Control (Dynamic T ) Motivation: The frequency T at which the state-full subspace is redefined represents a trade-off between adaptability and computational cost. Frequent updates (small T ) allow the framework to quickly adapt to changing gradient dynamics but incur significant computational overhead from re-projecting gradients and resetting optimizer states. Infrequent updates (large T ) are more efficient but risk using a "stale" subspace that no longer captures the most important update directions. This trade-off is also phase-dependent: frequent updates are more critical early in training, while less frequent updates are sufficient once the model has stabilized. Mechanism: We propose a loss-aware adaptive schedule to dynamically adjust T . The core idea is to increase the update interval when the model’s convergence plateaus. Specifically, we monitor the validation loss periodically. Every N_{\text{eval}} steps, we calculate the relative change in validation loss, \Delta L_{\text{rel}} , over the last evaluation window as follows: \Delta L_{\text{rel}}=\frac{|L_{\text{val}}(k-N_{\text{eval}})-L_{\text{val}}(k)|}{L_{\text{val}}(k-N_{\text{eval}})} (2) where L_{\text{val}}(k) is the validation loss at step k . If this relative change falls below a predefined stability threshold \tau_{\text{low}} , we deem the training to be stable and increase T : T_{\text{new}}=\min(T_{\text{max}},T_{\text{current}}\cdot\gamma_{\text{increase}}) (3) where T_{\text{max}} is a maximum allowed frequency and \gamma_{\text{increase}} is an increase factor. This mechanism allows AdaFRUGAL to reduce computational overhead automatically in later training stages without manual intervention. 3.3 The Integrated AdaFRUGAL Algorithm By integrating both dynamic control mechanisms, we formulate the complete AdaFRUGAL algorithm, as detailed in Algorithm 1 . The algorithm dynamically computes the state-full ratio \rho_{k} and adjusts the update frequency T_{k} at each step, using them to periodically redefine the state-full projection P_{k} . The core gradient splitting and hybrid update steps from the original FRUGAL are then executed within this adaptive context. Algorithm 1 The Integrated AdaFRUGAL Algorithm 1: Input: Model f_{\theta}

Read the complete text