Parameter Efficiency Is Not Memory Efficiency: Rethinking Fine-Tuning for On-Device LLM Adaptation
Irene Tenison, Stella Ahn, Miriam Kim, Ebtisam Alshehri, Lalana Kagal · 2026
Read Parameter Efficiency Is Not Memory Efficiency: Rethinking Fine-Tuning for On-Device LLM Adaptation by Irene Tenison, Stella Ahn, Miriam Kim, Ebtisam Alshehri, Lalana Kagal (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22783.txt.
Introduction
Tenison and four colleagues cut the memory cost of fine-tuning language models with LARS, which adapts models inside a sequence-level low-rank subspace instead of storing token-level adapter activations. The approach shrinks stored activation intermediates so their size no longer grows with sequence length in the same way. It lowers memory use by about 23 percent on GPUs while keeping performance competitive with LoRA across reasoning, retrieval, and long-context tasks.
Catalog record
| CREATOR | Irene Tenison, Stella Ahn, Miriam Kim, Ebtisam Alshehri, Lalana Kagal |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51795 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.22783
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Parameter Efficiency Is Not Memory Efficiency: Rethinking Fine-Tuning for On-Device LLM Adaptation Irene Tenison, Stella Ahn, Miriam Kim, Ebtisam Alshehri, Lalana Kagal Abstract Parameter-Efficient Fine-Tuning (PEFT) has become the standard for adapting large language models (LLMs). In this work we challenge the wide-spread assumption that parameter efficiency equates memory efficiency and on-device adaptability. We show that this is not true - while methods like LoRA 9 and IA3 16 significantly reduce trainable parameters, they remain bound by intermediate tensors that scale linearly with sequence length, often triggering out-of-memory errors on-device. In this work, we introduce LARS (Low-memory Activation-Rank Subspace), a novel adaptation framework that decouples memory consumption from sequence length. While prior PEFT methods apply low-rank constraints to model parameters, LARS instead constrains the activation subspace used during training, directly targeting the dominant source of memory consumption and fundamentally flattening the memory growth rate. LARS reduces the memory footprint by an average of 33.54% on GPUs and 51.95% on CPUs in comparison to LoRA across reasoning, understanding and long-context datasets using different models while maintaining competitive accuracy and throughput. Besides GPUs, we deploy on Raspberry Pi and consumer-grade CPUs to demonstrate that LARS provides a scalable path for sophisticated LLM personalization on resource-constrained hardware and edge devices. 1 Introduction Large language models (LLMs) and transformer-based architectures have become the backbone of modern natural language processing 25 ; 23 . While these models exhibit remarkable zero-shot capabilities, fine-tuning remains essential for specialized tasks, privacy-preserving local adaptation, and low-latency personalization in mobile and edge environments 26 . However, the high memory requirements of LLM adaptation pose a significant barrier to deployment on resource-constrained hardware, where available memory is often restricted to a few gigabytes. To address these constraints, the research community has pivoted toward Parameter-Efficient Fine-Tuning (PEFT) 7 . Methods such as LoRA 9 , prefix tuning 14 , and IA3 16 update only a minute fraction of the model’s total weights—often by orders of magnitude—while maintaining competitive downstream accuracy. This success has solidified a fundamental design assumption: that reducing the number of trainable parameters directly translates to improved deployability in memory-limited environments. Figure 1: Accuracy vs. peak memory (GB) for state-of-the-art PEFT methods. Bubble size represents the count of trainable parameters. Our analysis reveals a critical disconnect: trainable-parameter count is a poor proxy for actual memory footprint. In this work, we challenge this assumption. As illustrated in Figure 1 , there is a striking lack of correlation between a method’s parameter efficiency and its actual physical memory footprint during adaptation. For instance, IA3 —one of the most parameter-efficient methods—requires significantly more peak memory than LoRA, despite having fewer trainable weights. A similar pattern was observed with gradient checkpointing (GC) as shown in Figure 10 . We argue that the prevailing focus on parameter count overlooks the primary bottleneck of on-device adaptation: intermediate activation storage. Because most PEFT methods leave the forward computational graph largely unchanged, they still incur massive activation overhead that scales with batch size and sequence length, regardless of how few parameters are updated 24 . Consequently, reducing parameter count alone provides diminishing returns when activation memory dominates the peak footprint 15 . To address this limitation, we propose LARS, an adaptation module designed to reduce activation memory during adaptation. Our key findings are: • We argue that on-device adaptation efficiency should be evaluated based on peak memory, rather than solely on the number of trainable parameters. • Low-rank activation subspace (LARS) performs adaptation in a sequence-pooled, low-rank subspace, reducing the size of stored activations during backpropagation. • Across multiple models and tasks, LARS reduces peak training memory by an average of 33.54% on GPUs and 51.95% on CPUs while maintaining competitive accuracy and throughput with state-of-the-art PEFT approaches. Figure 2: Peak memory scaling vs. sequence length. (Left) Without GC, LARS reduces the memory growth rate by at least 31.82%. (Right) Even with GC enabled, LARS defines a more efficient Pareto frontier, enabling the fine-tuning of sequences longer than LoRA within the same hardware constraints. 2 Background Adapting LLMs on-device requires understanding the runtime memory footprint beyond parameter counts. Peak memory is dominated by: M_{peak}\approx M_{params}+M_{grads}+M_{opt}+M_{acts} where M_{params},M_{grads},M_{opt},\text{ and }M_{acts} denote the memory consumed by model parameters, \theta (including frozen weights), gradients of trainable parameters, optimizer states, and activations respectively where M_{params}=\mathcal{O}(|\theta|) and M_{grads}=M_{opt}=\mathcal{O}(|\theta_{trainable}|) . However activation memory is beyond parameter count. For transformer models with depth L , hidden dimension H , batch size B and sequence length S , activations scale at least as M_{acts}=\mathcal{O}(BSHL) . This reveals a structural asymmetry: parameter-related terms scale with model size, while activation memory scales with data-dependent dimensions (B,S) in addition to model size. 2.1 Memory Consumption in PEFT Methods PEFT methods operate in the regime where the trainable parameter fraction \rho=\frac{|\theta_{trainable}|}{|\theta|}<<1 9 ; 7 . Here, parameter-related terms vanish and M_{grads} and M_{opt} become negligible. However, minimizing \theta_{trainable} alone does not reduce peak memory usage. The dominant term, activations, persists regardless of \rho and scales at least as M_{acts}=\mathcal{O}(BSHL) . This bottleneck persists because most PEFT methods preserve the full token-level computational graph. Consider a representative low-rank update (LoRA) 9 : Wx+ABx , where A\in\mathbb{R}^{H\times R},B\in\mathbb{R}^{R\times H},\text{ and }R<<H . Although trainable parameters scale with R , the input must still be retained in memory to compute \nabla A and \nabla B . Consequently, for any adapter that preserves token-level hidden representations, M_{\text{acts}}=\underbrace{\mathcal{O}(BSHL)}_{\text{Base Activations}}+\underbrace{\mathcal{O}(BSRL)}_{\text{Adapter Activations}} . Standard PEFT methods remain bound by a "Sequence Length Ceiling" because their adapter activations scale as \mathcal{O}(BSRL) . While system-level optimizations like Gradient Checkpointing (GC) 3 and FlashAttention 24 reduce absolute memory constants, they do not alter this linear dependence on sequence length, which LARS does by decoupling adapter activations from S , providing a scalable Pareto frontier for memory-constrained hardware (more details in Appendix A.4 ). 3 Methods We now present our memory-centric adaptation framework. Building on the analysis in Section 2 , we formalize the design questions that motivate this method and describe the method in detail. 3.1 Rethinking PEFT Objectives for On-Device Adaptation While in-cloud training prioritizes parameter reduction for model sharding and throughput, on-device adaptation is strictly constrained by the peak memory spike. Unlike parameter memory, which is a static one-time cost, activation memory is a dynamic penalty that grows with every additional token of context, that standard PEFT fails to waive. This discrepancy leads to our research questions: Q1: Beyond Parameter Sparsity. Can adaptation methods be designed to target the true hardware bottleneck—activations—rather than just trainable parameters? As shown in Figure 1 , even methods with negligible |\theta_{trainable}| can exceed device limits due to token-level hidden states. Q2: Sequence-Decoupled Adaptation. Is it possible to restructure the adaptation module such that memory-dominant hidden states are never fully stored? We hypothesize that for many tasks, a pooled representation can maintain semantic expressivity without preserving the full [B,S,R] backward-pass graph. LARS addresses these questions by shifting the design goal from parameter reduction to activation-aware memory efficiency. Problem Setting: Let M be a pre-trained Transformer with L layers and hidden dimension H . For an input sequence X\in\mathbb{R}^{B\times S\times H} , standard PEFT methods (e.g., LoRA) compute an update \Delta X=f(X;\theta_{trainable}) . The primary memory overhead in these methods arises from the adapter-specific