Memory Constrained Dynamic Subnetwork Update for Transfer Learning

Aël Quélennec, Pavlo Mozharovskyi, Van-Tam Nguyen, Enzo Tartaglione · 2025

Read Memory Constrained Dynamic Subnetwork Update for Transfer Learning by Aël Quélennec, Pavlo Mozharovskyi, Van-Tam Nguyen, Enzo Tartaglione (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20979.txt.

Introduction

Aël Quélennec leads three co-authors on MeDyate, a framework for fine tuning neural networks on devices with tight memory limits. On device training would let pretrained models adapt locally, but memory caps block updating whole networks.

A layer ranking metric pre selects which layers matter most. A dynamic channel sampling plan exploits the stability of channel importance during fine tuning, resampling channels between epochs by importance weighted chance. Theory grounds the choices, and comprehensive coverage of salient channels is maintained within the memory budget. The approach suits phones and embedded gear that must personalize without cloud help.

Catalog record

CREATORAël Quélennec, Pavlo Mozharovskyi, Van-Tam Nguyen, Enzo Tartaglione
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-20979

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Memory Constrained Dynamic Subnetwork Update for Transfer Learning Aël Quélennec, Pavlo Mozharovskyi, Van-Tam Nguyen, Enzo Tartaglione Abstract On-device neural network training faces critical memory constraints that limit the adaptation of pre-trained models to downstream tasks. We present MeDyate, a theoretically-grounded framework for memory-constrained dynamic subnetwork adaptation. Our approach introduces two key innovations: LaRa (Layer Ranking), an improved layer importance metric that enables principled layer pre-selection, and a dynamic channel sampling strategy that exploits the temporal stability of channel importance distributions during fine-tuning. MeDyate dynamically resamples channels between epochs according to importance-weighted probabilities, ensuring comprehensive parameter space exploration while respecting strict memory budgets. Extensive evaluation across a large panel of tasks and architectures demonstrates that MeDyate achieves state-of-the-art performance under extreme memory constraints, consistently outperforming existing static and dynamic approaches while maintaining high computational efficiency. Our method represents a significant step towards enabling efficient on-device learning by demonstrating effective fine-tuning with memory budgets as low as a few hundred kB of RAM. 1 Introduction The exponential growth of deep neural networks has fundamentally transformed artificial intelligence, enabling breakthrough performances across diverse domains including Computer Vision krizhevsky2012imagenet ; simonyan2014very ; minaee2021image , Natural Language Processing vaswani2017attention ; tenney2019bert , and Speech Recognition deng2013new ; nassif2019speech . This remarkable progress stems from the continuous scaling of model complexity, with the number of parameters doubling every 8 to 17 months since the advent of the ”Large Scale Era” marked by AlphaGo’s release in 2015 sevilla2022compute . While this trend further demonstrates the intrisic generalization potential of deep learning, it raises profound ecological and technical concerns. Training and exploitation of these architectures require very high energy consumption, and their deployment in real-world environments is impossible without extensive compression, leading to performance degradation and creating an increasingly critical tension between model capabilities and practical deployment constraints. The proliferation of Internet of Things (IoT) devices and the growing demand for Edge AI applications have intensified the need for efficient on-device solutions. Traditional approaches to edge deployment follow a paradigm of offline training on powerful hardware, followed by model compression and deployment for inference-only applications on resource-constrained devices. This research domain encompasses five principal methodologies, namely quantization techniques for precision reduction, low-rank decomposition for parameter compression, architectural innovations for compact model design, knowledge distillation for teacher-student learning, and network sparsification (commonly referred to as pruning) for structural optimization 8253600 ; deng2020model . However, this paradigm suffers from fundamental limitations as models trained on static datasets inevitably experience performance degradation when deployed in dynamic real-world environments due to data drift phenomena sahiner2023data . The inability to adapt and learn continuously after deployment limits the practical viability of edge AI systems, particularly in applications requiring personalization, privacy preservation, and real-time adaptation to evolving data distributions incel2023device . The primary obstacle preventing widespread adoption of on-device learning lies in the prohibitive computational and memory demands of backpropagation, as it requires storing intermediate activations and computing gradients, leading to memory requirements that can exceed device capabilities by several orders of magnitude. This has motivated exploration of alternative learning paradigms, including the Forward-Forward algorithm hinton2022forward , hyperdimensional computing yang2023device , and gradient-free optimization methods pau2023suitability . However, these approaches consistently underperform compared to traditional backpropagation-based techniques, creating a performance-efficiency trade-off that has yet to be satisfactorily resolved. Recent advances in memory-constrained transfer learning have demonstrated that strategic subnetwork selection offers a promising path toward bridging this performance gap. Pioneering work by lin2022device showed that fine-tuning within extreme memory budgets (256kB) is feasible through static subnetwork pre-selection. Improving on this concept, Quélennec et al. quélennec2025studytrainingdynamicsmemoryconstrained propose to study the Tra ining Dy namics (TraDy) of network fine-tuning, thus yielding an approach that leverages the heavy-tailed behavior of stochastic gradients and the architectural consistency of layer importance across downstream tasks to dynamically resample channels between epochs, within pre-selected layers. Their method demonstrates that the synergistic combination of strategic layer pre-selection and dynamic channel selection enables state-of-the-art performance even under extreme memory constraints. Activation compression represents an orthogonal approach to memory-efficient training that targets the storage of intermediate representations rather than parameter selection, as demonstrated by Nguyen et al. nguyen2025beyond who compress activations stored for backpropagation. Despite achieving considerable memory compression and computational speedup, this method does not address the fundamental question of which network components to train, defaulting to a heuristic that updates only the final layers. Building upon Quélennec et al. insights, this work presents a comprehensive extension of the TraDy strategy in the form of Me mory-constrained Dy namic subnetwork upd ate (MeDyate). Our main contributions can be summarized as follows. • We propose an improved layer ranking by leveraging the the heavy-tailed behavior of stochastic gradient (Sec. 3.2 ). • We demonstrate that, for a given downstream task, the channel topology remains stable during training (Sec. 3.3 ). • Given memory constraints, only a subset of relevant channels can be updated at a time. By dynamically adjusting the channels updated between epochs, performance improves compared to static approaches. We propose an adaptive algorithm that dynamically samples channels within key layers, assigning update probabilities based on their importance (Sec. 3.4 ). • We test MeDyate in typical transfer learning setups, under extreme memory constraints, observing that it can achieve state-of-the-art performance across multiple efficient architectures (Sec. 4.3 ). Our approach allows us to drastically reduce FLOPs, weight and activation memory during training while demonstrating superior performance compared to other similar strategies. 2 Related Works Activation Map Compression. A critical observation for memory-efficient training is that activation maps (the outputs of each layer after non-linearity application) occupy significantly more memory space than parameters during backpropagation, as they are essential for computing weight derivatives cai2020tinytl . This insight has motivated a dedicated research direction focused on compressing activation maps using techniques adapted from weight compression literature, including quantization 10446393 ; pan2021mesa , sparsification pmlr-v119-kurtz20a ; jiang2022back , entropy encoding georgiadis2019accelerating , wavelet transform methods NEURIPS2022_81f19c0e and most notably, Nguyen et al. ’s application of tensor decomposition strategies, allowing for drastic reduction of memory usage alongside acceleration of the training process nguyen2025beyond . While MeDyate is primarily designed as a strategy for surgically selecting which parameters to fine-tune, it naturally induces activation sparsity, thereby contributing to this research domain as well. Gradient Pruning. The concept of gradient pruning describes the selection of a specific subnetwork to train during backpropagation, while the remainder of the network remains frozen. It differs from classical pruning by maintaining the complete network architecture intact during inference, while selectively modifying the backpropagation phase through criterion-based gradient computation. Gradient pruning can either be applied to reduce the memory footprint in constrained environments, or to accelerate training while preserving on-task performance zhang2024gradientbasedparameterselectionefficient ; bragagnolo2022update ; li2023scotti ; ye2020accelerating ; mcdanel2022accelerating . Within the transfer learning context, Lee et al. lee2022surgical provide particularly relevant insights by framing gradient pruning as a regularization mechanism, demonstrating that network blocks exhibit task-dependent contributions to downstream performance. In their work, they observe that such contribution can either be constructive or destructive with respect to the task and is predicted by the ration of gradient norm to parameter norm. On-Device Learning. Regarding our design of the MeDyate algorithm, three lines of work have largely influenced our approach, each presenting strategies

Read the complete text