ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning

Xianming Li, Zongxi Li, Tsz-fung Andrew Lee, Jing Li, Haoran Xie, Qing Li · 2026

Read ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning by Xianming Li, Zongxi Li, Tsz-fung Andrew Lee, Jing Li, Haoran Xie, Qing Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19254.txt.

Introduction

Li and colleagues propose ShadowPEFT, a fine-tuning approach that gathers adaptation into a separate shadow component. It is built around a compact shadow model and light coupling modules for each transformer layer. A lasting shadow state refines the frozen backbone and is updated from it in turn. Since the shadow model learns to predict on its own, it can run alone without the base model. It matches or beats LoRA and DoRA on text and image tests at similar trainable sizes.

Catalog record

CREATORXianming Li, Zongxi Li, Tsz-fung Andrew Lee, Jing Li, Haoran Xie, Qing Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46236
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.19254

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning Xianming Li, Zongxi Li, Tsz-fung Andrew Lee, Jing Li, Haoran Xie, Qing Li Abstract Parameter-efficient fine-tuning (PEFT) reduces the training cost of full-parameter fine-tuning for large language models (LLMs) by training only a small set of task-specific parameters while freezing the pretrained backbone. However, existing approaches, such as Low-Rank Adaptation (LoRA), achieve adaptation by inserting independent low-rank perturbations directly to individual weights, resulting in a local parameterization of adaptation. We propose ShadowPEFT, a centralized PEFT framework that instead performs layer-level refinement through a depth-shared shadow module. At each transformer layer, ShadowPEFT maintains a parallel shadow state and evolves it repeatedly for progressively richer hidden states. This design shifts adaptation from distributed weight-space perturbations to a shared layer-space refinement process. Since the shadow module is decoupled from the backbone, it can be reused across depth, independently pretrained, and optionally deployed in a detached mode, benefiting edge computing scenarios. Experiments on generation and understanding benchmarks show that ShadowPEFT matches or outperforms LoRA and DoRA under comparable trainable-parameter budgets. Additional analyses on shadow pretraining, cross-dataset transfer, parameter scaling, inference latency, and system-level evaluation suggest that centralized layer-space adaptation is a competitive and flexible alternative to conventional low-rank PEFT. Figure 1 : (Left) Conventional LoRA configuration. (Right) Our proposed ShadowPEFT. 1 Introduction Parameter-efficient fine-tuning (PEFT) alleviates the high training costs of full-parameter fine-tuning and provides a practical solution by efficiently adapting the large language models (LLMs) on various downstream applications ( 7 ) . Representative PEFT approaches include prompt- and prefix-based methods ( 16 ) , adapter modules inserted into Transformer blocks ( 9 ) , and and low-rank weight adaptation methods such as LoRA ( 11 ) and its variants (e.g., QLoRA ( 6 ) , DoRA ( 19 ) , inter alia ). Among these methods, LoRA-style PEFT has become the dominant practical choice due to its simplicity, effectiveness, and compatibility with existing LLM training pipelines. LoRA injects trainable low-rank updates into selected linear projections while keeping the pretrained weights frozen. Despite its empirical success, LoRA adopts a fundamentally linear-local parameterization: each selected linear layer receives its own trainable update, and task adaptation emerges from the aggregate effect of many independent perturbations distributed across depth. Although these modules are optimized jointly, the adaptation mechanism itself remains fragmented, since each linear learns a separate transformation without explicitly sharing an adaptation state or function across the network. Moreover, the fragmented adaptation is tied closely to the backbone’s internal weight structure and cannot be decoupled from the backbone model. In this work, we explore an alternative PEFT design in which adaptation is centralized in a shared functional module that operates on hidden representations of transformer layers. We propose ShadowPEFT, which augments a frozen backbone with a lightweight centralized shadow network (architecturally similar to the base model but scaled down) that is reused across transformer layers. The shadow network maintains a parallel hidden state, iteratively updates it across depth, and produces additive corrections to the backbone activations. In contrast to learning a collection of linear weight perturbations, ShadowPEFT performs transformer layer-level refinement with cross-layer parameter sharing. In this way, ShadowPEFT shifts the locus of adaptation from decentralized linear-level perturbations to centralized layer-level refinement, treating the PEFT process as learning a portable functional overlay rather than altering the backbone parameters. Since the shadow module is architecturally decoupled from the backbone, it can be trained, stored, and deployed as a standalone component, benefiting edge computing. This enables two appealing properties that are difficult to obtain with standard LoRA-style PEFT. First, the shadow can be attached or detached without modifying the frozen backbone weights, enabling modular deployment and independent versioning of adaptation modules. Second, the shadow model can be initialized from a smaller pretrained model, allowing a compact model to serve as a reusable adaptation module for a larger backbone. For example, a smaller model such as Qwen-0.5B can serve as the shadow model for a larger backbone like Qwen-8B. In this configuration, shadow model’s adaptation capacity can be reused across model scales. This perspective expands PEFT beyond lightweight parameter injection toward reusable, cross-scale adaptation dynamics. We study both randomly initialized and pretrained shadows, and show that pretraining substantially improves both attached and detached performance. We evaluate ShadowPEFT on multiple standard benchmarks spanning both generation and understanding tasks, including MMLU ( 8 ) , GSM8K ( 5 ) , and SQuAD V2 ( 23 ) . ShadowPEFT achieves competitive or improved performance relative to LoRA and DoRA under comparable, and in some cases slightly smaller, trainable-parameter budgets. We further study shadow pretraining, detached shadow-only inference, out-of-distribution transfer, parameter scaling, efficiency, and system-level robot intent evaluation. These results suggest that centralized ShadowPEFT is a viable and flexible alternative to conventional low-rank adaptation. Our contributions are summarized as follows: 1. We present ShadowPEFT, a PEFT framework that leverages a shared shadow module integrated at the transformer layer level rather than performing independent adaption at linear updates. We also introduce a stateful shadow mechanism that maintains and updates a parallel hidden representation across transformer layers (depth) and uses it to refine the frozen backbone hidden states. 2. We show that the shadow module is modular and portable: it can be attached or detached from the base backbone and can be initialized from a smaller pretrained model for cross-scale adaptation. 3. Across multiple benchmarks and backbone scales, ShadowPEFT achieves competitive or improved performance relative to LoRA and DoRA under comparable parameter budgets, while also enabling additional deployment modes unavailable to standard low-rank PEFT. 2 Related Work The rapid scaling of LLMs to hundreds of billions of parameters has rendered full fine-tuning increasingly impractical for multiple downstream tasks. An alternative to weight updates is prompt engineering, including few-shot prompting ( 18 ) and Chain-of-Thought reasoning ( 27 ) , which inject task instructions directly into the input without modifying model parameters. While highly parameter-efficient, these approaches are limited by context length and lack persistent task-specific adaptation. These constraints have motivated extensive research into parameter-efficient adaptation, which aims to finetune pretrained models by updating only a small subset of parameters while keeping the backbone frozen ( 28 ) . Early parametric adaptation attempts focus on soft prompt-based PEFT methods. Such methods, like Prompt Tuning ( 14 ) , Prefix Tuning ( 16 ) , and P-Tuning ( 20 ) , optimize a small set of continuous prompt parameters while leaving the backbone unchanged. However, the expressive capacity of prompt-based PEFT is fundamentally constrained by the limited dimensionality of prompt vectors. Later, adapter-based methods gain attention because of their compatibility with LLMs’ stacking-layer architecture. Adapter-based approaches insert lightweight bottleneck module into Transformer backbone while freezing backone weights ( 9 ) . Subsequent extensions improve modularity and transferability, including AdapterSoup ( 4 ) , Tiny-attention adapter ( 33 ) , and Compacter ( 12 ) . Although effective and modular, adapters remain independently optimized. Each layer learns task-specific transformations without explicit cross-layer coordination, potentially introducing redundancy and inconsistent depth-wise adaptation. While superficially similar to shared adapters, ShadowPEFT differs fundamentally in that it maintains a persistent state that evolves across layers, enabling global coordination and iterative refinement. Low-rank adaptation has emerged as one of the most influential PEFT paradigms. Instead of updating

Read the complete text