Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instruction
Zesheng Ye, Chengyi Cai, Ruijiang Dong, Jianzhong Qi, Lei Feng, Pin-Yu Chen, Feng Liu · 2025
Read Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instruction by Zesheng Ye, Chengyi Cai, Ruijiang Dong, Jianzhong Qi, Lei Feng, Pin-Yu Chen, Feng Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-04650.txt.
Introduction
Ye and co-authors tie three tune ways for big nets in one frame. Big pre-trained nets grow in size and skill year by year. How to bend them to set tasks, cheap, is a key ask. Past tune ways grew in lone lanes with dim ties twixt them. Their frame, called re-program skill, joins those lanes in one view. Three lanes join: model re-work, cue tune, and cue teach. All leave the net's core weights froze in place.
Each lane works the net by shifts at its rim points. Shifts can hit the in-feed, mid coats, or tale bits in text. Nets heed such shifts, which lets plain cues steer acts. A four-part sort maps the lanes by shift form and more. Shift form splits as fixed or learn-able in the sort.
Catalog record
| CREATOR | Zesheng Ye, Chengyi Cai, Ruijiang Dong, Jianzhong Qi, Lei Feng, Pin-Yu Chen, Feng Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 87 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-04650
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instruction Zesheng Ye, Chengyi Cai, Ruijiang Dong, Jianzhong Qi, Lei Feng, Pin-Yu Chen, Feng Liu Abstract As large-scale pre-trained foundation models continue to expand in size and capability, efficiently adapting them to specific downstream tasks has become increasingly critical. Despite substantial research progress, existing adaptation approaches have evolved largely in isolation, without a clear understanding of their interrelationships. This survey introduces neural network reprogrammability as a unifying framework that bridges mainstream model adaptation techniques–model reprogramming, prompt tuning, and prompt instruction–previously fragmented research areas yet converges on a shared principle: repurposing a pre-trained model by manipulating information at the interfaces while keeping the model parameters frozen. These methods exploit neural networks’ sensitivity to information manipulation on different interfaces, be it through perturbing inputs, inserting tokens into intermediate layers, or providing task-specific examples in context, to redirect model behaviors towards desired outcomes. Building on the concept of reprogrammability, we present a taxonomy that categorizes such information manipulation-based adaptation approaches across four key dimensions: manipulation format (fixed or learnable), location (interfaces where manipulations occur), operator (how they are applied), and output alignment requirement (post-processing needed to align outputs with downstream tasks). Notably, this framework applies consistently across data modalities, independent of specific model architectures. Moreover, viewing established techniques such as in-context learning and chain-of-thought prompting through this lens reveals both their theoretical connections and practical distinctions. We further analyze remaining technical challenges and ethical considerations, positioning neural network reprogrammability as a fundamental paradigm for efficient model adaptation. We lastly identify promising research directions emerging from this integrative viewpoint. \correspondingauthors [email protected] \equalcontributions 1 Introduction The rise of foundation models is revolutionizing the landscape of modern machine learning. By learning general-purpose data representations, these powerful pre-trained architectures have delivered unprecedented results in computer vision ( Dosovitskiy et al. 2021 ; Zhao et al. 2023 ) and natural language processing ( Vaswani et al. 2017 ; Devlin et al. 2019 ; Brown et al. 2020 ) , but their massive scale–often billions of parameters–creates significant deployment challenges. Conventionally, adapting these models to specific downstream tasks necessitates an intensive process of fine-tuning , where a substantial portion, if not all, of the model’s parameters are updated using task-specific data ( Pan and Yang 2009 ; Chen 2024 ) . This parameter-centric adaptation strategy (PCA), depicted in Figure 1 (a), directly modifies the model’s learned weights. While straightforward and often effective, PCA may come with several limitations, especially when resources are constrained. The computational demands of retraining these models often become prohibitively expensive. Access to model internals may not be available often when models are provided as services (i.e., black-box setting). Furthermore, the need for specialized hardware (e.g., high-performance GPUs) creates barriers to entry, and the significant energy footprint raises valid environmental concerns. Figure 1: Paradigm shift from conventional parameter-centric adaptation (i.e., modifying model parameters) to reprogrammability-centric adaptation (i.e., modifying input data and model output). This represents a shift in thought from modifying the model to align with the task to modifying the task to align with the model . A more efficient paradigm is gaining traction. Rather than modifying the internal parameters , a growing body of work now focuses on adapting pre-trained models for downstream tasks (i.e., target tasks) without extensive (or, in some cases, any) parameter updates . As Figure 1 (b) shows, these approaches strategically manipulate the target task-specific inputs or contextual information provided to a fixed pre-trained model, and then align the model output to the requirement of the target task’s output space. In this paper, we term such methods reprogrammability-centric adaptation (RCA) and formally define the concept of reprogrammability in Sec. 2.1 . Importantly, the shift of adaptation focus in RCA methods leads to a compelling benefit: model adaptation costs (in terms of trainable parameters) become largely independent of model size, correlating instead with the complexity of the task itself. Namely, the input and output dimensionality of the target data. The efficiency benefits can be substantial. Consider a typical neural network, such as a multi-layer perceptron (MLP) with L layers of dimension D processing inputs of dimension d , we typically find that D\times L\gg d –the model’s complexity substantially exceeds that of the data it processes since neural networks are heavily over-parameterized ( Zhang et al. 2017 ; Allen-Zhu et al. 2019 ) . This implies that RCA methods can reduce the training burden as they shift the model adaptation efforts from the parameter space to a much lower-dimensional input space. Figure 2 showcases a concrete illustration, where we compare the number of trainable parameters when adapting an ImageNet-pretrained ViT/B-32 ( Dosovitskiy et al. 2021 ) to a new remote sensing image classification task ( Helber et al. 2019 ) . The comparison is between various fine-tuning (i.e., PCA ) scenarios and reprogramming (i.e., RCA ), confirming that RCA consistently involves significantly fewer trainable parameters than most PCA scenarios. The implications of this efficiency are far-reaching. By significantly reducing computational demands, RCA methods promise to democratize access to cutting-edge AI capabilities, enable deployment in resource-constrained environments, and reduce the carbon footprint of AI applications. Figure 2: Comparison of trainable parameters for fine-tuning (i.e., PCA) vs. reprogramming (i.e., RCA) an ImageNet-pretrained ViT/B-32 on a new classification task. Different PCA scenarios are shown (e.g., "E+L10B+LN+H" indicates fine-tuning the embedding layer, the last 10 Transformer blocks, layer normalization, and the classification head). The dashed line indicates the number of trainable parameters for RCA, which is consistently lower than that of most PCA configurations. 1.1 Motivation: The Need for Terminological Clarity Despite the promise of this emerging paradigm, a significant obstacle to progress in this field is the bewildering array of inconsistent terminology. Techniques like “prompt tuning” and “model reprogramming” and variations thereof are often used interchangeably or, worse, with conflicting definitions across research communities, such as the same terms applied to fundamentally different approaches. Consider how the term “prompt tuning” operates in the literature. In some contexts, it broadly encompasses any modifications to the raw input data ( Brown et al. 2020 ) , yet in other contexts, it specifically refers to adding trainable continuous vectors to input embeddings ( Li and Liang 2021 ; Lester et al. 2021 ) . Similarly, “model reprogramming” can denote input transformations ( Elsayed et al. 2019 ; Chen 2024 ) and, elsewhere, describe manipulations within embedding spaces ( Neekhara et al. 2019 ) . This terminological ambiguity creates unnecessary barriers to knowledge transfer across research communities. Accordingly, researchers who work on similar problems may struggle to identify and build upon each other’s work or conduct fair comparisons, ultimately slowing progress in such a field with vast potential to broaden access to state-of-the-art machine learning capabilities. 1.2 Purpose: Unifying Disparate Approaches To address these issues, this survey introduces Neural Network Reprogrammability as a unifying conceptual framework to bring clarity and structure to the diverse landscape of parameter-efficient adaptation methods. We define reprogrammability as the inherent capability of pre-trained models to be repurposed for new tasks without internal parameter updates ; whilst the adaptation is achieved by strategically manipulating the inputs (and/or auxiliary) information provided to the model ( Elsayed et al. 2019 ; Chen 2024