Model Reprogramming Demystified: A Neural Tangent Kernel Perspective

Ming-Yu Chung, Jiashuo Fan, Hancheng Ye, Qinsi Wang, Wei-Chen Shen, Chia-Mu Yu, Pin-Yu Chen, Sy-Yen Kuo · 2025

Read Model Reprogramming Demystified: A Neural Tangent Kernel Perspective by Ming-Yu Chung, Jiashuo Fan, Hancheng Ye, Qinsi Wang, Wei-Chen Shen, Chia-Mu Yu, Pin-Yu Chen, Sy-Yen Kuo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00620.txt.

Introduction

Ming-Yu Chung and co-authors explain why model reprogramming works, using neural tangent kernel theory, in a 2025 paper. Reprogramming adapts a big pretrained model to a new task by changing only its inputs and a small output layer. It succeeds surprisingly often, but theory lagged behind practice.

The authors show that success is governed by the eigenvalue pattern of the kernel matrix on the target data. The quality of the source model matters just as much. Their framework ties source and target together in one account. Experiments back the predictions. Clear rules for when reprogramming will work help practitioners choose source models instead of guessing.

Catalog record

CREATORMing-Yu Chung, Jiashuo Fan, Hancheng Ye, Qinsi Wang, Wei-Chen Shen, Chia-Mu Yu, Pin-Yu Chen, Sy-Yen Kuo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00620

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Model Reprogramming Demystified: A Neural Tangent Kernel Perspective Ming-Yu Chung, Jiashuo Fan, Hancheng Ye, Qinsi Wang, Wei-Chen Shen, Chia-Mu Yu, Pin-Yu Chen, Sy-Yen Kuo Abstract Model Reprogramming (MR) is a resource-efficient framework that adapts large pre-trained models to new tasks with minimal additional parameters and data, offering a promising solution to the challenges of training large models for diverse tasks. Despite its empirical success across various domains such as computer vision and time-series forecasting, the theoretical foundations of MR remain underexplored. In this paper, we present a comprehensive theoretical analysis of MR through the lens of the Neural Tangent Kernel (NTK) framework. We demonstrate that the success of MR is governed by the eigenvalue spectrum of the NTK matrix on the target dataset and establish the critical role of the source model’s effectiveness in determining reprogramming outcomes. Our contributions include a novel theoretical framework for MR, insights into the relationship between source and target models, and extensive experiments validating our findings. 1 Introduction In recent years, machine learning has achieved remarkable success in various domains, including computer vision (CV), medical AI, and large language models (LLMs). However, these advancements often rely on models with an enormous number of parameters, sometimes exceeding a billion or more. Ensuring the performance of specific tasks necessitates training such large models on vast datasets. Given this scale, developing a machine learning pipeline or algorithm becomes an expensive and time-intensive endeavor, as exemplified by ChatGPT-3 ( Brown et al. 2020 ) . When faced with numerous tasks, it becomes economically and temporally impractical to train multiple models. Figure 1: Schematic illustration of Model Reprogramming (MR): The target model f_{T} is composed with input transformation layer a , output mapping b and source model f_{S} . For neural tangent kernel (NTK), k(\cdot,\cdot) is the NTK induced by the source model f_{S} , while \Theta_{T}^{A} and \Theta_{T}^{B} are NTK induced by input transformation layer and output mapping respectively. It is important to mention that the input transformation layer and output mapping are derived by optimizing Eq. 1 , while the source model is frozen during the optimization process. In this paper, we aim to utilize the NTK Theory to identify the relation between the source model and the target model. Given the substantial resources invested in developing large models for specific tasks, an intriguing question has emerged: can these costly large models be repurposed to tackle different tasks? This inquiry has led to the development of a technique known as Model Reprogramming (MR) ( Chen 2024 ; Yang et al. 2023 ; Yang et al. 2021 ; Elsayed et al. 2018 ) , also referred to as Visual Prompt (VP) ( Bahng et al. 2022 ; Chen et al. 2023 ; Tsao et al. 2023 ) , specifically for computer vision tasks. MR is a machine learning framework that leverages an existing source model (a large model designed for a particular source task), a small amount of target data, and a minimal number of trainable parameters to create a model capable of addressing a new target task. According to ( Chen 2024 ) , MR is often implemented by adding a trainable input transformation layer and an output mapping layer (either trainable or pre-specified) to a source model for reprogramming (see Figure 1 for the MR in the classification setting). Examples include reprogramming human acoustic models for time series classification ( Yang et al. 2021 ) , reprogramming English language models for protein sequence learning ( Melnyk et al. 2023 ; Vinod et al. 2023 ) , and reprogramming LLMs for time-series forecasting ( Jin et al. 2023 ) , to name a few. Despite the impressive empirical success of MR, most MR-like algorithms face two significant limitations: (1) the lack of holistic theoretical analysis, (2) insufficient investigation into the relationship between MR, target data, and source data. In this paper, we address these issues through the framework of the Neural Tangent Kernel (NTK) ( Jacot et al. 2018 ; Lee et al. 2019 ; Huang and Yau 2020 ; He et al. 2020 ) . We begin by describing the mechanics of MR within the NTK framework. Based on this NTK representation, we demonstrate that the success of MR can be characterized by the eigenvalues of the kernel matrix evaluated on the target dataset. Our analysis also explains why the success of MR often depends on the effectiveness of the source model, as has been empirically found in prior works such as ( Tsao et al. 2023 ; Li et al. 2023 ) . Furthermore, we support our theoretical findings with extensive experiments, showing that the fast algorithm is effective in real-world scenarios. The experimental results suggest that our theoretical prediction is valid. Our contributions are summarized as follows. (1) We develop a theoretical framework for analyzing Model Reprogramming (MR) using the Neural Tangent Kernel (NTK). Specifically, our approach leverages the eigenvalue spectrum of the NTK matrix to characterize the success of MR. (2) We provide an explanation for the observed phenomenon that the success of MR is closely tied to the effectiveness of the source model ( Tsao et al. 2023 ; Li et al. 2023 ) . In particular, we identify the relationship between the eigenvalue spectrum of the source model and that of the target model, highlighting this connection as a critical factor underlying the phenomenon. (3) We perform comprehensive experiments to validate our theoretical findings. 2 Preliminaries and Related Works Model Reprogramming We formally define Model Reprogramming (MR) in the context of classification tasks as follows. Given a source dataset D_{S}=\{(x_{i},y_{i})\}_{i=1}^{N_{S}} sampled from the source distribution (x_{s},y_{s})\sim\mathfrak{D}_{S} , where x_{s}\in\mathcal{X}_{S}\subset\mathbb{R}^{d_{S}} and y_{s}\in\mathcal{Y}_{S}\subset\mathbb{R}^{c_{S}} , and a source model f_{S}:\mathbb{R}^{d_{S}}\rightarrow\mathbb{R}^{c_{S}} trained on the source dataset D_{S} , along with a target dataset D_{T}=\{(x_{i},y_{i})\}_{i=1}^{N_{T}} sampled from the target distribution (x_{t},y_{t})\sim\mathfrak{D}_{T} , where x_{t}\in\mathcal{X}_{T}\subset\mathbb{R}^{d_{T}} and y_{t}\in\mathcal{Y}_{T}\subset\mathbb{R}^{c_{T}} , we aim to solve the following optimization problem: a^{*},b^{*}={\arg\min}_{a\in\mathcal{H}_{A},b\in\mathcal{H}_{B}}\mathbb{E}_{(x,y)\sim D_{T}}\|b\circ f_{S}\circ a(x)-y\|_{2}^{2}. (1) Here, \mathcal{H}_{A}\subset\{a:\mathbb{R}^{d_{T}}\rightarrow\mathbb{R}^{d_{S}}\} and \mathcal{H}_{B}\subset\{b:\mathbb{R}^{c_{S}}\rightarrow\mathbb{R}^{c_{T}}\} are the hypothesis classes (neural networks) for the input transformation layer and the output mapping, respectively. The optimal solutions a^{*} and b^{*} are selected as the input transformation layer and the output mapping, respectively. MR generates a target model f_{T}:\mathbb{R}^{d_{T}}\rightarrow\mathbb{R}^{c_{T}} defined as b^{*}\circ f_{S}\circ a^{*} to solve the target problem. The proposed target model f_{T} is expected to achieve high accuracy on the target distribution \mathfrak{D}_{T} . The schematic diagram of MR is demonstrated in Figure 1 . Figure 2: This graph demonstrate our thinking for Model Reprogramming (MR). In this paper, we aims to identify the relationship between performance of source model and performance of target model. Our strategy is that, by NTK Theory, we first relate the performance of model with the minimum eigenvalue of its kernel matrix. Then, we bridge the relationship between the source model and target model by proving the proportional relation between minimum eigenvalues corresponding to the source model and the target model, under some additional assumptions. Recently, Chen 2024 provided a comprehensive survey of MR. MR has achieved significant success in different data domains. For continuous data domains, Yang et al. 2021 reprogrammed a speech model for time-series classification problems. Tsai et al. 2020 improved the settings of adversarial reprogramming for black-box models. For discrete data

Read the complete text