Continual Fine-Tuning with Provably Accurate and Parameter-Free Task Retrieval
Hang Thi-Thuy Le, Long Minh Bui, Minh Hoang, Trong Nghia Hoang · 2026
Read Continual Fine-Tuning with Provably Accurate and Parameter-Free Task Retrieval by Hang Thi-Thuy Le, Long Minh Bui, Minh Hoang, Trong Nghia Hoang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13235.txt.
Introduction
Hang Thi-Thuy Le and co-authors work on continual fine-tuning with task retrieval that needs no learned retriever. A pretrained model must learn new tasks in order and keep skill on old ones. Prompt methods can forget how to retrieve, while fixed methods adapt less well. Their method adapts parameters yet lets input embeddings shift at test time. Bounds tie retrieval error to how task clusters are shaped. Tests under large task shifts show the two parts work well together.
Catalog record
| CREATOR | Hang Thi-Thuy Le, Long Minh Bui, Minh Hoang, Trong Nghia Hoang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 89573 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.13235
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Continual Fine-Tuning with Provably Accurate and Parameter-Free Task Retrieval Hang Thi-Thuy Le, Long Minh Bui, Minh Hoang, Trong Nghia Hoang Abstract Continual fine-tuning aims to adapt a pre-trained backbone to new tasks sequentially while preserving performance on earlier tasks whose data are no longer available. Existing approaches fall into two categories which include input- and parameter-adaptation. Input-adaptation methods rely on retrieving the most relevant prompts at test time, but require continuously learning a retrieval function that is prone to forgetting. Parameter-adaptation methods instead use a fixed input embedding function to enable retrieval-free prediction and avoid forgetting, but sacrifice representation adaptability. To combine their best strengths, we propose a new parameter-adaptation method that enables adaptive use of input embeddings during test time with parameter-free retrieval. We derive task-retrieval error bounds for a clustering-based, parameter-free paradigm, providing theoretical guarantees that link low retrieval error to structural properties of task-specific representation clusters, revealing a fresh insight into how well-organized clustering structure will enable reliable retrieval. Motivated by this insight, our method is designed with two key components: (i) an adaptive module composition strategy that learns informative task-specific updates to preserve and complement prior knowledge, and (ii) a clustering-based retrieval mechanism that captures distinct representation signatures for each task, enabling adaptive representation use at test time. Extensive experiments show that these components work synergistically to improve retrieval and predictive performance under large shifts in task semantics. 1 Introduction Continual learning (CL) is a classical challenge in machine learning (ML) where a model must learn from a sequence of tasks that arrive one at a time and with their own distinct data distributions. A key constraint in this setting is that data from previous tasks cannot be retained once new tasks arrive, obscuring the overall view of all task distributions. This leads to catastrophic forgetting ( 41 ; 31 ) in which learning new tasks interferes with previous knowledge. Classical approaches often mitigate forgetting via regularization ( 39 ; 33 ) , replay buffers ( 4 ; 5 ; 7 ) , or pruning ( 17 ; 11 ) . However, these strategies either update/prune large shared models, or maintain/retrieve from sizable parameter sets or representative samples for each task, which struggle to scale when task sequences become long with increasing shifts in semantics. Against these limitations, the recent rise of large foundation models ( 8 ) has opened up new possibilities for CL designs, enabling a paradigm where a pre-trained backbone is frozen to ensure knowledge retention. During training, task-specific updates are incorporated via either input-adaptation or parameter-adaptation methods. To elaborate, input adaptation refers to adaptation methods that append learnable parameters (i.e., prompts) to the tokenized input sequences (e.g., sequences of image patches). Examples include prompt-tuning methods ( 24 ) . Parameter adaptation refers to methods that add low-complexity trainable parameter blocks to its pre-trained parameters. Examples include LoRA-based methods ( 16 ) . Learning these units requires significantly smaller memory footprint and computation cost than maintaining and updating large sets of shared parameters with sophisticated regularization. For example, RanPAC addresses the issue of forgetting via (1) fine-tuning a low-rank adaptation (LoRA) unit using data from the first task; and (2) using it to compute input and class embeddings across all tasks ( 32 ) . A parameter-free classification model (e.g., nearest neighbor or linear discriminant analysis) is then employed to map unseen inputs to their most probable classes. Its use of a pre-tuned set of adaptation parameters (using data from the first task) for inputs across all tasks however limits its adaptability in real-world applications involving significant semantic gaps across tasks ( 20 ) (see Section 5.2 ). Alternatively, retrieval-based continual fine-tuning frameworks often adopt a two-stage strategy: (1) learning task-specific fine-tuning parameters to mitigate interference between tasks ( 45 ; 46 ; 47 ; 43 ; 32 ) ; and (2) retrieving the appropriate parameter set at inference time to handle previously unseen inputs. For instances, L2P maintains a shared pool of prompts to compartmentalize task-specific knowledge and optimizes a prompt-selection mechanism to retrieve relevant prompts for each input ( 47 ) . This leads to several generalizations such as separating global and local prompts ( 46 ) , using prompt ensemble to enhance knowledge transfer ( 43 ) , or maintaining a growing prompt pool ( 37 ; 43 ; 45 ) . While their learnable retrieval mechanisms help mitigate knowledge interference and enable flexible test-time adaptation, these methods retain the risk of forgetting because the parameters of their retrieval modules must be continuously updated as new tasks arrive. Contribution. To alleviate key issues with the lack of representation adaptability in parameter-free classification ( 32 ) and forgetting in the retrieval-based methods , we develop Proteus which is a pro vably accura te , parameter-free task retrieval framework for continual fine-t u ning s ystems. It achieves provably low retrieval error rate and hence superior performance compared to existing state-of-the-arts. This is enabled by the following contributions: 1. Provably Accurate and Parameter-Free Retrieval. We develop a theoretical analysis of retrieval error rates for a broad class of parameter-free, signature-based representation retrieval methods. Our framework characterizes retrieval error in terms of the clustering structure of the distributions of signature patterns (see Definition 3.1) created for different task-specific adaptation modules. This provides a principled foundation applicable across diverse retrieval-based CL methods. Concretely, we study algorithms that learn distinct signature patterns from each task’s input embeddings to capture both intra- and inter-task variability. These signature patterns serve as keys to retrieve the most relevant representation adaptation modules for each input during test time. The retrieval method is parameter-free, and its error rate is provably controlled by statistical properties of the signature clusters and is shown to be vanishingly small (Section 3.2 ). 2. Adaptive Knowledge Transfer and Discovery. Motivated by the theoretical insight that retrieval error rates are controlled by the clustering properties of signature patterns, we develop an adaptive knowledge transfer and discovery method designed to improve cluster separation (see Definition 3.3) and thus reduce retrieval error. This is achieved via learning new representation components that complement those from previous tasks. This in turn helps segregate new representation attributes from old ones and creates more distinct clusters of representation signatures for new tasks while preserving knowledge transferability. As a result, each task-specific adaptation module and its induced input embedding or representation distributions become easier to discriminate and retrieve during test-time inference, ensuring low retrieval error and improving predictive performance (Section 4 ). 3. Empirical Studies. We demonstrate empirically that the integrated synergy between our adaptive knowledge transfer and parameter-free retrieval mechanism with provably low error rate significantly enhances the performance and scalability of continual fine-tuning. As reported in our empirial studies, our proposed method Proteus establishes new SOTA results on a wide variety of prominent class-incremental CL benchmarks, including the highly challenging Visual Task Adaptation Benchmark (VTAB) featuring diverse task sequences large semantic gap ( 52 ) . It achieves up to 57% and 30% gains in retrieval and classification performance while attaining the best top-1 average forgetting metric, which corroborates our theoretical insights (Section 5 ). Figure 1: The overall workflow of Proteus . A: A LoRA unit is optimized for each