Bilinear Coordinate Alignment for Training-Free Task-Vector Transfer

Jungyong Son, Jinwook Jung, Minhee Park, Sungyong Baik · 2026

Read Bilinear Coordinate Alignment for Training-Free Task-Vector Transfer by Jungyong Son, Jinwook Jung, Minhee Park, Sungyong Baik (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28444.txt.

Introduction

Jungyong Son and three colleagues tackle reuse of fine-tuning expertise when a new base model version appears. Task vectors, the parameter gap between fine-tuned and base models, cannot transfer directly across different parameterizations. They show such vectors arise as accumulated bilinear interactions between input activations and output gradients. Their BiCo framework aligns both spaces with orthogonal mappings from one forward-backward pass on a small calibration sample, beating existing transfer methods across vision and language benchmarks.

Catalog record

CREATORJungyong Son, Jinwook Jung, Minhee Park, Sungyong Baik
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77287
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.28444

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bilinear Coordinate Alignment for Training-Free Task-Vector Transfer Jungyong Son, Jinwook Jung, Minhee Park, Sungyong Baik Abstract Fine-tuning large-scale pre-trained models is a recent prevalent paradigm for adapting general representations to specialized tasks. However, when a new version of a pre-trained model becomes available, expertise acquired through fine-tuning cannot be directly reused because it is tied to the parameterization of the original model, requiring another costly fine-tuning. To address this inefficiency, recent work uses task vectors—defined as the parameter difference between a fine-tuned model and its base model—to transfer expertise across models. While existing methods bridge disparate models by matching activations or gradients, a significant performance gap remains relative to direct fine-tuning, suggesting that these partial correspondences are insufficient. In this work, instead of viewing a task vector merely as a parameter offset, we revisit the formation of task vectors and show that they can be derived as accumulated bilinear interactions between input-side activations and output-side gradients. Motivated by this observation, we formulate task-vector transfer as a dual-space alignment problem and propose BiCo , a training-free framework for transferring task vectors through Bi linear Co ordinate alignment. BiCo estimates orthogonal Procrustes mappings in both spaces using a single forward–backward pass on a small calibration set, without any parameter update. Across extensive computer vision and natural language processing benchmarks, BiCo consistently outperforms existing transfer methods across models that differ in width, depth, and pre-training configuration. 1 Introduction Modern machine learning has increasingly adopted a paradigm in which large-scale pre-trained models [ 1 , 31 , 12 ] are adapted to downstream tasks through task-specific fine-tuning [ 6 , 34 , 36 ] . While this approach is highly effective, it becomes inefficient as pre-trained model families are continually expanded with new variants that follow similar architectural designs but differ in model scale, pre-training data, or training recipes [ 9 , 13 ] . Since fine-tuned knowledge is encoded in the parameter space of the specific pre-trained model on which it was learned, it cannot be directly transferred to a different model variant. Consequently, each newly released pre-trained model often requires separate fine-tuning to acquire similar task-specific capabilities, leading to repeated computational overhead and increased storage demands. This work addresses the challenge of transferring task-specific knowledge across pre-trained model variants without requiring additional fine-tuning. To represent and manipulate this fine-tuned expertise, Task Arithmetic [ 16 ] introduces the concept of task vectors, defining them as the parameter difference between a fine-tuned model and its base model (i.e., \bm{\tau}={\bm{\theta}}_{\mathrm{ft}}-{\bm{\theta}}_{\mathrm{pre}} ). This formulation has led to a growing line of research on task-vector transfer [ 24 , 25 , 26 ] : given a task-vector obtained by fine-tuning a source pre-trained model, the goal is to transfer this vector to a separate target pre-trained model so that the target acquires the corresponding task-specific capability without undergoing full fine-tuning. One prominent direction exploits permutation symmetries to align the weight spaces of different models [ 2 , 24 ] . However, such parameter-level alignment typically relies on strong architectural compatibility and matching parameter dimensions, limiting its applicability to models that differ in scale or width. To overcome these limitations, recent studies have explored using task-specific signals, such as gradients or activations, to transform task vectors across disparate models [ 26 , 25 ] . GradFix [ 25 ] uses target-model gradients to mask task vectors according to the local loss landscape, while THESEUS [ 26 ] transfers task vectors by matching intermediate activation spaces. While these approaches provide valuable partial correspondences, a significant performance gap remains relative to direct fine-tuning. In this work, we first show that a task vector can be derived as the accumulation of the bilinear interactions coupling input-side activations with output-side gradients. Building on this insight, we argue that treating activations and gradients as separate, independent proxies for task knowledge fails to capture the underlying mechanics of the parameter update. Rather than naively combining these disjoint alignment strategies, we formulate task-vector transfer as a holistic, coupled dual-space alignment problem. To this end, we propose BiCo, a training-free framework that leverages Bi linear Co ordinate alignment to transfer task-specific expertise across pre-trained models. By applying orthogonal Procrustes alignment [ 27 ] to both spaces, BiCo transforms the source task vector into a target-compatible update while preserving the geometry of the aligned feature and gradient spaces. Importantly, the required alignments are estimated from a small calibration set using a single forward–backward pass through frozen pre-trained models, requiring neither parameter updates nor task-specific hyperparameter tuning. We demonstrate the effectiveness of BiCo across a wide range of benchmarks in computer vision and natural language processing, showing that it enables robust knowledge transfer even across models with architectural and pre-training variations. Our contributions are summarized as follows: • We revisit the mechanistic formation of task vectors and formulate their transfer as a dual-space coordinate alignment problem involving both input-side activations and output-side gradients. • We introduce BiCo, which uses orthogonal Procrustes alignment to transfer task vectors from a small calibration set using a single forward–backward pass, without any parameter updates. • Through extensive experiments in both computer vision and natural language processing, we demonstrate that BiCo significantly outperforms existing methods and substantially narrows the gap to directly fine-tuning the target model. 2 Related work Model merging. Model merging aims to combine multiple task-specific models without additional training [ 16 , 38 , 39 , 33 , 14 ] . A representative approach is Task Arithmetic [ 16 ] , which represents fine-tuned expertise as task vectors in parameter space and manipulates them through linear operations such as addition and negation. Subsequent methods, including TIES-Merging [ 38 ] and DARE [ 39 ] , further improve merging by reducing interference among task vectors. Recent work further improves this paradigm by identifying consensus directions among task vectors [ 33 ] or by exploiting their low-rank singular structure to reduce task interference [ 14 ] . These studies demonstrate that task vectors provide a useful abstraction for reusing and composing fine-tuned knowledge. However, most model-merging methods operate under a shared-parameter-space assumption, typically requiring task vectors to be defined relative to the same pre-trained model. This limits their applicability when models differ in architecture, width, or initialization, and points to a broader question of how to relate models whose parameters are not directly aligned. Weight and representation alignment. From the parameter-space perspective, Re-basin methods [ 2 , 24 ] are based on the observation that models with the same architecture but different initializations can learn similar functions while representing them with different hidden-unit orderings. They address this parameter-space mismatch by exploiting permutation symmetries to map models into a shared basin. However, these techniques are generally restricted to models with identical architectures and widths. Representation-level studies provide another perspective by comparing models through functional similarity rather than parameter correspondence [ 18 , 11 , 4 , 21 ] . Centered kernel alignment (CKA) [ 18 ] reveals shared internal structure across independently trained networks, while model stitching [ 11 , 4 , 21 ] shows that such representations can sometimes be connected through simple mappings. These studies suggest that independently trained models can exhibit compatible internal representations, motivating the use of representation alignment

Read the complete text