Scaling Linear Mode Connectivity and Merging to Billion Parameter Pretrained Transformers
Tianyi Li, Zhiqiang Shen · 2026
Read Scaling Linear Mode Connectivity and Merging to Billion Parameter Pretrained Transformers by Tianyi Li, Zhiqiang Shen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-23607.txt.
Introduction
Li and Shen scale up a method for merging separately trained neural networks into one, in a 2026 paper on deep learning. Linear mode connectivity means two models can be joined by a straight line in weight space without the loss spiking midway. When that holds, merging is as simple as averaging weights. In practice, trained models sit in mismatched arrangements, since many weight patterns compute the same function, and naive averaging hits a loss barrier.
Their framework applies function preserving transformations to align the two models. A dual learning procedure then trains both transformations at once, each moving toward a shared straight path. On WikiText tasks the method reaches near zero loss barriers at medium scale. A large vision transformer keeps accuracy above sixty nine percent along the path, and billion parameter models show only small barriers.
Catalog record
| CREATOR | Tianyi Li, Zhiqiang Shen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.23607
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scaling Linear Mode Connectivity and Merging to Billion Parameter Pretrained Transformers Tianyi Li, Zhiqiang Shen Abstract Linear mode connectivity (LMC) provides a promising foundation for understanding and merging independently trained neural networks, but existing methods typically optimize the interpolation path from only one model endpoint, limiting their scalability and effectiveness for large pretrained transformers. We propose a novel and scalable framework for enabling LMC-based model merging to billion-parameter pretrained transformers . Our method applies properly parameterized functionality-preserving weight transformations to align functionally equivalent solutions, and introduces a dual learning procedure in which both models jointly learn their corresponding transformations toward a shared linear interpolation path. This bidirectional optimization substantially reduces interpolation barriers and enables more reliable merging across large-scale architectures. Empirically, we show that our approach achieves near-zero loss barriers on WikiText for language models with medium-sized parameters, representing, to our knowledge, the first demonstration of near-barrier-free linear connectivity at this scale. In the vision domain, ViT-L maintains above 69% ImageNet top-1 accuracy throughout the interpolation path, while modern billion-parameter LLMs exhibit only small loss barriers. These results suggest that properly resolving parameter symmetries enables large pretrained Transformers to be connected and merged through simple linear paths with substantially improved interpolation performance. Code: https://github.com/VILA-Lab/Dual-Learned-Matching . 1 Introduction Deep neural network weights have recently become an increasingly important object of study in the machine learning community [ 15 , 13 , 2 , 35 , 11 ] . Beyond specifying a model’s input-output behavior, trained weights are increasingly viewed as reusable artifacts that can be inspected, edited, adapted, merged, or even treated as data for training other models [ 31 , 33 ] . This perspective is especially important in the era of large pretrained models, where each checkpoint represents substantial investment in data, computation, optimization, and engineering [ 24 ] . Understanding the structure of weight space, and how different checkpoints are geometrically and functionally related, is therefore not only a theoretical question about loss landscapes, but also a practical foundation for scalable model reuse and composition. A central idea for studying weight-space structure is linear mode connectivity (LMC) [ 13 , 9 ] . Two models are linearly mode connected if the linear interpolation between their weights remains in a low-loss region. When such connectivity exists, simple weight interpolation becomes a natural and efficient mechanism for model merging, without requiring ensembles, additional architectures, or expensive retraining. However, independently trained models often fail to exhibit low-loss linear paths in their raw parameterization. Even when two networks implement similar functions, they may occupy apparently distant regions of weight space due to differences in initialization, training order, datasets, and hyperparameters [ 11 , 20 ] . As a result, naive interpolation can encounter large loss barriers, making model merging unreliable. A major reason for this failure is that neural networks exhibit many function-preserving symmetries: distinct parameter configurations can represent the same underlying function [ 5 , 10 ] . A classic example is neuron permutation symmetry, where hidden units within a layer can be permuted together with corresponding inverse permutations in adjacent layers without changing the network function [ 16 , 6 , 10 ] . In small networks, resolving such symmetries can align independently trained models before interpolation and reveal low-loss paths that are hidden in the original coordinates [ 10 , 25 , 27 ] . Existing alignment methods commonly rely on activation matching, which uses data to align intermediate representations [ 21 , 27 , 2 ] , or weight matching, which aligns parameters directly in a data-free manner [ 25 , 2 ] . Beyond discrete symmetries like permutation, Transformers, however, introduce a substantially richer and more structured symmetry space than standard multilayer networks. Their residual connections, normalization layers, multi-head attention modules, positional encodings, and feed-forward blocks induce multiple classes of functionality-preserving transformations [ 3 , 28 ] . Recent studies have begun to exploit these Transformer-specific symmetries to uncover hidden connectivity between trained models [ 34 ] . Nevertheless, most activation- or weight-matching methods remain task-agnostic: they align representations or parameters without directly optimizing the performance of interpolated models. This mismatch limits their ability to reduce the actual loss barrier, especially for pretrained Transformers where small misalignments can accumulate across many layers and severely degrade interpolation performance. Figure 1: Loss barriers and scaling behavior. Left: Interpolation loss barriers for ViT-S under naive weight interpolation, learned matching, and LMC-DM (Ours). Middle: Interpolation loss curves for Pythia models across different parameter scales. Right: Scaling behavior of LMC-DM–merged Pythia models compared to endpoint models as model size increases. Learned matching partially addresses this issue by optimizing symmetry transformations using the task loss of interpolated models [ 28 ] . However, existing learned matching approaches still have several limitations. First, they typically exploit only a subset of the full Transformer symmetry group and often rely on simple parameterizations, such as unconstrained variables that are projected during the forward pass. Second, they apply symmetry transformations to only one endpoint model, leading to an asymmetric formulation in which one model is optimized toward a fixed counterpart. This one-sided optimization can create suboptimal landscapes and restrict the search for better-aligned representatives of both models. Consequently, prior demonstrations have largely been limited to models with millions or tens of millions of parameters trained on relatively small datasets. Whether LMC can be effectively revealed in large pretrained Transformers, and whether symmetry learning can scale to billion-parameter models, remains largely unexplored. In this work, we propose a scalable framework for linear mode connectivity and model merging in billion-parameter pretrained Transformers. We systematically formulate a broad family of functionality-preserving weight transformations for Transformers, including normalization absorption, residual-space rotation and scaling, attention-head permutation, head-internal transformations, and feed-forward permutation and scaling. We parameterize these transformations under appropriate structural constraints and optimize them directly with respect to the loss along the interpolation path. Most importantly, we introduce a dual learning procedure, where both endpoint models learn their own symmetry transformations toward a shared linear path. This bidirectional formulation gives both models the flexibility to resolve their internal symmetries, substantially reduces interpolation barriers, and enables more reliable merging at scale. Empirically, we show that ViT-L maintains over 69% ImageNet-1K top-1 accuracy across the entire interpolation path; language models on 160M parameters achieve near-zero loss barriers on WikiText, the first such result in this setting, outperforming all prior counterparts such as Weight Matching [ 2 ] and Learned Matching [ 28 ] ; and modern billion-parameter LLMs exhibit only small loss barriers. These results suggest that large pretrained Transformers can be effectively aligned and merged by simple linear interpolation when equipped with properly parameterized and jointly optimized symmetry transformations. Our contributions are summarized as follows: • We formulate a broad family of functionality-preserving Transformer symmetries. Specifically, our framework covers normalization absorption, residual-space rotation and scaling, attention-head permutation, head-internal transformations, and feed-forward permutation and scaling. We investigate practical parameterizations of continuous symmetry variables, enabling functionality-preserving transformations to be learned directly from the interpolation loss. • We propose a dual learning procedure for bidirectional endpoint alignment. Unlike prior one-sided methods, both models learn their own transformations toward a shared linear interpolation path, leading to better-conditioned optimization and lower interpolation barriers. • We show strong and