Activation-Guided Consensus Merging for Large Language Models
Yuxuan Yao, Shuqi Liu, Zehua Liu, Qintong Li, Mingyang Liu, Xiongwei Han, Zhijiang Guo, Han Wu, Linqi Song · 2025
Read Activation-Guided Consensus Merging for Large Language Models by Yuxuan Yao, Shuqi Liu, Zehua Liu, Qintong Li, Mingyang Liu, Xiongwei Han, Zhijiang Guo, Han Wu, Linqi Song (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14009.txt.
Introduction
Yuxuan Yao, Linqi Song, and seven co-authors merge language models by watching their inner activity. Their paper, from twenty twenty-five, is called ACM. Merging blends skills from several tuned models into one, with no new training. Old merge methods treat every layer as equally key, which wastes the fact that layers do different jobs.
ACM sets a merge weight for each layer from how much its signals agree. Shared signals between the base and tuned models guide the mix. No gradient math or extra training is needed. Tests on long-to-short reasoning and general merges beat all rival methods. One trial on a mid-size model family shows clear gains.
Catalog record
| CREATOR | Yuxuan Yao, Shuqi Liu, Zehua Liu, Qintong Li, Mingyang Liu, Xiongwei Han, Zhijiang Guo, Han Wu, Linqi Song |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-14009
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Activation-Guided Consensus Merging for Large Language Models Yuxuan Yao, Shuqi Liu, Zehua Liu, Qintong Li, Mingyang Liu, Xiongwei Han, Zhijiang Guo, Han Wu, Linqi Song Abstract Recent research has increasingly focused on reconciling the reasoning capabilities of System 2 with the efficiency of System 1. While existing training-based and prompt-based approaches face significant challenges in terms of efficiency and stability, model merging emerges as a promising strategy to integrate the diverse capabilities of different Large Language Models (LLMs) into a unified model. However, conventional model merging methods often assume uniform importance across layers, overlooking the functional heterogeneity inherent in neural components. To address this limitation, we propose A ctivation-Guided C onsensus M erging ( ACM ), a plug-and-play merging framework that determines layer-specific merging coefficients based on mutual information between activations of pre-trained and fine-tuned models. ACM effectively preserves task-specific capabilities without requiring gradient computations or additional training. Extensive experiments on Long-to-Short (L2S) and general merging tasks demonstrate that ACM consistently outperforms all baseline methods. For instance, in the case of Qwen-7B models, TIES-Merging equipped with ACM achieves a 55.3% reduction in response length while simultaneously improving reasoning accuracy by 1.3 points. We submit the code with the paper for reproducibility, and it will be publicly available. 1 Introduction The cognitive evolution of LLMs has progressed from System 1 to System 2 paradigms [ 44 , 16 ] , characterized by the emergence of advanced reasoning models like OpenAI’s o1/o3 [ 29 , 30 ] , QwQ [ 31 ] , and DeepSeek-R1 [ 6 ] . System 1 implementations [ 7 , 5 ] , such as GPT-4o [ 28 ] , LLaMA-3 [ 7 ] , DeepSeek-V3 [ 5 ] , leverage rapid intuitive processing for immediate responses but struggle with complex reasoning tasks. In contrast, System 2 architectures are fine-tuned with extended thinking chains to promote deliberate analysis through iterative self-assessment, error mitigation, and verification, albeit facing challenges related to redundancy. This dual-system dichotomy motivates the Long-to-Short (L2S) framework [ 36 ] , which seeks to reconcile System 2’s analytical depth with System 1’s operational efficiency. Beyond computationally intensive training-based approaches [ 1 , 10 ] and instability-prone prompting methods [ 23 ] , a promising alternative lies in model merging techniques to seamlessly integrate System 2 models with their counterparts without incurring additional computational overhead [ 39 ] . Model merging [ 38 , 11 ] refers to the process of integrating the parameters of a pre-trained (PT) model with those of multiple fine-tuned (FT) models to create a single unified model. This approach aims to enhance performance, generalization, and robustness by leveraging the strengths and diverse insights of each individual model. Within the framework of model merging, task vectors [ 11 ] have emerged as essential components for encoding task-specific knowledge representations. Defined as parametric deltas between pre-trained and fine-tuned model weights, these vectors theoretically enable cross-model capability integration through linear arithmetic operations. Current task vector-based techniques [ 41 , 43 ] predominantly use static coefficient protocols that enforce uniform scaling across all parameters and tasks. This simplification assumes uniform task relevance across all neural weights, regardless of their layer-specific functional criticality. However, layer importance is often heterogeneous. In tasks such as L2S, layers critical for specific functions, such as the lm_head layer for generation length control, may require different weighting coefficients than other layers. To mitigate the problems, recent efforts utilize gradients [ 19 ] or activations [ 27 ] to derive layer-specific coefficients for task vectors. Nonetheless, these methods either overlook inter-model relationships by solely focusing on PT model’s activations or necessitate complex backpropagation. In this work, we tackle the aforementioned challenges by theoretically analyzing how activation values relate to weight salience, then designing customized layer-specific weighting coefficients to avoid complex gradient calculations. Mutual information (MI) is selected to measure activations, as it can select features that are highly relevant while preserving critical information, quantifying feature redundancy [ 14 , 8 ] . Specifically, we propose A ctivation-Guided C onsensus M erging ( ACM ), a novel and efficient approach based on activation MI among models. Different from the previous element-wise activation merging method [ 27 ] , which concentrates on maintaining the pre-trained model’s capacity, our ACM first computes layer-wise activations for both PT and FT models using a shared calibration dataset. Next, ACM measures the mutual information (MI) between corresponding PT and task-specific FT activations at each layer, normalizing these values to obtain layer-specific weighting coefficients. The overall framework is presented at Figure 1 . Crucially, these coefficients are inversely proportional to the mutual information scores, layers with higher similarity (greater MI) receive lower weights to reduce redundancy, while layers showing significant divergence (lower MI) are assigned higher weights to preserve FT-specific capabilities. Overall, our contributions include: 1) We analyze the correlation between activation space and weight salience, propose a novel merging approach dubbed as ACM, which leverages MI to compute the weight coefficients of each layer. 2) Extensive experiments in the L2S merging scenario demonstrate ACM’s efficiency in maintaining performance and reducing redundancy compared to both training-based and traditional merging baselines. For instance, in the case of Qwen-7B models, TIES-Merging equipped with ACM achieves a 55.3% reduction in response length while simultaneously improving reasoning accuracy by 1.3 points. 3) General merging tasks further validate ACM’s soundness and efficacy. The results show that ACM serves as a plug-and-play solution, consistently improving the performance of existing merging methods. 2 Related Work Long-to-Short Reasoning The paradigm shift in LLM has evolved from rapid, straightforward System 1 reasoning to deliberate, analytical System 2 reasoning frameworks [ 16 , 13 ] . While System 1 models demonstrate proficiency in generating intuitive responses with low latency, their performance degrades significantly when confronted with complex tasks. In contrast, System 2 architectures employ recursive self-evaluation mechanisms and error correction protocols to enhance correctness and robustness in multi-step reasoning scenarios. However, this methodological rigor introduces computational inefficiencies, generating excessively verbose responses. Recent advancements in L2S reasoning [ 36 ] have emerged as a promising direction to address this fundamental trade-off. Figure 1 : Overall framework of our Activation-Guided Consensus Merging Method, which extracts task-specific layer-wise activation patterns from a shared calibration corpus, quantifies their mutual information with the base model, and performs a weighted synthesis of parameters across models. Current research prioritizes two complementary strategies for optimizing reasoning efficiency, namely Chain-of-Thought (CoT) compression and task-adaptive computation allocation. The former approach targets semantic distillation of reasoning paths. For example, TokenSkip [ 40 ] implements hierarchical token importance scoring followed by strategic omission, while CoT-Valve [ 24 ] optimizes parameter space trajectories through gradient-guided step size modulation. For computational resource allocation, O1-Pruner [ 23 ] integrates reinforcement learning (RL) with curriculum-based sampling to prioritize high-complexity tasks, whereas DAST [ 33 ] establishes a token-length difficulty metric to dynamically calibrate reward functions. Parallel efforts explore budget-aware RL frameworks for adaptive reasoning length control [ 1 , 10 ] . Despite progress in prompt engineering techniques [ 36 , 23 ] , like few-shot response truncation and conciseness-oriented demonstration design [ 26 ] , these methods exhibit critical limitations. Their performance demonstrates pronounced sensitivity to different models and prompts, resulting in inconsistent generalization