E-PMQ: Expert-Guided Post-Merge Quantization with Merged-Weight Anchoring

Wenjun Wang, Yanggan Gu, Shuo Cai, Yuanyi Wang, Pengkai Wang, Jianmin Wu, Hongxia Yang · 2026

Read E-PMQ: Expert-Guided Post-Merge Quantization with Merged-Weight Anchoring by Wenjun Wang, Yanggan Gu, Shuo Cai, Yuanyi Wang, Pengkai Wang, Jianmin Wu, Hongxia Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16882.txt.

Introduction

Wang and a team present E-PMQ, a way to shrink merged neural models to low-bit form for weak devices. Quantization of a merged model fails because two errors mix: one from shrinking bits and one inherited from merging experts. Their method uses source expert weights as targets during layer tuning and anchors the merged weights to keep the combined behavior steady. On merged vision and language tests, it lifts low-bit scores above GPTQ, including a jump from about 35 percent to 77 percent on a 20-task vision merge.

Catalog record

CREATORWenjun Wang, Yanggan Gu, Shuo Cai, Yuanyi Wang, Pengkai Wang, Jianmin Wu, Hongxia Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66419
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.16882

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

E-PMQ: Expert-Guided Post-Merge Quantization with Merged-Weight Anchoring Wenjun Wang, Yanggan Gu, Shuo Cai, Yuanyi Wang, Pengkai Wang, Jianmin Wu, Hongxia Yang Abstract Low-resource deployment constraints have made model quantization essential for deploying neural networks while preserving performance. Meanwhile, model merging has become an increasingly practical low-resource strategy for integrating multiple task- or domain-specialized experts into a single model without joint training or multi-model serving. Together, quantization and model merging enable an efficient low-resource deployment pipeline by integrating multiple experts into one low-bit model. We formulate this setting as Post-Merge Quantization (PMQ). We show that directly applying post-training quantization (PTQ) to a merged model is unreliable because two distinct deviations are coupled: the quantization deviation introduced by low-bit reconstruction and the expert-relative merging deviation inherited from model merging. To mitigate these deviations, we propose E-PMQ , an expert-guided PMQ framework that uses source expert weights to provide expert-guided output targets during layer-wise calibration, together with merged-weight anchoring to stabilize the calibration and preserve the integrated behavior of the merged model. On CLIP-ViT-B/32 eight-task merging, E-PMQ improves 4-bit GPTQ from 65.0% to 73.6% under Task Arithmetic and from 69.1% to 74.8% under TIES-Merging. On harder settings, E-PMQ improves GPTQ from 34.8% to 76.7% on 20-task CLIP-ViT-L/14 and from 78.26% to 83.34% on FLAN-T5-base GLUE. These results demonstrate that E-PMQ enables effective post-merge quantization and low-bit deployment. 1 Introduction Low-resource deployment constraints have made model quantization essential for deploying neural networks while preserving performance. Low-bit post-training quantization (PTQ) is one of the most practical techniques for this setting, as it converts full-precision weights into low-bit representations using only a small calibration set and without expensive end-to-end retraining. Existing PTQ methods have achieved strong results for independently trained models, where the full-precision model is typically treated as a reliable reconstruction target during layer-wise quantization ( 9 ; 18 ; 37 ; 20 ; 17 ) . Model merging is also an increasingly practical low-resource strategy. Instead of jointly training a multi-task model or serving multiple experts, merging integrates several task- or domain-specialized models into a single model ( 36 ; 13 ; 19 ; 40 ; 43 ; 3 ) . This makes merging attractive for resource-constrained adaptation and deployment: the resulting model can combine capabilities from multiple experts while avoiding multi-model serving. However, a merged model is not necessarily an independently optimized multi-task model. Since it is obtained through parameter composition, it may already deviate from the expert behaviors that merging aims to preserve. These two low-resource techniques naturally meet in deployment: after experts are merged into a single model, the resulting model may still need to be quantized for low-bit inference. We formulate this setting as Post-Merge Quantization (PMQ), where the quantization target is a merged model rather than an independently trained model. This distinction is important because naive PMQ couples two distinct deviations. The first is the quantization deviation introduced by low-bit reconstruction. The second is the expert-relative merging deviation inherited from model merging. Directly applying ordinary PTQ methods such as GPTQ ( 9 ) to a merged model only reconstructs the merged model itself, and therefore treats this potentially deviated model as the sole target. As a result, naive PMQ may preserve expert-relative merging deviations and further compound them with quantization deviation, making the standard merge-then-quantize pipeline unreliable, especially under aggressive low-bit settings. To mitigate these deviations, we propose E-PMQ , an expert-guided PMQ framework with merged-weight anchoring. During layer-wise calibration, E-PMQ uses source expert weights to provide expert-guided output targets. These targets introduce expert-relative guidance into the quantization process, rather than passively reconstructing only the merged model. Together with this expert guidance, merged-weight anchoring stabilizes the calibration and preserves the integrated behavior of the merged model. The expert models are accessed only during the post-merge calibration stage. After quantization, the deployed model remains a single low-bit merged model, without experts or additional inference-time modules. Figure 1 illustrates this distinction. Figure 1 : Overview of ordinary PTQ, naive PMQ, and E-PMQ. Ordinary PTQ quantizes a trained model by reconstructing a reliable full-precision target. Naive PMQ first merges multiple expert checkpoints and then directly quantizes the merged model, thereby reconstructing an imperfect merged target and suffering from accumulated merging and quantization deviation. E-PMQ instead uses expert-guided output targets during layer-wise calibration and anchors the quantized weight to the merged checkpoint W_{m} , turning post-merge quantization into expert-guided calibration with merged-weight anchoring. Experiments show that E-PMQ consistently improves low-bit merged models across vision and text settings. On CLIP-ViT-B/32 eight-task merging, E-PMQ improves 4-bit GPTQ from 65.0% to 73.6% under Task Arithmetic and from 69.1% to 74.8% under TIES-Merging. The gains remain strong in harder settings: under Task Arithmetic, E-PMQ improves GPTQ from 34.8% to 76.7% on 20-task CLIP-ViT-L/14 and from 78.26% to 83.34% on FLAN-T5-base GLUE. Further experiments show consistent gains across merging methods, task scales, modalities, and quantization bit-widths. We summarize the main contributions of this work as follows: ❶ We formulate Post-Merge Quantization (PMQ) as a distinct low-bit deployment setting for merged models, and identify a key failure mode of naive PMQ: directly reconstructing the merged model couples the quantization deviation introduced by low-bit reconstruction with the expert-relative merging deviation inherited from model merging. ❷ We introduce E-PMQ , an expert-guided PMQ framework that uses source expert weights to provide expert-guided output targets during layer-wise calibration, together with merged-weight anchoring to stabilize the calibration and preserve the integrated behavior of the merged model. ❸ We validate E-PMQ on CLIP and FLAN-T5, showing consistent gains over naive PMQ baselines such as GPTQ across merging methods, task scales, modalities, and quantization bit-widths. 2 Related Work Model Merging. Model merging composes multiple specialized models into a single model without joint training or deploying one model per task. Existing methods include weight averaging, Fisher merging, task arithmetic, TIES-Merging, DARE, and adaptive or data-free task-vector approaches ( 36 ; 19 ; 13 ; 40 ; 43 ) . Recent surveys and systems work frame model fusion as a scalable alternative to repeatedly training or serving many experts ( 45 ; 44 ; 32 ; 33 ) , while broader fusion methods explore preference- or distillation-based composition ( 11 ; 34 ) . These works focus on building, scaling, or managing merged models; our work instead studies how to quantize an already merged model more reliably. Post-training quantization. Post-training quantization compresses a trained full-precision model into low-bit weights without end-to-end retraining, typically through calibration-based rounding, scaling, or layer-wise reconstruction ( 20 ; 17 ; 9 ; 18 ; 37 ; 42 ) . Ordinary PTQ generally assumes that the full-precision model is a reliable target to preserve, which is natural for independently trained models but less reliable for merged models. Low-precision training and inference recipes further highlight the importance of numerical efficiency for scalable deployment ( 31 ) . Our work studies PMQ, where naive merge-then-quantize baselines apply ordinary PTQ such as GPTQ to a merged model. Instead of only reconstructing the merged model, E-PMQ uses source expert weights during layer-wise quantization to construct expert-guided calibration targets and anchors the solution to the merged model for stability. 3 Preliminaries and Problem Formulation Notation. Let \{W_{i}\}_{i=1}^{K} denote K task-specialized expert models, and W_{m}=\mathcal{M}(\{W_{i}\}_{i=1}^{K}) be the merged model

Read the complete text