InfiGFusion: Graph-on-Logits Distillation via Efficient Gromov-Wasserstein for Model Fusion

Yuanyi Wang, Zhaoyi Yan, Yiming Zhang, Qi Zhou, Yanggan Gu, Fei Wu, Hongxia Yang · 2025

Read InfiGFusion: Graph-on-Logits Distillation via Efficient Gromov-Wasserstein for Model Fusion by Yuanyi Wang, Zhaoyi Yan, Yiming Zhang, Qi Zhou, Yanggan Gu, Fei Wu, Hongxia Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13893.txt.

Introduction

Yuanyi Wang and six co-authors present a way to merge large language models in a twenty twenty-five paper. The aim is one model that keeps the strengths of many open models. Past merge methods work fast but treat each word slot on its own, which misses how word slots link to each other.

Their method, InfiGFusion, builds a graph from the top scores a model gives for next words. Links in the graph show which words tend to fire at once. A fast math trick cuts the cost of matching such graphs by a huge amount. In tests on eleven tasks, the merged models beat past methods, with big gains on multi-step math and cause-and-effect tasks.

Catalog record

CREATORYuanyi Wang, Zhaoyi Yan, Yiming Zhang, Qi Zhou, Yanggan Gu, Fei Wu, Hongxia Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE115 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-13893

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

InfiGFusion: Graph-on-Logits Distillation via Efficient Gromov-Wasserstein for Model Fusion Yuanyi Wang, Zhaoyi Yan, Yiming Zhang, Qi Zhou, Yanggan Gu, Fei Wu, Hongxia Yang Abstract Recent advances in large language models (LLMs) have intensified efforts to fuse heterogeneous open-source models into a unified system that inherits their complementary strengths. Existing logit-based fusion methods maintain inference efficiency but treat vocabulary dimensions independently, overlooking semantic dependencies encoded by cross-dimension interactions. These dependencies reflect how token types interact under a model’s internal reasoning and are essential for aligning models with diverse generation behaviors. To explicitly model these dependencies, we propose InfiGFusion , the first structure-aware fusion framework with a novel Graph-on-Logits Distillation (GLD) loss. Specifically, we retain the top- k k logits per output and aggregate their outer products across sequence positions to form a global co-activation graph, where nodes represent vocabulary channels and edges quantify their joint activations. To ensure scalability and efficiency, we design a sorting-based closed-form approximation that reduces the original O ⁡ ( n 4 ) O(n^{4}) cost of Gromov-Wasserstein distance to O ⁡ ( n ​ log ⁡ n ) O(n\log n) , with provable approximation guarantees. Experiments across multiple fusion settings show that GLD consistently improves fusion quality and stability. InfiGFusion outperforms SOTA models and fusion baselines across 11 benchmarks spanning reasoning, coding, and mathematics. It shows particular strength in complex reasoning tasks, with +35.6 improvement on Multistep Arithmetic and +37.06 on Causal Judgement over SFT, demonstrating superior multi-step and relational inference. 1 Introduction Recent advances in LLMs have sparked growing interest in aggregating diverse model capabilities to build stronger, more general-purpose systems. Existing collective approaches span a broad design space. Prompt-based techniques Schick et al. (2023) ; Yang et al. (2023) dynamically chain or compose pretrained LLMs with tools or APIs to expand functional scope. Multi-agent systems Wang et al. (2024a) ; Dubois et al. (2024) employ multiple LLMs to collaborate or debate, often outperforming single-agent models on complex reasoning tasks. Model ensembling Jiang et al. (2023) ; Fang et al. (2024) aggregates outputs from different models to boost accuracy and robustness, while Mixture-of-Experts (MoE) Komatsuzaki et al. () ; Du et al. (2022) ; Chen et al. (2022) architectures improve scalability by activating specialized subnetworks. Meanwhile, parameter-level merging models Yadav et al. (2023) ; Matena and Raffel (2022) ; Wortsman et al. (2022) aim to consolidate multiple fine-tuned checkpoints into a single model. Each of these paradigms explores a different trade-off between performance, efficiency, and compatibility. Among fusion strategies, model fusion via logit distillation Wan et al. () ; Shi et al. (2024) has emerged as a flexible and efficient paradigm. It enables a single pivot model to absorb knowledge from multiple source models—often with diverse domains, sizes, or architectures—without increasing inference cost. A representative cross-tokenizer approach is ULD Boizard et al. () , which introduces a universal logit-distillation loss for aligning models with different tokenizers. Unlike traditional distillation, which transfers intermediate features or outputs, these methods align only the output logits using token-level objectives such as KL divergence or Wasserstein distance Yan et al. (2025) . However, token-level methods inherently treat each vocabulary dimension independently, overlooking semantic dependencies —the co-activation patterns and relational structures among tokens that underlie a model’s reasoning process. This limits their ability to align models with divergent generation behaviors, particularly for tasks requiring multi-step reasoning or fine-grained relational inference. Fig. 1 illustrates this limitation. While token-level losses enforce dimension similarity, they fail to capture the structured dependencies among tokens. InfiGFusion addresses this by modeling logits as graphs, aligning not just distributions but also the interactions that reflect a model’s internal reasoning. Figure 1: Token-level vs. Structure-aware Fusion. Given pivot and source logits of shape [L,3] (sequence length L , vocab size 3 ), token-level methods (left) align dimensions independently, ignoring token interactions. GLD (right) aggregates outer products into [3,3] co-activation graphs, capturing semantic dependencies via structure-aware graph alignment. To address this challenge, we model semantic dependencies in the logit space using graph structures. Rather than treating token logits independently, we represent model outputs as feature-level graphs, where nodes correspond to token dimensions and edges capture their co-activation patterns across sequence positions. This representation reflects how a model internally organizes semantic relationships, beyond surface-level token probabilities, to support coherent reasoning and inference. Constructing such graphs from high-dimensional logits is nontrivial: vocabulary sizes are large, and most logits are near-zero Zhuang et al. (2024) ; Lu et al. (2024) , making full pairwise modeling inefficient and noisy. To mitigate this, we apply top- k sparsification to retain salient dimensions, followed by inner-product similarity to define edge weights. This yields interpretable graphs that faithfully reflect model reasoning behaviors. Building on this, we propose InfiGFusion , a semantic structure-aware fusion framework that distills knowledge from multiple source models by aligning their logit graphs. At its core is the Graph-on-Logits Distillation (GLD) loss, which captures semantic dependencies by aligning structure-level information between source and pivot models. Specifically, we employ the Gromov-Wasserstein (GW) distance to match graph structures Xu et al. (2019) ; Peyré et al. (2016) . However, standard GW incurs \mathcal{O}(n^{4}) complexity, making it impractical for large vocabularies. To address this, we design a closed-form approximation based on node aggregation and sorting-based matching, reducing the cost to \mathcal{O}(n\log n) . We further provide theoretical guarantees on the approximation error, ensuring alignment fidelity. We validate InfiGFusion across diverse fusion scenarios, demonstrating consistent improvements over state-of-the-art methods. By explicitly modeling semantic dependencies through logit graph alignment, InfiGFusion excels in tasks involving multi-step reasoning, relational inference, and fine-grained alignment—scenarios where token-level objectives fall short. Extensive experiments on 11 benchmarks, including reasoning, mathematics, and coding, confirm its effectiveness in producing more capable fusion models. Overall, our contributions can be summarized as follows: • InfiGFusion is the first structure-aware fusion framework that models semantic dependencies via feature-level logit graphs and aligns them via a novel Graph-on-Logits Distillation loss. • We propose a novel approximation to Gromov-Wasserstein distance, reducing complexity from \mathcal{O}(n^{4}) to \mathcal{O}(n\log n) , with provable error bounds and rigorous theoretical analysis. • Experiments on 11 benchmarks show that InfiGFusion consistently outperforms baselines, with significant gains on complex reasoning tasks (+35.6 on Multistep Arithmetic, +37.06 on Causal Judgement), demonstrating superior multi-step and relational inference. 2 Related Work Collective LLM: To aggregate the capabilities of specialized LLMs, several strategies have been proposed. Prompt-based integration methods, such as Toolformer Schick et al. (2023) , ReAct Yao et al. (2023) , and HuggingGPT Shen et al. (2023) , guide LLMs to invoke tools or auxiliary models through natural language prompts. Multi-agent collaboration frameworks like ChatArena Zheng et al. (2023) and AgentBench Liu et al. (2024) treat LLMs as interacting agents that debate or coordinate to improve reasoning. Model ensembling aggregates outputs via voting Wang et al. () or reranking, but suffers from high inference cost. Mixture-of-Experts (MoE) models Komatsuzaki et al. () ; Du et al. (2022) ; Chen et al. (2022) activate sparse expert subsets per query, offering scalability but introducing routing complexity. Parameter merging methods Yadav et al. (2023) ; Matena and Raffel (2022) ; Wortsman et al. (2022) combine weights from fine-tuned models, but require architectural alignment. While these strategies explore different trade-offs between performance, efficiency, and flexibility, they

Read the complete text