AdaFuse: Adaptive Ensemble Decoding with Test-Time Scaling for LLMs
Chengming Cui, Tianxin Wei, Ziyi Chen, Ruizhong Qiu, Zhichen Zeng, Zhining Liu, Xuying Ning, Duo Zhou, Jingrui He · 2026
Read AdaFuse: Adaptive Ensemble Decoding with Test-Time Scaling for LLMs by Chengming Cui, Tianxin Wei, Ziyi Chen, Ruizhong Qiu, Zhichen Zeng, Zhining Liu, Xuying Ning, Duo Zhou, Jingrui He (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-06022.txt.
Introduction
Chengming Cui and 8 co-authors propose AdaFuse in a 2026 paper on combining several large language models while they write. Different models have different strengths from their training and design. Mixing their outputs at answer time can help, but fixed mixing rules cannot adapt mid-sentence.
AdaFuse chooses fitting fusion units on the fly, using words as building blocks for alignment. An uncertainty check decides at each step whether mixing is needed. When the model is confident it writes alone; when unsure, it explores varied continuations from other models. On question answering, arithmetic reasoning, and translation tests, AdaFuse beat strong ensemble baselines by a clear margin on average.
Catalog record
| CREATOR | Chengming Cui, Tianxin Wei, Ziyi Chen, Ruizhong Qiu, Zhichen Zeng, Zhining Liu, Xuying Ning, Duo Zhou, Jingrui He |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-06022
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AdaFuse: Adaptive Ensemble Decoding with Test-Time Scaling for LLMs Chengming Cui, Tianxin Wei, Ziyi Chen, Ruizhong Qiu, Zhichen Zeng, Zhining Liu, Xuying Ning, Duo Zhou, Jingrui He Abstract Large language models (LLMs) exhibit complementary strengths arising from differences in pretraining data, model architectures, and decoding behaviors. Inference-time ensembling provides a practical way to combine these capabilities without retraining. However, existing ensemble approaches suffer from fundamental limitations. Most rely on fixed fusion granularity, which lacks the flexibility required for mid-generation adaptation and fails to adapt to different generation characteristics across tasks. To address these challenges, we propose AdaFuse , an adaptive ensemble decoding framework that dynamically selects semantically appropriate fusion units during generation. Rather than committing to a fixed granularity, AdaFuse adjusts fusion behavior on the fly based on the decoding context, with words serving as basic building blocks for alignment. To be specific, we introduce an uncertainty-based criterion to decide whether to apply ensembling at each decoding step. Under confident decoding states, the model continues generation directly. In less certain states, AdaFuse invokes a diversity-aware scaling strategy to explore alternative candidate continuations and inform ensemble decisions. This design establishes a synergistic interaction between adaptive ensembling and test-time scaling, where ensemble decisions guide targeted exploration, and the resulting diversity in turn strengthens ensemble quality. Experiments on open-domain QA, arithmetic reasoning, and machine translation demonstrate that AdaFuse consistently outperforms strong ensemble baselines, achieving an average relative improvement of 6.88%. The code is available at https://github.com/CCM0111/AdaFuse . 1 Introduction Large language models (LLMs) have demonstrated strong performance across a wide range of natural language processing tasks 11 ; 6 ; 24 ; 32 ; 1 ; 39 ; 4 and are increasingly deployed in real-world applications 9 ; 25 . However, model performance is not uniform across tasks: differences in pretraining data, model architectures, and decoding strategies lead individual models to exhibit heterogeneous strengths 5 . Models that excel at structured or multi-step reasoning may perform less reliably on open-domain or commonsense question answering, while models optimized for such queries may struggle with complex reasoning 17 ; 18 ; 13 ; 10 . As a result, inference-time ensembling has emerged as a practical approach for improving robustness by combining complementary model strengths without retraining 16 ; 21 ; 36 ; 31 . Existing inference-time ensemble methods differ primarily in the granularity at which model outputs are fused 3 . Sample-level approaches combine complete model responses via reranking or response fusion 16 ; 29 ; 22 ; 30 , but perform fusion only after generation has finished, preventing any mid-generation correction. Span-level methods assemble multi-token segments produced by different models 21 ; 33 ; 23 ; 7 , yet rely on predefined or fixed span boundaries that limit flexibility across tasks and reasoning depths. Token-level approaches aggregate next-token distributions at each decoding step 36 ; 14 ; 37 ; 38 , but typically require token alignment across models, restricting applicability under heterogeneous tokenizers; related byte-level alternatives alleviate this mismatch at the cost of increased computation and weakened semantic structure 26 ; 28 . Despite operating at different granularities, most existing methods rely on a fixed fusion resolution, which restricts their ability to adapt the fusion scope to evolving semantic context, task demands, and reasoning uncertainty during generation 35 . These trade-offs motivate the need for an adaptive inference-time ensemble framework that overcomes the limitations imposed by fixed-granularity generation. We adopt word-level units as the basic building blocks, as they provide a natural abstraction that supports step-wise integration and semantic coherence, while avoiding explicit token alignment across heterogeneous tokenizers. Based on this design choice, we propose AdaFuse , an adaptive word-level ensembling framework that enables flexible inference-time fusion and mid-generation correction at natural word boundaries, which preserves semantic integrity across models. Concretely, AdaFuse adopts an adaptive decoding strategy that balances effectiveness and efficiency during generation. We introduce an uncertainty-based criterion to decide whether to apply ensembling at each decoding step. Under confident decoding states, the model proceeds with direct generation. In less certain states, AdaFuse invokes a diversity-aware scaling strategy to explore alternative candidate continuations and support more effective ensemble decisions. This design is guided by the evolving decoding context rather than predefined rules, and establishes a synergistic interaction between adaptive ensembling and test-time scaling. In particular, selective exploration under uncertainty improves test-time scaling effectiveness, while the resulting diversity further strengthens ensemble quality. In summary, our paper contributes the following: • Confidence-Guided Adaptive Decoding : AdaFuse uses model confidence to decide when to commit longer word spans and when to consider more candidate continuations, enabling adaptive mid-generation correction without relying on fixed-length spans. • Diversity-Aware Ensemble Scaling : Building on confidence-guided decoding, AdaFuse employs an exploration strategy to generate diverse candidate continuations for ensemble scoring only when uncertainty arises, while avoiding unnecessary computation in confident states. • Strong Empirical Performance Across Diverse Tasks : AdaFuse achieves consistent gains over the strongest ensemble baselines, with an average relative improvement of 6.88% across open-domain question answering, arithmetic reasoning, and machine translation benchmarks. 2 Related Works Ensemble methods in LLMs have evolved significantly, with prominent approaches categorized into three main levels: Sample-level, Span-level, Token-level ensembling , each designed to harness complementary model strengths yet plagued by its own drawbacks. Sample - Level Ensembling aggregates entire generated responses from multiple LLMs. LLM - Blender 16 trains a pairwise ranker to compare and select the strongest candidate, then applies a generative fusion module to produce a final answer. Routing methods 29 ; 22 assign each query to the most suitable expert model, but their effectiveness is capped when every candidate output contains errors 12 . Fusion networks 30 go one step further by learning to merge several complete outputs into a single consolidated response, boosting overall quality but still facing challenges in generalizing across diverse tasks and overlooking valuable token - level probability cues generated during decoding 20 ; 19 . Span-Level Ensembling combines output segments. Cool-Fusion 21 merges model outputs once they reach common word boundaries, while SweetSpan 33 generates and merges spans based on perplexity scores. SpecFuse 23 aggregates multiple LLM outputs by predicting the next segment and combining them. RLAE 7 uses reinforcement learning to adjust weights dynamically during the merging process. While span-level methods improve coherence and fluency, they rely on the quality of the generated segments, making them susceptible to errors in weak segments. Furthermore, span-level methods offer limited flexibility due to their fixed fusion granularity and lack of adaptive control, making them less effective on novel or diverse inputs. Figure 1: Illustration of the AdaFuse framework. AdaFuse ensembles multiple LLMs during decoding by dynamically adjusting fusion behavior. It expands in confident decoding states and applies diversity-aware scaling under uncertainty. Candidate word sequences are unified and selected using NLL scores, enabling adaptive and stable generation. Token - Level Ensembling merges each model’s next - token distributions at every decoding step. UniTE 36 forms the union of each model’s top - k token candidates at each step, avoiding full - vocabulary alignment and cutting inference cost. DeePEn 14 projects each model’s logits into a universal relative space via anchor - token similarity, aggregates them, and applies a search - based inverse transformation to recover logits in