Weight Ensembling Improves Reasoning in Language Models
Xingyu Dang, Christina Baek, Kaiyue Wen, Zico Kolter, Aditi Raghunathan · 2025
Read Weight Ensembling Improves Reasoning in Language Models by Xingyu Dang, Christina Baek, Kaiyue Wen, Zico Kolter, Aditi Raghunathan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10478.txt.
Introduction
Xingyu Dang and four co-authors report a simple fix for a training flaw in reasoning models in this two thousand twenty-five paper. As fine-tuning proceeds, a model's answers grow more alike. Its first guess keeps improving, but its chance of solving a problem in many tries falls. That loss hurts test-time strategies like majority voting. The authors blend the weights of the final model with an early checkpoint, a trick called WiSE-FT.
The blend nearly restores the lost diversity while keeping the first-guess gains. It also improves voting scores and later reinforcement learning results. Extra gains appear that decoding tricks alone cannot give. The team frames the effect as a bias-variance trade-off. The recipe is cheap and easy for model builders to adopt.
Catalog record
| CREATOR | Xingyu Dang, Christina Baek, Kaiyue Wen, Zico Kolter, Aditi Raghunathan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-10478
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Weight Ensembling Improves Reasoning in Language Models Xingyu Dang, Christina Baek, Kaiyue Wen, Zico Kolter, Aditi Raghunathan Abstract We investigate a failure mode that arises during the training of reasoning models, where the diversity of generations begins to collapse, leading to suboptimal test-time scaling. Notably, the Pass@1 rate reliably improves during supervised finetuning (SFT), but Pass@k rapidly deteriorates. Surprisingly, a simple intervention of interpolating the weights of the latest SFT checkpoint with an early checkpoint, otherwise known as WiSE-FT, almost completely recovers Pass@k while also improving Pass@1. The WiSE-FT variant achieves better test-time scaling (Best@k, majority vote) and achieves superior results with less data when tuned further by reinforcement learning. Finally, we find that WiSE-FT provides complementary performance gains that cannot be achieved only through diversity-inducing decoding strategies, like temperature scaling. We formalize a bias-variance tradeoff of Pass@k with respect to the expectation and variance of Pass@1 over the test distribution. We find that WiSE-FT can reduce bias and variance simultaneously, while temperature scaling inherently trades off between bias and variance. 1 Introduction Recent advances in large language models (LLMs) have showcased their remarkable ability to perform complex reasoning, yet these successes often hinge on test-time scaling strategies ( Lightman et al. 2023 ; Snell et al. 2024 ; Wu et al. 2024 ) . In many applications, such as math problems, puzzles, and logical reasoning, LLMs employ a verification framework where it is significantly easier for the model to verify a candidate solution than to generate one from scratch. This distinction has given rise to strategies that sample multiple “reasoning traces” or sequences of reasoning steps during inference, selecting the best final guess through an outcome reward model (ORM) or majority vote. In this setting, an upper bound on the performance a model could achieve is measured by \mathrm{Pass@K} , or the probability that at least one out of K independently sampled reasoning traces is correct. Unfortunately, while the standard training pipeline of supervised finetuning (SFT) followed by reinforcement learning (RL) dependably improves \mathrm{Pass@1} for reasoning, \mathrm{Pass@K} tends to drop early into finetuning ( Cobbe et al. 2021 ; Chow et al. 2024a ; Chen et al. 2025 ) . This mismatch arises from a symptom of finetuning called diversity collapse, where overtuned models yield less diverse generations. This is detrimental to \mathrm{Pass@K} since the model wastes K attempts on only a handful of guesses. In fact, by analyzing the model’s error rate i.e., 1-\mathrm{Pass@1} , across the test distribution, we derive a \mathbf{\mathrm{Pass@K}} bias-variance trade-off . To improve expected test \mathrm{Pass@K} , one can either reduce the bias which is the expected error rate or how much the model’s error rate varies across problems. The latter term is connected to diversity — more diversity allows models to hedge and do uniformly well across all test questions. In particular, during SFT, \mathrm{Pass@1} improves (bias \downarrow ) at the cost of diversity collapse (variance \uparrow ). Surprisingly, common ways of alleviating diversity collapse, such as early stopping at peak \mathrm{Pass@K} or decoding with high temperature, suffer from the reverse trade-off : diversity improves (variance \downarrow ) at the cost of overall \mathrm{Pass@1} degrading (bias \uparrow ). Consequently, in this paper we are concerned with a central question: Is it possible to simultaneously improve both \mathrm{Pass@1} and \mathrm{Pass@K} , thereby overcoming the bias-variance tradeoff inherent in current approaches? In our work, we introduce a simple, scalable and effective intervention that allows models to achieve both high \mathrm{Pass@K} and \mathrm{Pass@1} across mathematical reasoning tasks GSM8k, MATH, and AIME. The specific technique we use is a variant of WiSE-FT ( Wortsman et al. 2022 ) where we interpolate the weights of the latest SFT checkpoint \mathbf{w}_{t} with an early checkpoint w_{0} as \mathbf{w}_{\mathsf{WiSE}(t)}=\nicefrac{{1}}{{2}}\cdot\mathbf{w}_{0}+\nicefrac{{1}}{{2}}\cdot\mathbf{w}_{t} . Our key finding is that WiSE-FT successfully merges the diverse sampling capabilities of earlier checkpoints while retaining or surpassing the \mathrm{Pass@1} of later checkpoints. In Figure 1 , we observe that the WiSE-FT model achieves both higher \mathrm{Pass@K} and \mathrm{Pass@1} with more SFT steps t , unlike naive SFT which suffers from an early decay in \mathrm{Pass@K} . Moreover, the gains with WiSE-FT is unachievable by early-stopping or diversity-aware decoding alone. Figure 1 : Pass@k of WiSE-FT versus SFT on GSM8k Gemma-2-2B supervised finetuned and evaluated on GSM8k. At each SFT timestep t , we evaluate Pass@k of checkpoint \mathbf{w}_{t} (in dashed) with its WiSE-FT variant \nicefrac{{1}}{{2}}\cdot\mathbf{w}_{t}+\nicefrac{{1}}{{2}}\cdot\mathbf{w}_{0} (in solid), where traces are independently sampled with temperature T=[0.7,1.0,1.3,1.6] . Thus, we propose a new paradigm of training reasoning models: 1.) Train extensively using SFT as long as \mathrm{Pass@1} improves, 2.) Perform WiSE-FT with an earlier SFT checkpoint, 3.) Continue tuning the WiSE-FT variant using RL. Overall, the WiSE-FT model has the following immediate practical benefits: • Better Test-Time Scaling Across all datasets and base models, the WiSE-FT variant achieves the highest performance with test-time scaling (Majority Vote, ORM) compared to an overtrained SFT model paired with diversity-aware decoding. • Better Reinforcement Learning Since RL uses self-generated data to tune models, to generalize reliably, it is important for generations to provide sufficient learning signal while also having high coverage over the data space. We find that continued RL training starting from WiSE-FT weights achieves superior results with less synthetic data compared to initializing RL from the last SFT checkpoint and even early-stopped SFT. In summary, we provide a comprehensive analysis of how reasoning models suffer from diversity collapse during SFT and its negative downstream impact during RL and test-time scaling. We first discuss our WiSE-FT findings in § 4 . Motivated by this discovery, we investigate two fundamental questions. First, we investigate diversity collapse during SFT and RL of reasoning models in § 5 . Diversity collapse not only impacts the model’s ability to attempt different guesses. In fact, we make an even stronger observation — the generations of reasoning models converge towards a single reasoning trace for each test question. We theoretically prove that standard RL algorithms (i.e., REINFORCE and GRPO) fail to recover lost diversity in a simplified discrete bandit setting. Second, we formalize the competing goals of \mathrm{Pass@1} and \mathrm{Pass@K} as a bias-variance trade-off in § 6 . We empirically measure and compare the bias and variance of WiSE-FT versus early-stopping versus high temperature decoding. Notably, only WiSE-FT reduces both bias and variance. We conclude with a remark on the limitations of decoding strategies such as top-k ( Shao et al. 2017 ) , nucleus ( Holtzman et al. 2020 ) , and min-p ( Nguyen et al. 2024 ) , at eliciting the maximum capabilities with test-time scaling from current reasoning models. 2 Related Works Diversity collapse with SFT: The standard pipeline for enhancing reasoning in LLMs involves an initial phase of supervised fine-tuning (SFT) followed by reinforcement learning (RL) ( Guo et al. 2025 ; Setlur et al. 2024 ) . SFT is critical for instilling interpretable and readable reasoning chains and ensuring that the model adheres to a consistent rollout templates ( Guo et al. 2025 ) . However, a number of recent works have identified critical pitfalls of SFT that hinders the model’s ability to explore and ultimately it’s overall problem solving ability. Notably, Cobbe et al. 2021 observe diversity collapse when finetuning on