Consolidating Rewarded Perturbations for LLM Post-Training
Zheyu Zhang, Shuo Yang, Gjergji Kasneci Technical University of MunichMunich Center for Machine Learning (MCML) {name.surname}@tum.de · 2026
Read Consolidating Rewarded Perturbations for LLM Post-Training by Zheyu Zhang, Shuo Yang, Gjergji Kasneci Technical University of MunichMunich Center for Machine Learning (MCML) {name.surname}@tum.de (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-31494.txt.
Introduction
Zhang, Yang, and Kasneci present CoRP, a way to fold rewarded perturbations into a single model for language-model post-training. Recent weight-space methods sample noisy variants around a base model and ensemble the best rewarded ones at test time. That costs many forward passes per example. CoRP instead combines reward-weighted updates with a held-out validation gate, using no gradients through the model. Across five models and five tasks, the average gain over the base model is 8.1 points.
Catalog record
| CREATOR | Zheyu Zhang, Shuo Yang, Gjergji Kasneci Technical University of MunichMunich Center for Machine Learning (MCML) {name.surname}@tum.de |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52806 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.31494
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Consolidating Rewarded Perturbations for LLM Post-Training Zheyu Zhang, Shuo Yang, Gjergji Kasneci Technical University of MunichMunich Center for Machine Learning (MCML) {name.surname}@tum.de Abstract Post-training of language models is commonly framed as a sample-score-update loop implemented by gradient descent. A recent line of work, exemplified by RandOpt, relocates this loop to weight space, sampling Gaussian perturbations around a pretrained model and ensembling the top- K K rewarded specialists at inference. While competitive with PPO and GRPO under matched training compute, this prediction-level ensemble incurs K K forward passes per test example and does not extend cleanly to free-form generation. We ask whether the rewarded population can instead be folded into a single deployable model, replacing the inference-time ensemble with one consolidated update. A split-half analysis over 25 model-task pairs reveals reproducible low-rank structure in every case. We turn this geometry into CoRP ( Co nsolidating R ewarded P erturbations), a gradient-free operator that combines reward-weighted aggregation, compatibility-aware reweighting, and a held-out validation gate, with no gradient flowing through the language model. Across five language models from 0.5 0.5 B to 8 8 B and five tasks covering math, code, and creative writing, CoRP improves the base model by 8.1 8.1 points on average. Using one tenth of RandOpt’s perturbation budget, CoRP exceeds single-inference RandOpt by 6.5 6.5 points and recovers more than half of the gain of the 50-pass majority-vote ensemble, at one forward pass per test example. 1 1 footnotetext: Equal contribution. 1 Introduction Post-training is the alignment stage, where a broadly capable language model is adapted to produce responses preferred by humans while maintaining or improving performance, especially on reasoning-intensive tasks. Two families dominate current practice. Reinforcement learning with verifiable rewards (RLVR) ( 29 ; 36 ; 38 ; 18 ) drives much of the recent progress on reasoning models ( 12 ; 28 ; 43 ) , and on-policy distillation (OPD) ( 1 ) trains the model on its own trajectories under a stronger teacher and now appears as a core stage in pipelines like Qwen3 and MiMo ( 50 ; 42 ) . Different as the two families look, both follow the same pattern: sample from the model, derive a per-trajectory or per-token learning signal, and update the weights to amplify capabilities that pretraining placed within reach. Output space Base model \theta_{0} Sampled trajectories \tau Reward / teacher signal Updated \theta^{\prime} sample score gradient RLVR, OPD Weight space Base weights \theta_{0} Perturbations \theta_{0}+\Delta_{i} Support-set rewards r_{i} Updated \hat{\theta} sample score select RandOpt: vote CoRP: merge Figure 1: Two routes for self-elicitation post-training. Output-space methods sample trajectories and fold reward or teacher feedback back through gradients. Weight-space methods sample perturbations around \theta_{0} and fold rewarded variation back by aggregation. CoRP follows the weight-space route and returns a single deployable model instead of an inference-time ensemble. A growing body of evidence suggests that what these updates accomplish is mainly to activate capabilities pretraining already made accessible. Chain-of-thought prompting elicits multi-step reasoning from models that were never trained on intermediate traces ( 47 ; 17 ) . RLVR continues to improve a base model even when its reward is partially incorrect or spurious, which motivates the hypothesis that the gradient update may operate beyond direct reward maximization ( 37 ) . Mechanistic studies of reasoning fine-tuning reach a similar conclusion. They find that the update repurposes directions that are already active in the base model rather than installing new ones ( 46 ; 21 ; 9 ) . Taken together, these results suggest that post-training often exposes and stabilizes capabilities the pretraining step had already placed within reach. As our baseline, 8 push this view from output space to weight space. The neighborhood of a contemporary pretrained model is dense with task specialists, and a single round of rewarded Gaussian perturbations is enough to expose them. RandOpt evaluates each perturbation on a small support set, keeps the top- K specialists, and votes their predictions at inference. The resulting ensemble matches or exceeds common post-training methods like PPO and GRPO at equal training compute, with no gradient through the language model. Where output-space recipes sample what the model can say, RandOpt samples what the model could be (see Figure 1 ). This is effective at training but expensive at inference. Good performance needs K\approx 50 , which means 50 forward passes per test example, and the prediction-level vote has to be redesigned for tasks whose outputs are not categorical. The natural fix is to consolidate the rewarded population into a single weight update. Reward-weighted averaging is the obvious first attempt, but rewarded perturbations behave as nearly orthogonal specialists, so naive averaging cancels rather than combines. Existing model-merging tools ( 48 ; 7 ; 16 ; 49 ; 51 ; 24 ) assume independently fine-tuned checkpoints. Since random rewarded perturbations around a single base model are not fine-tuned checkpoints, their compatibility must be evaluated directly. Our study begins with a measurement: Across 25 model-task pairs spanning five families and five tasks, a reward-driven random search around one base model concentrates its useful variance on a reproducible low-rank subspace in every single case we examine. A reproducible mean direction, by contrast, exists in only 40 percent of the cases. These results indicate the presence of shared structure, which is distributed across a family of compatible directions rather than a single broad mean. Building on this, we introduce Consolidating Rewarded Perturbations (CoRP 1 1 1 Our code is available at https://github.com/oooranz/CoRP . ), a gradient-free operator that turns a rewarded perturbation population into one deployable update. CoRP first forms a reward-weighted candidate direction. It then reweights each perturbation by its alignment with that direction and its orthogonal dispersion, suppressing high-reward perturbations that would inject incompatible structure. A held-out validation gate commits a candidate only when its gain is stable on examples that did not contribute to its construction ( 41 ; 19 ) , and lets the method abstain when no candidate is reliable. Across five language models from 0.5 B to 8 B parameters and five tasks, CoRP uses 500 rewarded perturbations, one tenth of RandOpt’s budget, and recovers more than half of its K{=}50 ensemble gain at one forward pass per test example. 2 Background 2.1 Rewarded Perturbations 8 treat post-training as a problem of sampling and filtering in weight space. Let \theta_{0}\in\mathbb{R}^{d} denote the pretrained weights of a language model. Given a finite set of noise scales \Sigma\subset\mathbb{R}_{>0} and a sample size N , RandOpt draws N Gaussian perturbations of \theta_{0} , \Delta_{i}\;=\;\sigma_{i}\,\xi_{i},\qquad\xi_{i}\sim\mathcal{N}(0,I_{d}),\qquad\sigma_{i}\in\Sigma,\qquad i=1,\dots,N, (1) where each \sigma_{i} is sampled uniformly from \Sigma and each \xi_{i} is an independent isotropic Gaussian direction in \mathbb{R}^{d} . RandOpt then scores each perturbed model on a small support set D_{\mathrm{sup}} of task examples, r_{i}\;=\;\frac{1}{|D_{\mathrm{sup}}|}\sum_{x\in D_{\mathrm{sup}}}R(\theta_{0}+\Delta_{i};\;x), (2) where R(\theta;\,x)\in\mathbb{R} is the task reward of the model with weights \theta on input x . We refer to the resulting set \mathcal{P}=\{(\Delta_{i},r_{i})\}_{i=1}^{N} a rewarded perturbation population . It is the basic object our method operates on. 2.2 Prediction-Level Ensembling RandOpt predicts by retaining the top- K entries of a rewarded population and voting over the corresponding perturbed models, E_{K}=\operatorname{TopK}\!\big(\{r_{i}\}_{i=1}^{N}\big),\quad\hat{y}(x^{\prime})=\operatorname{MajVote}\!\big(\{f(\theta_{0}+\Delta_{i};\,x^{\prime}):i\in E_{K}\}\big), (3) where f(\theta;\,x) denotes the model output at input x under weights \theta . The ensemble works