Consolidating Rewarded Perturbations for LLM Post-Training

Zheyu Zhang, Shuo Yang, Gjergji Kasneci Technical University of MunichMunich Center for Machine Learning (MCML) {name.surname}@tum.de · 2026

Read Consolidating Rewarded Perturbations for LLM Post-Training by Zheyu Zhang, Shuo Yang, Gjergji Kasneci Technical University of MunichMunich Center for Machine Learning (MCML) {name.surname}@tum.de (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-31494.txt.

Introduction

Zhang, Yang, and Kasneci present CoRP, a way to fold rewarded perturbations into a single model for language-model post-training. Recent weight-space methods sample noisy variants around a base model and ensemble the best rewarded ones at test time. That costs many forward passes per example. CoRP instead combines reward-weighted updates with a held-out validation gate, using no gradients through the model. Across five models and five tasks, the average gain over the base model is 8.1 points.

Catalog record

CREATORZheyu Zhang, Shuo Yang, Gjergji Kasneci Technical University of MunichMunich Center for Machine Learning (MCML) {name.surname}@tum.de
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52806
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.31494

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Consolidating Rewarded Perturbations for LLM Post-Training Zheyu Zhang, Shuo Yang, Gjergji Kasneci Technical University of MunichMunich Center for Machine Learning (MCML) {name.surname}@tum.de Abstract Post-training of language models is commonly framed as a sample-score-update loop implemented by gradient descent. A recent line of work, exemplified by RandOpt, relocates this loop to weight space, sampling Gaussian perturbations around a pretrained model and ensembling the top- K K rewarded specialists at inference. While competitive with PPO and GRPO under matched training compute, this prediction-level ensemble incurs K K forward passes per test example and does not extend cleanly to free-form generation. We ask whether the rewarded population can instead be folded into a single deployable model, replacing the inference-time ensemble with one consolidated update. A split-half analysis over 25 model-task pairs reveals reproducible low-rank structure in every case. We turn this geometry into CoRP ( Co nsolidating R ewarded P erturbations), a gradient-free operator that combines reward-weighted aggregation, compatibility-aware reweighting, and a held-out validation gate, with no gradient flowing through the language model. Across five language models from 0.5 0.5 B to 8 8 B and five tasks covering math, code, and creative writing, CoRP improves the base model by 8.1 8.1 points on average. Using one tenth of RandOpt’s perturbation budget, CoRP exceeds single-inference RandOpt by 6.5 6.5 points and recovers more than half of the gain of the 50-pass majority-vote ensemble, at one forward pass per test example. 1 1 footnotetext: Equal contribution. 1 Introduction Post-training is the alignment stage, where a broadly capable language model is adapted to produce responses preferred by humans while maintaining or improving performance, especially on reasoning-intensive tasks. Two families dominate current practice. Reinforcement learning with verifiable rewards (RLVR) ( 29 ; 36 ; 38 ; 18 ) drives much of the recent progress on reasoning models ( 12 ; 28 ; 43 ) , and on-policy distillation (OPD) ( 1 ) trains the model on its own trajectories under a stronger teacher and now appears as a core stage in pipelines like Qwen3 and MiMo ( 50 ; 42 ) . Different as the two families look, both follow the same pattern: sample from the model, derive a per-trajectory or per-token learning signal, and update the weights to amplify capabilities that pretraining placed within reach. Output space Base model \theta_{0} Sampled trajectories \tau Reward / teacher signal Updated \theta^{\prime} sample score gradient RLVR, OPD Weight space Base weights \theta_{0} Perturbations \theta_{0}+\Delta_{i} Support-set rewards r_{i} Updated \hat{\theta} sample score select RandOpt: vote CoRP: merge Figure 1: Two routes for self-elicitation post-training. Output-space methods sample trajectories and fold reward or teacher feedback back through gradients. Weight-space methods sample perturbations around \theta_{0} and fold rewarded variation back by aggregation. CoRP follows the weight-space route and returns a single deployable model instead of an inference-time ensemble. A growing body of evidence suggests that what these updates accomplish is mainly to activate capabilities pretraining already made accessible. Chain-of-thought prompting elicits multi-step reasoning from models that were never trained on intermediate traces ( 47 ; 17 ) . RLVR continues to improve a base model even when its reward is partially incorrect or spurious, which motivates the hypothesis that the gradient update may operate beyond direct reward maximization ( 37 ) . Mechanistic studies of reasoning fine-tuning reach a similar conclusion. They find that the update repurposes directions that are already active in the base model rather than installing new ones ( 46 ; 21 ; 9 ) . Taken together, these results suggest that post-training often exposes and stabilizes capabilities the pretraining step had already placed within reach. As our baseline, 8 push this view from output space to weight space. The neighborhood of a contemporary pretrained model is dense with task specialists, and a single round of rewarded Gaussian perturbations is enough to expose them. RandOpt evaluates each perturbation on a small support set, keeps the top- K specialists, and votes their predictions at inference. The resulting ensemble matches or exceeds common post-training methods like PPO and GRPO at equal training compute, with no gradient through the language model. Where output-space recipes sample what the model can say, RandOpt samples what the model could be (see Figure 1 ). This is effective at training but expensive at inference. Good performance needs K\approx 50 , which means 50 forward passes per test example, and the prediction-level vote has to be redesigned for tasks whose outputs are not categorical. The natural fix is to consolidate the rewarded population into a single weight update. Reward-weighted averaging is the obvious first attempt, but rewarded perturbations behave as nearly orthogonal specialists, so naive averaging cancels rather than combines. Existing model-merging tools ( 48 ; 7 ; 16 ; 49 ; 51 ; 24 ) assume independently fine-tuned checkpoints. Since random rewarded perturbations around a single base model are not fine-tuned checkpoints, their compatibility must be evaluated directly. Our study begins with a measurement: Across 25 model-task pairs spanning five families and five tasks, a reward-driven random search around one base model concentrates its useful variance on a reproducible low-rank subspace in every single case we examine. A reproducible mean direction, by contrast, exists in only 40 percent of the cases. These results indicate the presence of shared structure, which is distributed across a family of compatible directions rather than a single broad mean. Building on this, we introduce Consolidating Rewarded Perturbations (CoRP 1 1 1 Our code is available at https://github.com/oooranz/CoRP . ), a gradient-free operator that turns a rewarded perturbation population into one deployable update. CoRP first forms a reward-weighted candidate direction. It then reweights each perturbation by its alignment with that direction and its orthogonal dispersion, suppressing high-reward perturbations that would inject incompatible structure. A held-out validation gate commits a candidate only when its gain is stable on examples that did not contribute to its construction ( 41 ; 19 ) , and lets the method abstain when no candidate is reliable. Across five language models from 0.5 B to 8 B parameters and five tasks, CoRP uses 500 rewarded perturbations, one tenth of RandOpt’s budget, and recovers more than half of its K{=}50 ensemble gain at one forward pass per test example. 2 Background 2.1 Rewarded Perturbations 8 treat post-training as a problem of sampling and filtering in weight space. Let \theta_{0}\in\mathbb{R}^{d} denote the pretrained weights of a language model. Given a finite set of noise scales \Sigma\subset\mathbb{R}_{>0} and a sample size N , RandOpt draws N Gaussian perturbations of \theta_{0} , \Delta_{i}\;=\;\sigma_{i}\,\xi_{i},\qquad\xi_{i}\sim\mathcal{N}(0,I_{d}),\qquad\sigma_{i}\in\Sigma,\qquad i=1,\dots,N, (1) where each \sigma_{i} is sampled uniformly from \Sigma and each \xi_{i} is an independent isotropic Gaussian direction in \mathbb{R}^{d} . RandOpt then scores each perturbed model on a small support set D_{\mathrm{sup}} of task examples, r_{i}\;=\;\frac{1}{|D_{\mathrm{sup}}|}\sum_{x\in D_{\mathrm{sup}}}R(\theta_{0}+\Delta_{i};\;x), (2) where R(\theta;\,x)\in\mathbb{R} is the task reward of the model with weights \theta on input x . We refer to the resulting set \mathcal{P}=\{(\Delta_{i},r_{i})\}_{i=1}^{N} a rewarded perturbation population . It is the basic object our method operates on. 2.2 Prediction-Level Ensembling RandOpt predicts by retaining the top- K entries of a rewarded population and voting over the corresponding perturbed models, E_{K}=\operatorname{TopK}\!\big(\{r_{i}\}_{i=1}^{N}\big),\quad\hat{y}(x^{\prime})=\operatorname{MajVote}\!\big(\{f(\theta_{0}+\Delta_{i};\,x^{\prime}):i\in E_{K}\}\big), (3) where f(\theta;\,x) denotes the model output at input x under weights \theta . The ensemble works

Read the complete text