The Good, The Bad, and The Hybrid: A Reward Structure Showdown in Reasoning Models Training
Subramanyam Sahoo · 2025
Read The Good, The Bad, and The Hybrid: A Reward Structure Showdown in Reasoning Models Training by Subramanyam Sahoo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-13016.txt.
Introduction
Subramanyam Sahoo compares ways of scoring a language model's homework during training. Reinforcement learning from human feedback steers models with rewards, and the reward can be hard, a simple right or wrong, continuous, a graded score, or hybrid. Sahoo builds a unified framework for the three styles and tests them fine-tuning Qwen3-4B with LoRA on GSM8K math problems, with reward signals combining correctness, perplexity, reasoning quality, and consistency.
His adaptive hybrid scheduler shifts between discrete and continuous signals during training, balancing exploration against stability. The hybrid structures converged faster and trained more stably than purely hard or purely continuous rewards. Reward design is often treated as an afterthought in alignment work, tuned by feel. A controlled comparison on a standard reasoning benchmark gives practitioners evidence for a concrete choice: blend the signals, and schedule the blend, rather than committing to one kind of feedback.
Catalog record
| CREATOR | Subramanyam Sahoo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-13016
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Good, The Bad, and The Hybrid: A Reward Structure Showdown in Reasoning Models Training Subramanyam Sahoo Abstract Reward design is central to reinforcement learning from human feedback (RLHF) and alignment research. In this work, we propose a unified framework to study hard , continuous , and hybrid reward structures for fine-tuning large language models (LLMs) on mathematical reasoning tasks. Using Qwen3-4B with LoRA fine-tuning on the GSM8K dataset, we formalize and empirically evaluate reward formulations that incorporate correctness, perplexity, reasoning quality, and consistency. We introduce an adaptive hybrid reward scheduler that transitions between discrete and continuous signals, balancing exploration and stability. Our results show that hybrid reward structures improve convergence speed and training stability over purely hard or continuous approaches, offering insights for alignment via adaptive reward modeling. Keywords: RLHF,Reasoning Models,Reward Modelling, Scalable Oversight 1 Introduction Recent advances in large language models have enabled impressive chain-of-thought (CoT) reasoning capabilities, where models generate explicit reasoning steps to solve complex tasks. For example, prompting a 540B model with CoT exemplars achieves state-of-the-art accuracy on GSM8K, a benchmark of grade-school math word problems. However, aligning LLMs to consistently produce correct reasoning remains challenging as these models often cannot reliably self-evaluate or refine their reasoning without external guidance. Reinforcement learning from human (or automated) feedback (RLHF) 6 is a common approach to align LLM outputs to desired behaviors. In the RLHF pipeline, a pretrained LM is fine-tuned with an RL algorithm (e.g. PPO) using a scalar reward from a reward model (or other metric). In practice, specifying an appropriate reward signal for open-ended reasoning tasks is nontrivial. Simple scalar rewards (e.g. exact match accuracy) can be very sparse, whereas dense rewards (e.g. likelihood or intermediate metrics) may not align perfectly with the final goal. Moreover, reward models trained on preference data can generalize poorly out-of-distribution, especially on reasoning tasks. To address this, we explore adaptive reward strategies that combine multiple reward components and dynamically schedule their influence during training. Our key contributions are: Reward design : We propose three reward formulations: (1) a hard binary reward (correct/incorrect + format bonus), (2) a continuous multi-component reward combining correctness, language model likelihood, reasoning quality, and consistency, and (3) hybrid combinations that transition between these. We provide detailed mathematical definitions for each (see Sec. 3). Hybrid scheduler : We formalize a schedule for mixing hard vs continuous rewards. In the continuous-to-hard schedule, the training starts with full weight on the continuous reward and linearly shifts to the hard reward over a fixed number of steps; the hard-to-continuous schedule does the opposite. We give precise formulas for the time-dependent weights (Sec. 3.3). Evaluation on math reasoning : We implement these reward schemes in a LoRA-fine-tuning framework 4 using Qwen3-4B 10 and evaluate on GSM8K. We compare final accuracies and training dynamics. Notably, the hard reward achieved the best accuracy (40%) and convergence, while the continuous reward lagged (28%) despite richer signals. Hybrids performed moderately (33%). Analysis and insights : We analyze training stability and the contributions of each reward component, and discuss why continuous/hybrid schemes may underperform with naive weighting. We relate this to known alignment challenges: LLMs struggle to evaluate their own reasoning, and naive reward components (e.g. perplexity) may not capture true correctness. We also highlight how multi-objective reward models (e.g. multi-exemplar or consistency-based) could be integrated in our framework. Our work emphasizes the importance of reward modeling in RL fine-tuning of LLMs, especially for open-ended tasks like math reasoning. We show how to systematically construct, formalize, and compare different reward structures and schedules. The framework is designed to be extensible (e.g. supporting new rewards or domains), and all training loop pseudocode, logging, and metrics are detailed for reproducibility. 2 Related Work RLHF and alignment: Reinforcement learning has become a cornerstone for aligning LMs to human preferences, as popularized by ChatGPT and Claude. In typical RLHF, a reward model is trained to score outputs, and PPO is used to optimize the policy (LM) to maximize expected reward. Reward models themselves are often opaque and brittle: they may have only 60–70% accuracy even on in-distribution tasks, and can fail on out-of-distribution examples. These issues are exacerbated for reasoning tasks, where ground-truth supervision is hard to obtain (e.g. verifying each reasoning step is laborious). Recent work (RewardBench ) 5 has begun to systematically evaluate RM generalization, finding shortcomings in reasoning and instruction-following. In short, RLHF is powerful but reward design remains a key challenge in aligning LMs. Math reasoning and chain-of-thought: Datasets like GSM8K (8.5K grade-school math problems) and MATH (contest problems) are common benchmarks for evaluating LLM reasoning. Chain-of-thought prompting, which elicits intermediate reasoning steps, dramatically improves accuracy on GSM8K and similar tasks. For example, Wei et al. (2022) 9 report that a 540B model with CoT prompts achieved state-of-the-art performance on GSM8K. Our method similarly encourages the model to produce an explicit reasoning segment (using XML tags), so that rewards can penalize incoherent or implausible chains. However, whereas CoT prompting relies on handcrafted prompts or exemplars, our approach uses RL to fine-tune the model with learned rewards. Reward shaping and multi-component rewards: Beyond binary correctness, reward shaping techniques are often used in RL to guide agents via additional terms. In LLM training, some works combine fluency and factuality with task-specific rewards. For example, one might include a KL or perplexity penalty to keep outputs fluent. Our continuous reward is in spirit a form of shaping: it rewards not only correct final answers but also low-perplexity (high-confidence) generations, explicit logical steps, and consistency between reasoning and answer. Similar ideas appear in RL for LLMs and self-training methods. The idea of enforcing consistency in reasoning models (for example, through self-consistency) has also been extended into preference-based training settings where the preferences are derived automatically, without requiring human-labeled supervision. Unlike prior work, we combine multiple handcrafted metrics into one continuous reward function, and systematically compare it to simpler reward signals 2 . 3 Methodology We fine-tune a pretrained causal LM using reinforcement learning with different reward schemes. We first describe the model and training algorithm, then detail each reward type and the hybrid scheduler. 3.1 Model and Training Setup Our policy is the Qwen3-4B model (a 4B-parameter open LLM) with LoRA adapters activated. We tokenize prompts and ensure outputs contain an XML structure: each model output includes <reasoning>...</reasoning><answer>...</answer> . The <answer> tag should contain the final numeric answer. We prepare the GSM8K dataset of math problems, converting each to a prompt. Training is done with a PPO-like optimizer. We use the Hugging Face TRL library’s GRPOTrainer (Group Relative PPO) which handles multiple sampled completions per prompt and normalizes rewards. At each training step we sample a batch of prompts. For each prompt we generate G candidate outputs under the current policy. For each completion we compute a scalar reward (as described below). We then compute advantages by centering and scaling the rewards within each group: A_{i}=\frac{r_{i}-\bar{r}}{\sigma}, (1) where \bar{r} and \sigma are the group mean and standard deviation. The policy is updated to maximize the RL objective with a KL-constraint to the reference (initial) policy. Hyperparameters (learning rate, epochs, etc.) are set as in GRPO reference implementations. The training loop is summarized as: Algorithm 1 Training loop 1: