When LLM Reward Design Fails: Diagnostic-Driven Refinement for Sparse Structured RL
Youting Wang, Yuan Tang, Bowen Liu, Xuan Liu, Dingyan Shang · 2026
Read When LLM Reward Design Fails: Diagnostic-Driven Refinement for Sparse Structured RL by Youting Wang, Yuan Tang, Bowen Liu, Xuan Liu, Dingyan Shang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28918.txt.
Introduction
Youting Wang and colleagues treat LLM reward design for sparse structured tasks as debugging, not one-shot generation. Auditing PPO agents on MiniGrid, with MuJoCo reaching and locomotion as stress tests, they find two recurring one-shot failures: reward flooding and semantic or API misunderstanding. Their diagnostic-driven iterative refinement, guided by training diagnostics and a failure-mode taxonomy, lifts DoorKey-8x8 success sharply. It moves from about 2 percent without shaping to about 98 percent, and KeyCorridor from about 31 percent to about 87 percent.
Catalog record
| CREATOR | Youting Wang, Yuan Tang, Bowen Liu, Xuan Liu, Dingyan Shang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 98003 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.28918
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When LLM Reward Design Fails: Diagnostic-Driven Refinement for Sparse Structured RL Youting Wang, Yuan Tang, Bowen Liu, Xuan Liu, Dingyan Shang Abstract For sparse, structured reinforcement-learning tasks with semantic reward-function interfaces, LLM-generated reward shaping is better framed as a debugging problem than as pure one-shot generation. We study PPO-trained agents, using MiniGrid as the core sparse-structured evaluation and MuJoCo reaching/locomotion as boundary stress tests. Our audit finds two dominant one-shot failure modes—reward flooding and semantic/API misunderstanding—plus a rarer, less reliably labeled weak-shaping case. We propose diagnostic-driven iterative refinement , where training diagnostics and a failure-mode taxonomy guide targeted reward-function revision. On sparse structured tasks with diagnosable reward failures, refinement improves DoorKey-8 × \times 8 from 2.3 % 2.3\% success without shaping to 97.6 % 97.6\% , and KeyCorridor from 31.2 % 31.2\% one-shot to 86.7 % 86.7\% with high seed-to-seed variance. Controls indicate that these gains are not attributable to generic retrying or extra training alone: metrics-only re-prompting produces large drops (DoorKey-8 × \times 8: 97.6 % → 68.6 % 97.6\%\to 68.6\% ; KeyCorridor: 86.7 % → 11.5 % 86.7\%\to 11.5\% ), while a static-vocabulary control recovers much of the gap (DoorKey-8 × \times 8: 87.6 % 87.6\% ; KeyCorridor: 70.7 % 70.7\% ), showing that the taxonomy prompt itself is a major mechanism and that dynamic trigger labels provide only partially isolated incremental evidence. Budget-matched and Best-of-3 comparisons help separate refinement from selection and training-time effects. Component-removal stress tests, sensitivity analyses, and an audit against author labels provide converging evidence for the debugging interpretation while revealing calibration limits. Continuous-control results expose the boundary: success-based diagnostics can misfire in dense-reward locomotion, and return-trend feedback removes one false-positive mechanism without producing robust locomotion gains. The low-call protocol is a protocol-cost contrast with population-based reward search, not a shared-benchmark performance comparison. In the four environments where we run a fully crossed variance design, variance point estimates are consistent with larger gains from reward correction when LLM reward-function variance dominates, but bootstrap intervals are wide and temper exact share claims. The method’s scope is deliberately bounded to sparse structured tasks with reliable structured interfaces under PPO training; richer semantic fields such as event_text can help, hurt, or be neutral depending on alignment with the task structure. 1 Introduction Reinforcement learning in environments with sparse rewards remains a fundamental challenge. An agent exploring randomly in a multi-step task—such as finding a key, unlocking a door, and reaching a goal—may never encounter the terminal reward signal within feasible training budgets. Reward shaping ( 19 ) addresses this by providing intermediate feedback, but designing effective shaping functions demands substantial domain expertise and iterative manual tuning. Recent work has explored LLMs as reward function generators: Eureka performs evolutionary optimization over reward code with GPU-parallel simulation, while Text2Reward generates dense reward code from language and compact environment representations ( 18 ; 34 ) . These systems achieve impressive results, but they provide limited insight into why LLM-generated rewards fail—treating reward generation primarily as a performance-driven search or refinement problem rather than analyzing the structure of failures. We argue that LLM reward design for sparse, structured tasks is better understood as a debugging problem than a generation problem. One-shot LLM generation fails in recurring, identifiable ways; lightweight diagnostics can often detect and repair these failures cheaply in the studied sparse structured setting; and the approach breaks predictably when diagnostics are misaligned with the task’s evaluation signal. This reframing leads to a practical method—diagnostic-driven iterative refinement—that improves over one-shot generation and non-diagnostic re-prompting using 1–4 LLM calls per reward-design run, repeated across seeds for evaluation, and CPU training, without population-based search. This is a protocol-level efficiency comparison, not a direct performance comparison with evolutionary reward-search systems on shared benchmarks. Our main contribution is a diagnostic-driven framework for identifying and repairing systematic failure modes in LLM-generated reward shaping, supported by targeted experiments that test the framework’s core assumptions. We provide three supporting findings : 1. A failure taxonomy for structured reward interfaces : prompt controls show that taxonomy-rich feedback is functionally important, but also that static taxonomy vocabulary explains a substantial part of the improvement, so automated trigger labels should not be read as the sole causal mechanism. The taxonomy itself is grounded in an audit where reward flooding and semantic/API misunderstanding are the two dominant failure patterns, while weak or negligible shaping appears as a rarer and less reliably labeled case. On the dominant-mode subset, an LLM auto-labeler agrees with author labels with 94\% precision (32/34 predicted dominant labels; 71.4% overall accuracy across all categories). 2. Variance-based evidence for when refinement helps: LLM generation variance has the clearest dominant point estimate in representative sparse tasks (DoorKey-8 \times 8: LLM std 47.5\% vs. RL std 0.3\% ; crossed LLM share 96\% with a wide [66,100]\% bootstrap interval), making diagnostic refinement most useful in this regime; residual and RL-training variance have larger point estimates in representative continuous-control decompositions, limiting the benefit of reward-function correction. 3. A domain-boundary stress test : success-based diagnostics produce false positives on dense-reward locomotion, systematically degrading the feedback loop. We provide root-cause analysis and evaluate a return-trend diagnostic adaptation, showing both where the framework breaks and how one specific failure mechanism can be removed. The method is strongest for sparse, structured tasks with reliable semantic state fields and exposed subgoal structure, and all experiments use PPO-trained policies. Richer natural-language event descriptions are not uniformly beneficial: an interface ablation shows that removing event_text is neutral on DoorKey-8 \times 8 and improves a KeyCorridor rerun. We explicitly characterize this scope: it is not yet a claim about raw-pixel reward design, arbitrary robotics tasks, or algorithm-agnostic reward-function validity, and dense-reward settings require different diagnostics (Section ). Roadmap and evidence status. Table provides the headline 10-seed MiniGrid and reaching results; Section tests the diagnostic story through lower-seed stress tests and prompt controls; Sections – report boundary, budget, variance, and selection analyses. The central positive claim is about sparse structured PPO tasks with diagnosable reward-function failures. MuJoCo locomotion, fixed-coefficient RND, model-sensitivity, and protocol-cost comparisons are included as context and boundary evidence rather than as broad superiority claims. 2 Background 2.1 Reward Shaping Potential-based reward shaping ( 19 ) provides a principled framework for adding intermediate rewards without altering the optimal policy. In practice, most reward shaping uses heuristic bonuses tied to task-specific subgoals (e.g., +0.2 for picking up a key). While effective, these heuristics require domain knowledge and manual tuning. 2.2 LLMs for Code Generation in RL Eureka ( 18 ) demonstrated that LLMs can generate reward functions for continuous-control tasks, using evolutionary search over hundreds of candidates evaluated in GPU-parallel simulation (IsaacGym). Text2Reward ( 34 ) generates dense executable reward code from language task descriptions and compact environment representations. Our work takes a complementary, diagnostic-driven approach: rather than searching broadly, we analyze why generated rewards fail and use targeted feedback to repair specific failure modes with minimal iteration. We evaluate primarily on discrete MiniGrid tasks and include continuous MuJoCo tasks as stress tests for the diagnostic assumptions. 2.3 MiniGrid Environments We use the MiniGrid framework ( 8 ) , which provides a suite of grid-world environments with partial observability. The agent receives a 7\times 7\times 3 symbolic observation encoding object type, color,