Hista and Numca: Estimate State Value Effectively for Large Language Model Reinforcement Learning
Zizhe Chen, Jiqian Dong, Yizhou Tian, Garry Yang, Yongqiang Chen, Zhitang Chen, James Cheng · 2026
Read Hista and Numca: Estimate State Value Effectively for Large Language Model Reinforcement Learning by Zizhe Chen, Jiqian Dong, Yizhou Tian, Garry Yang, Yongqiang Chen, Zhitang Chen, James Cheng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29782.txt.
Introduction
Zizhe Chen and six colleagues improve state value estimation for language-model reinforcement learning. Accurate state values steady classical training, yet critics in standard methods like PPO collapse to group averages in post-training. The team builds a benchmark, SVEB, to measure state estimation inside current frameworks, exposing that collapse. Two fixes follow: Numca, which uses numerical spans as gradable milestones, and Hista, which weights disjoint rollouts and returns by hidden-state representations. Both yield sharper value estimates and stronger training across algorithms and model sizes at little compute.
Catalog record
| CREATOR | Zizhe Chen, Jiqian Dong, Yizhou Tian, Garry Yang, Yongqiang Chen, Zhitang Chen, James Cheng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 113788 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.29782
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Hista and Numca: Estimate State Value Effectively for Large Language Model Reinforcement Learning Zizhe Chen, Jiqian Dong, Yizhou Tian, Garry Yang, Yongqiang Chen, Zhitang Chen, James Cheng Abstract Reinforcement learning (RL) refines large language models (LLMs) by directly optimizing model behavior through reward signals. While accurate state value estimation is critical for stable training in classical RL, it remains an underexplored challenge in LLM post‑training. In this work, we introduce the State Value Estimation Benchmark (SVEB) to assess state estimation within existing RL frameworks and show that critics in standard approaches like PPO collapse to a coarse group‑average baseline. To address this, we propose two techniques: Numca , which leverages numerical spans as gradable milestones for state value estimation, and Hista , a framework that uses LLM’s hidden states as representation to weighted average disjoint rollouts and their return. Extensive experiments demonstrate that both methods yield more accurate state value estimates and enhance training performance across different RL algorithms and model sizes without incurring significant computational overhead. Code available at https://github.com/VOXXXX1874/Hista . 1 Introduction The effectiveness of DeepSeek-R1 established GRPO as a foundational RL post training method for LLMs ( 42 ) ( 16 ) . Recent progress has focused on creating more advanced successors. These newer algorithms, such as DAPO ( 56 ) , GSPO ( 62 ) , and CSIPO ( 33 ) , surpass GRPO by optimizing components like KL-regularization, importance sampling calculation, and clip mechanism. Figure 1 : a. Credit assignment of Hista . b. Apply Hista to DAPO on Qwen2.5 base models, where each color represents one settings. Applying Hista achieves better convergence speed and reward. Despite these improvements, all these successors retain a core limitation inherited from GRPO: treating entire response as single action and apply the same state value to every token. This simplification stems directly from the difficulty of parsing intermediate states. More sophisticated approach, such as training process‑level reward models ( 27 ; 61 ) or applying Monte Carlo tree search ( 26 ; 15 ) attempt to mitigate this problem, but they inevitably incur prohibitive costs in labeled data or computational rollouts, rendering them impractical for large‑scale training. Given this reliance on coarse group-averaged reward baseline, current RL training methods for LLMs deviate strikingly from classical RL, whose central theme is the use of a learned state value function V^{\pi}(s) to estimate the expected return from any given state ( 22 ; 41 ) . The critic network provides this fine-grained valuation, enabling stable and guided policy updates by the actor. This contrast raises a question: How does accurate state value improve LLM RL training, and how can it be estimated effectively? To answer this question, we first construct a State Value Estimation Benchmark (SVEB) to quantify the discrepancy between ground‑truth state values and those estimated by various methods. Using this benchmark, we uncover a key limitation of standard PPO in LLM RL: although PPO includes a critic network that ostensibly provides fine‑grained token‑level credit assignment, the state values it produces consistently collapse toward the group‑averaged reward . In effect, this degrades the method to a GRPO‑like baseline, offering no more informative guidance than a coarse group average. Empirically, we conduct extensive experiments across different data sources using PPO and find that its value estimates inevitably regress to this average. To begin mitigating this issue, reframing mathematical problem solving as a sparse‑reward goal‑reaching problem. Inspired by Hindsight Experience Replay (HER) ( 2 ) , which relabels past experiences with alternative goals, we seek to identify meaningful ”milestones” within a text sequence. In mathematical reasoning (e.g., the tasks in AIME), we observe that numeric values naturally serve as easily parsable milestones. Based on this insight, we introduce Numca , a heuristic method that utlizes numbers as representation to group disjoint rollouts. Numca outperforms group‑average reward both on our SVEB benchmark and in downstream RL training. While Numca demonstrates the value of accurate state value, its reliance on domain‑specific heuristics limits generality. To address this, we further propose Hista , a general framework that represents each generation step using its hidden state embeddings. Hista estimates state values via probability‑weighted reward averaging, requiring no additional rollouts or training. This approach operationalizes the intuition that semantically similar states should receive similar state values. In Theorem 5.5 , we prove that Hista yields more accurate estimates than the group‑average baseline. Empirically, Hista outperforms baselines across all domains in SVEB, and RL‑trained models equipped with Hista surpass strong alternatives such as DAPO and CSIPO. In summary, this work makes three primary contributions: SVEB : We propose the SVEB for evaluating state value estimation methods. Using SVEB, we identify fundamental limitations of standard PPO in the LLM setting. Numca: We introduce Numca , a simple and highly efficient heuristic that provides effective state value estimation for mathematical reasoning. Hista: To build a general solution, we propose the Hista framework based on hidden states, with theoretical guarantee that Hista delivers better value estimates than GRPO. 2 Related Work Classic RL: To achieve stable policy optimization, actor-critic methods combine value-based ( 34 ) and policy-based ( 47 ) reinforcement learning by utilizing an actor network for action generation and a critic network for value estimation, as exemplified by PPO ( 41 ) . Following PPO, extensive efforts have been devoted to improving critic accuracy and stability. TD3 mitigates critic approximation errors through clipped double-Q learning ( 13 ) , while subsequent works further explore conservative, uncertainty-aware, and entropy-regularized value estimation ( 17 ; 23 ; 24 ) . Another important direction is Goal-Conditioned RL (GCRL) ( 39 ) , which studies credit assignment under sparse rewards. HER addresses sparse goal-reaching tasks through hindsight goal relabeling ( 2 ) , while more recent works focus on representation learning and reachability modeling for credit assignment in a more general settings ( 11 ; 53 ) . Interestingly, this perspective is closely related to option discovery in Hierarchical Reinforcement Learning ( 48 ) , where successor representations and latent state geometry are leveraged to discover reusable temporally-extended subgoals ( 36 ; 31 ) . Credit Assignment in LLM RL: Finer-grained credit assignment is also a central problem in reinforcement learning for large language models. PPO-style methods such as VAPO and VC-PPO ( 59 ; 58 ) introduce value pretraining techniques to alleviate critic initialization mismatch, but they still require training a critic with a scale comparable to the actor. Another line of work leverages Monte Carlo sampling or tree search to obtain more accurate state-value or action-value estimation ( 26 ; 15 ; 1 ) . Although these methods avoid maintaining a dedicated critic network, their computational cost grows rapidly with response length and sampling budget. Process Reward Models (PRMs) ( 61 ; 57 ; 27 ) provide dense supervision over intermediate reasoning steps, but they mainly serve as verifiers of reasoning correctness rather than explicit estimators of state value for policy optimization. Several lightweight alternatives have also been proposed to improve credit assignment ( 46 ; 52 ) , though they are generally not derived from the perspective of state-value estimation. In contrast, our method directly exploits hidden-state representations to achieve computational efficiency comparable to GRPO while providing provably better state-value