Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains

Soumya Rani Samineni, Durgesh Kalwar, Vardaan Gangal, Siddhant Bhambri, Subbarao Kambhampati · 2025

Read Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains by Soumya Rani Samineni, Durgesh Kalwar, Vardaan Gangal, Siddhant Bhambri, Subbarao Kambhampati (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18176.txt.

Introduction

Soumya Rani Samineni and four co-authors inspect the middle steps of math reasoning after verifiable-reward training, not just final answers. Such training is praised for better reasoning, yet most checks still score only the last answer or pass rates. The team asks whether the trace itself stays locally coherent and globally valid.

Their setup trains a small model on grade-school math with a group-based reinforcement method. A logic-based measure flags step errors and separates trace validity from trace coherence. The study gives the field a direct way to test claims that reward training improves thinking, rather than merely improving answers.

Catalog record

CREATORSoumya Rani Samineni, Durgesh Kalwar, Vardaan Gangal, Siddhant Bhambri, Subbarao Kambhampati
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-18176

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains Soumya Rani Samineni, Durgesh Kalwar, Vardaan Gangal, Siddhant Bhambri, Subbarao Kambhampati Abstract Reinforcement Learning with Verifiable Rewards (RLVR)-based post-training of Large Language Models (LLMs) has been shown to improve accuracy on reasoning tasks and continues to attract significant attention. Existing RLVR methods, however, typically treat all tokens uniformly without accounting for token-level advantages. These methods primarily evaluate performance based on final answer correctness or Pass@K accuracy, and yet make claims about RL post-training leading to improved reasoning traces. This motivates our investigation into the effect of RL post-training on intermediate tokens which are not directly incentivized. To study this, we design an experimental setup using the GRPO algorithm with Qwen-2.5-0.5B model on the GSM8K dataset. We introduce trace coherence, a First-Order Logic (FOL)-based measure to capture the consistency of reasoning steps by identifying errors in the traces. We distinguish between trace validity and trace coherence, noting that the former implies logical soundness while the latter measures local coherence via lack of errors. Our results show that RL post-training overall improves trace coherence with the most significant gains on problems where the base model fails but the RL model succeeds. Surprisingly, RL enhances local coherence without necessarily producing valid or correct solutions. This highlights a crucial distinction: improved local coherence in reasoning steps does not guarantee final answer correctness. We argue that claims of improved reasoning via RL must be examined with care, as these may be based on improved trace coherence, which may not translate into fully valid mathematical proofs. 1 Introduction Following the release of Deepseek R1 guo2025deepseek , post-training Large Language Models (LLMs) using Reinforcement Learning with Verifiable Rewards (RLVR) has gained widespread attention. Since then several works have expanded on reinforcement learning based post-training by altering the loss function, modifying advantage estimation, and utilizing base model resets ( liu2025understanding ; zhang2025grpo ; yu2025dapo ; liu2025prorl ) . However, recent analysis by samineni2025rl highlights structural limitations of current RLVR approaches, particularly due to the uniform distribution of advantages across all tokens. Further, yue2025does argued that the accuracy of RLVR models cannot surpass that of the base model demonstrating empirically that Pass@K accuracy drops relative to the base model as K increases. Also, cui2025entropy shows that the performance gains can be predicted using entropy of the base model. However, these works primarily focus on the limitations of RLVR in terms of final answer accuracy and do not examine its effect on intermediate tokens, or reasoning traces . Since RLVR verifies only the final answer and distributes rewards uniformly across all tokens, its impact on the reasoning process at the token level lacks investigation. While there have been frequent claims that RLVR improves reasoning, the effect of integrating verifier signals during training on the structure and quality of reasoning traces has not been formally studied. Since formal verification of intermediate reasoning steps is not tractable at scale, we cannot directly evaluate trace validity. Instead, we introduce a proxy metric called trace coherence , which reflects the consistency of reasoning steps. It is important to note here that while trace validity implies coherence, the opposite may not be true., Particularly, we measure trace coherence by analyzing the presence of errors (or lack thereof) in the reasoning steps, where errors are defined using a First-Order Logic (FOL) framework (§ 3 ). To experimentally analyze this problem, we design an experimental setup (§ 4 ) to study the effect of RLVR on traces using a mathematical reasoning benchmark, particularly GSM8K cobbe2021training . Our results (§ 5 ) show that RLVR surprisingly improves trace coherence across Pass@K evaluations, especially on problems where the base model fails but the RL-trained model produces a correct final answer. These findings highlight a key distinction that RL post-training can improve local coherence in reasoning traces, captured through error patterns while not guaranteeing the correctness or full trace validity. This distinction is important for interpreting the effects of RLVR on reasoning quality beyond final answer accuracy. 2 Related Work Reasoning traces have been studied since the release of DeepSeek R1 guo2025deepseek to better interpret LLMs and their improved task performance. Thoughtology marjanovic2025deepseek provides a systematic analysis of the length, structure, and content of traces generated by DeepSeek R1, focusing on interpretability and safety. However, bhambri2025cognitively show that optimizing traces for user interpretability can reduce model performance, revealing a trade-off between interpretability and LLM’s task performance. In addition, Stechly2025BeyondST ; bhambri2025interpretable demonstrate a lack of correlation between trace validity and final answer correctness during Supervised Fine-Tuning (SFT) on maze problems where it is feasible to check trace validity because of formal verifiers. However, trace validity or trace coherence in the context of RLVR post-training, especially for mathematical reasoning tasks, remains underexplored due to challenges in trace analysis. In this work, we systematically address these challenges by defining FOL-based error tags and LLM-as-a-judge gu2024survey to evaluate trace coherence. In early evaluations of LLM performance using Chain-of-Thought (CoT) traces, various taxonomies of errors specific to mathematical reasoning were introduced lewkowycz2022solving ; wei2022chain ; li2024evaluating , which are summarized in Table 1 . These categories have been widely used to identify reasoning mistakes and systematically evaluate LLM performance. Table 1: Comparison of error categorizations for intermediate trace analysis across prior works. Minerva CoT ( lewkowycz2022solving ) Chain-of-Thought ( wei2022chain ) Examiner ( li2024evaluating ) Incorrect Data, Format Error, Incorrect Calculation, Misunderstands Question, Incorrect Reasoning, Solution Too Short, Hallucinated, Repeats Question, Other Mistakes Calculator Error, Symbol Mapping Error, One Step Missing, Semantic Understanding Error, Incoherent Chain-of-Thought Calculation Error, Counting Error, Context Value Error, Hallucination, Unit Conversion Error, Operator Error, Formula Confusion Error, Missing Step, Contradictory Step While these error categories have been effective for evaluation and improve the reasoning performance of LLMs through error identification, they often overlap and are neither mutually exclusive nor exhaustive, limiting their use for formally defining trace coherence. To address this, we propose a new set of error categories, motivated by First-Order Logic and designed to be mutually exclusive to study trace coherence (see Table 2 ). 3 Methodology 3.1 Error Categories for Mathematical Reasoning Table 2: Error categories and their subtypes. Error Category Description / Subtypes False Premise Conceptual Misunderstanding; Semantic Error; False Assumption; Units Misinterpretation; Incorrect Derivation Step from problem description; False Rule Type / Operand Mismatch; Inference Violation; Operation Misapplication; Quantifier Misuse; Missing Necessary Steps Calculator Error Simple numeric or arithmetic mistakes Format Error Final answer not formatted as required, e.g., missing \boxed Mathematical reasoning problems, particularly popular benchmarks like GSM8K, typically require between two and eight steps to solve. These problems primarily involve performing a sequence of elementary calculations using basic arithmetic operations ( + , - , \times , \div ) to arrive at the final answer. Each problem can be formalized as comprising: assumptions or facts, an expression or formula to compute the answer, and intermediate steps to produce quantities essential for deriving the final answer. This structure naturally aligns with the construction of FOL representations. Based on the error categorization, we define a set of error types with clear distinctions between them shown in 2 . Figure 1: Evaluation accuracy 3.2 Trace Coherence and Pass@K Evaluation Using LLMs To assess trace

Read the complete text