Evaluating Step-by-step Reasoning Traces: A Survey
Jinu Lee, Julia Hockenmaier · 2025
Read Evaluating Step-by-step Reasoning Traces: A Survey by Jinu Lee, Julia Hockenmaier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12289.txt.
Introduction
Jinu Lee and Julia Hockenmaier survey how researchers grade the step-by-step reasoning of language models in a 2025 paper. Showing work step by step helps models solve hard problems, and grading those steps is key to improving them. Yet grading habits vary widely, which fragments progress across the field.
The survey organizes evaluation criteria into four groups: factuality, validity, coherence, and utility. It then reviews the data sets used, the automated graders built, and what recent studies have found. Gaps and clashes in current practice are set out plainly. The authors close with directions they see as promising, aiming to give the field a shared map for building better graders.
Catalog record
| CREATOR | Jinu Lee, Julia Hockenmaier |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-12289
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating Step-by-step Reasoning Traces: A Survey Jinu Lee, Julia Hockenmaier Abstract Step-by-step reasoning is widely used to enhance the reasoning ability of large language models (LLMs) in complex problems. Evaluating the quality of reasoning traces is crucial for understanding and improving LLM reasoning. However, existing evaluation practices are highly inconsistent, resulting in fragmented progress across evaluator design and benchmark development. To address this gap, this survey provides a comprehensive overview of step-by-step reasoning evaluation, proposing a taxonomy of evaluation criteria with four top-level categories (factuality, validity, coherence, and utility). Based on the taxonomy, we review different evaluator implementations and recent findings, leading to promising directions for future research. 1 Introduction Large language models (LLMs) have demonstrated remarkable capabilities in reasoning on complex problems, such as logic, math, and science. At the core of this versatility lies step-by-step reasoning ( Wei et al. 2022 ; Kojima et al. 2022 ) , where the LLM generates an intermediate reasoning trace before presenting the final answer. Figure 1: Illustrative example of reasoning trace evaluation. The reasoning ability of LLMs is often measured in terms of answer accuracy , i.e. , finding the correct answer for a complex reasoning problem ( Cobbe et al. 2021 ; Zhong et al. 2021 ) . However, answer accuracy is generally insufficient for measuring LLMs’ reasoning ability, as the correct answer does not imply the correctness of the preceding reasoning trace ( Lanham et al. 2023 ; Mirzadeh et al. 2024 ; Paul et al. 2024 ) . Furthermore, assessing the quality of the reasoning trace can directly lead to better reasoning ability of LLMs by verifier-guided search ( Wang et al. 2023b ; Yao et al. 2023 ; Hao et al. 2024 ) and reinforcement learning ( Lu et al. 2024 ; Cui et al. 2025 ) . Consequently, reasoning trace evaluation is an active research topic, with numerous new evaluators and datasets continuously being proposed. However, this rapid growth has led to a proliferation of evaluators and datasets without establishing a consensus on the criteria ( what to evaluate ). In this survey, we aim to provide a systematic review of existing step-by-step reasoning evaluation criteria, which will serve as a foundation for implementing evaluators. Implementing an evaluator also introduces several practical decision choices. Different architectures involve trade-offs between computational cost and expected performance, while non-architectural factors like training data and data format also play a significant role. This survey seeks to categorize and compare various evaluator implementations, highlighting key trade-offs and revealing additional dimensions that merit consideration. The key contributions of this survey are: • Defining a clear, universal taxonomy of step-by-step evaluation criteria (§ 3 ). • Surveying existing evaluators for step-by-step reasoning evaluation based on their implementations, across diverse reasoning tasks and criteria (§ 4 ). • Identifying recent findings and promising directions for reasoning trace evaluation (§ 5 -§ 6 ). 2 Background 2.1 Step-by-step reasoning Step-by-step reasoning is where LLMs generate a series of intermediate natural language steps ("thoughts") before outputting the final answer ( Wei et al. 2022 ) . Each instance consists of two parts: a query and a reasoning trace , and the final answer . Upon seeing the query (user input), the LLM autoregressively generates its solution as a reasoning trace. Finally, a trace should include a final answer for the query, which can be compared to the ground truth. See Appendix A for details on different reasoning tasks. 2.2 Evaluation Reasoning trace evaluators assess the quality of the reasoning trace and assign a score, reflecting whether it is good or not based on the criterion. Evaluators can be intrinsic metrics like uncertainty to models specialized for reasoning trace evaluation; see Section 4 for different types of evaluators. 2.3 Meta-evaluation How can we evaluate these evaluators (meta-evaluation) ? Two common directions apply: (1) using meta-evaluation benchmarks with step-wise labels, or (2) measuring the improvement in the downstream task performance (Figure 2 ). 2.3.1 Meta-evaluation Benchmarks Meta-evaluation benchmarks contains labels indicating a step’s quality based on the predefined criteria. In this setting, the evaluator’s performance is measured by the classification accuracy of these labels. These benchmarks offer a fine-grained view of which criteria the evaluator can handle well and which cannot ( Song et al. 2025 ) . However, constructing these data often requires costly human annotation ( Lightman et al. 2024 ; Zheng et al. 2024a ) and the gains in meta-evaluation benchmark might not generalize to downstream performance ( Zhang et al. 2025 ) . Further details can be found in Appendix B . Figure 2: Illustration of three popular meta-evaluation methods: meta-evaluation benchmarks, verifier-guided search, and reinforcement learning. 2.3.2 Downstream performance improvement As the fundamental goal of evaluators is to improve the reasoning ability of LLMs, the evaluator’s quality can also be measured by the improvement in downstream reasoning tasks. Verifier-guided search uses evaluator scores to find the most promising trace after exploring different paths. Popular methods include Best-of-N decoding (independently sampling N traces and selecting one) ( Lightman et al. 2024 ; Zhang et al. 2024c ) and tree search (sampling multiple candidate steps and choosing the most promising path) ( Yao et al. 2023 ; Guan et al. 2024 ; Zhu et al. 2024b ) . The performance is often compared to majority voting without evaluators (Self-consistency; Wang et al. 2023b ), where a bigger gap indicates a better evaluator performance. Reinforcement learning (RL) uses evaluator scores as a reward to further train an LLM ( Uesato et al. 2022 ; Pan et al. 2023b ; Zhang et al. 2024a ) . If the evaluator provides useful training rewards, the trained model will reach higher final answer accuracy. Moreover, as evaluators that are vulnerable to spurious features like length lead to reward hacking , successful RL also indicates the evaluator’s robustness ( Zhang et al. 2024a ) . Figure 3: Illustration of the proposed categories of step-by-step reasoning evaluation criteria, i.e. factuality, validity, coherence, and utility. The left shows an example of a query and a reasoning trace. The other four blocks demonstrate examples that fail to satisfy the respective metric. Red filled rectangles indicate the error’s location, and the outlined boxes and arrows show the cause of the error. The trace example is originally from Lightman et al. 2024 . 3 Evaluation Criteria Previous studies have proposed various criteria for evaluating step-by-step reasoning ( Golovneva et al. 2023a ; Lightman et al. 2024 ; Wang et al. 2024c ; Jacovi et al. 2024 ) , but these works failed to propose a complete taxonomy that covers diverse reasoning tasks ( e.g. , literature in factual reasoning and math reasoning have focused on different criteria). In this section, we propose a unified taxonomy of reasoning trace evaluation criteria that spans different reasoning tasks and evaluators. We categorize them into four key dimensions: Factuality , Validity , Coherence , and Utility (Figure 3 ) 1 1 1 These criteria are independent but not mutually exclusive (a step can fail to satisfy multiple criteria). . 3.1 Factuality Factuality evaluates if the factual information can be grounded in reliable sources. The narrower notion of factuality is groundedness , where the generated trace should be true according