Logical Reasoning with Outcome Reward Models for Test-Time Scaling
Ramya Keerthy Thatikonda Wray Buntine Ehsan Shareghi · 2025
Read Logical Reasoning with Outcome Reward Models for Test-Time Scaling by Ramya Keerthy Thatikonda Wray Buntine Ehsan Shareghi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-19903.txt.
Introduction
Ramya Keerthy Thatikonda and co-authors train outcome reward models to grade logical reasoning and boost test-time scaling, in work from twenty twenty-five. Checking only final answers can still teach models to reason better. Their reward models learn from chains of thought, sampled singly and in groups. A novel echo tactic harvests extra training errors deliberately. Prompts seeded with wrong assumptions lure models into fresh mistakes.
Echo data covers error types that honest chains rarely make. Reward models trained on both data kinds grade more reliably. Three logic benchmarks test the graders across four language models. Performance improves consistently with the echo-augmented training. Deliberately wrong reasoning turns out to be valuable teaching material. Deductive reasoning joins the test-time scaling success story.
Catalog record
| CREATOR | Ramya Keerthy Thatikonda Wray Buntine Ehsan Shareghi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-19903
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Logical Reasoning with Outcome Reward Models for Test-Time Scaling Ramya Keerthy Thatikonda Wray Buntine Ehsan Shareghi Abstract Logical reasoning is a critical benchmark for evaluating the capabilities of large language models (LLMs), as it reflects their ability to derive valid conclusions from given premises. While the combination of test-time scaling with dedicated outcome or process reward models has opened up new avenues to enhance LLMs performance in complex reasoning tasks, this space is under-explored in deductive logical reasoning. We present a set of Outcome Reward Models (ORMs) for deductive reasoning. To train the ORMs we mainly generate data using Chain-of-Thought (CoT) with single and multiple samples. Additionally, we propose a novel tactic to further expand the type of errors covered in the training dataset of the ORM. In particular, we propose an echo generation technique that leverages LLMs’ tendency to reflect incorrect assumptions made in prompts to extract additional training data, covering previously unexplored error types. While a standard CoT chain may contain errors likely to be made by the reasoner, the echo strategy deliberately steers the model toward incorrect reasoning. We show that ORMs trained on CoT and echo-augmented data demonstrate improved performance on the FOLIO, JustLogic, and ProverQA datasets across four different LLMs. 1 1 1 Code is available at https://github.com/RamyaKeerthy/LogicORM 1 Introduction Logical reasoning in large language models (LLMs) has primarily been studied as a symbolic task using in-context learning Lam et al. 2024 ; Pan et al. 2023 ; Ye et al. 2023 ; Olausson et al. 2023 , and fine-tuning Thatikonda et al. 2024 ; Qi et al. 2025 . Current state-of-the-art techniques in reasoning which bundle test-time scaling Brown et al. 2024 ; Snell et al. 2024 with Process or Outcome Reward Models Wang et al. 2024 ; Lightman et al. 2024 ; Uesato et al. 2022 , proven effective in math and coding, while remaining heavily underexplored for logical reasoning. This presents a significant opportunity to assess and enhance LLMs’ reasoning capabilities using reward models at test-time with text-based reasoning. In this paper we explore logical reasoning with test-time scaling, demonstrating performance gains on three datasets; FOLIO Han et al. 2024 , ProverQA Qi et al. 2025 , and JustLogic Chen et al. 2025 when combined with verification via Outcome Reward Models (ORMs). Outcome Reward Models (ORMs) enable verification of the entire reasoning sequence by assigning a confidence score to the final output. A central challenge in training ORMs is acquiring high-quality, diverse training data. To address this, we generate multiple Chain-of-Thought Wei et al. 2022 reasoning candidates via sampling. Compared to an ORM trained on a single sample per reasoning question, this facilitates more effective re-ranking of solutions during inference. To further enrich the training data, we leverage the echoing behavior of LLMs - where models tend to align their reasoning with user-provided answers. This phenomenon can introduce hallucinated reasoning, which, when subsequently flagged as incorrect through a second-level filtering process, results in a diverse set of flawed reasoning paths. We show that incorporating these echoed errors into the training set helps the ORM learn to better distinguish between valid and invalid reasoning trajectories during use with test-time scaling. Figure 1: Methodology for generating training data for CoT and EcCoT ORMs. A. The LLM is prompted to generate CoT reasoning, and both correct and incorrect answers are used to construct the ORM dataset. B. The LLM is provided with a misleading answer to elicit reasoning. The resulting incorrect reasoning trajectories are then passed back to the LLM for evaluation. If the LLM fails to recognize its error, the trajectory is added to the dataset. Our contributions are as follows: (1) We demonstrate that training ORMs with multiple sampled CoT candidates (per example) significantly improves the reliability of the ORM on deductive reasoning tasks. (2) We show that augmenting CoT training data with data generated by echoed-error further enhances the resulting ORMs’ accuracy. (3) We compare ORM models on training data size, format, and reward distribution. 2 Outcome Reward Model for Logic In outcome supervision, the reward model evaluates the entire reasoning sequence and assigns a final score reflecting the quality of the outcome. During inference, this score can be used to re-rank the candidate solutions generated by a large language model (LLM). Following Lightman et al. 2024 recipe for mathematical reasoning, to train an Outcome Reward Model (ORM), each reasoning trace is compared against a gold-standard label: a positive reward is given when the final output matches the correct answer, and a negative reward is assigned otherwise. The ORM learns to map reasoning sequences to these rewards, producing logits that can be used to rank candidate outputs during inference. We explore two types of data generation strategies for training ORMs: Outcome supervision on standard Chain-of-Thought (CoT) reasoning, and Outcome supervision on Echo Chain-of-Thought (EcCoT) - a variant that incorporates the LLM’s echoing behavior to produce additional reasoning paths. See Figure 1 for an overview. CoT ORM Data Generation. For standard CoT data generation, we prompt an LLM with the context and question to generate step-by-step reasoning, with the final answer appended at the end. The LLM generates multiple reasoning candidates for a given question. These candidates are then parsed into individual CoT traces. We used " Please reason step by step, and put your final answer within \\boxed{} " as prompt. Each candidate is labeled with a reward: positive if the final answer matches the gold label, and negative otherwise. This data is used to train a second LLM as a classifier that predicts the reward score based on the reasoning sequence. Our approach builds on the ideas presented in Wang et al. 2024 , but differs by applying outcome supervision rather than process supervision, and by using a straightforward automated annotation to label the data. EcCoT ORM Data Generation. LLMs often exhibit a tendency to follow user-provided answers uncritically, sometimes hallucinating or forcefully aligning their reasoning to fit an incorrect answer. We exploit this behavior to generate challenging negative examples. In contrast to Li et al. 2025 , we prompt the LLM with the instruction " Given the answer is True, please reason step by step, and put your final answer within \\boxed{} " for all reasoning questions. This coerces the model into producing reasoning that unjustifiably supports the provided answer (where we deliberately provided an incorrect answer, e.g. "True" where the correct answer was "False") resulting in flawed reasoning trajectories. These incorrect yet plausible reasoning sequences, referred to as echoes, are valuable for training ORMs to penalize invalid rationales. While echoing can encourage LLMs to commit to flawed reasoning paths, we further filtered the collected echoes by prompting the LLM (i.e., " Judge if the reasoning logically follows from the input; respond only with Correct or Incorrect. ") itself to evaluate whether the reasoning trajectories were correct. We discarded the echoed examples that the LLM identified as incorrect, as these were deemed too obvious and unlikely to occur during inference. The remaining echoes (those involving incorrect reasoning and not easily recognized as such) were retained and added to the training data alongside the CoT examples. The statistics for CoT and EcCoT, together with sample prompt outputs, are provided in Appendix