A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematics
Ting-Ruen Wei, Haowei Liu, Xuyang Wu, Yi Fang · 2025
Read A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematics by Ting-Ruen Wei, Haowei Liu, Xuyang Wu, Yi Fang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-14333.txt.
Introduction
Ting-Ruen Wei and co-authors survey feedback methods for math reasoning in large language models in a 2025 paper. Chain-of-thought prompting taught models to solve problems step by step, and later work baked that process into training using rewards as feedback. The survey gathers these fast-growing approaches into one map, focused on mathematics for its clear logic.
The authors sort methods by feedback type. Some use rewards for each step, which need costly step labels, while others reward only the final outcome. A third group needs no training at all, calling on frozen models or outside tools for feedback during solving. The survey aims to give newcomers a firm base and to point researchers toward open gaps.
Catalog record
| CREATOR | Ting-Ruen Wei, Haowei Liu, Xuyang Wu, Yi Fang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-14333
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematics Ting-Ruen Wei, Haowei Liu, Xuyang Wu, Yi Fang Abstract Recent progress in large language models (LLM) found chain-of-thought prompting strategies to improve the reasoning ability of LLMs by encouraging problem solving through multiple steps. Therefore, subsequent research aimed to integrate the multi-step reasoning process into the LLM itself through process rewards as feedback and achieved improvements over prompting strategies. Due to the cost of step-level annotation, some turn to outcome rewards as feedback. Aside from these training-based approaches, training-free techniques leverage frozen LLMs or external tools for feedback at each step to enhance the reasoning process. With the abundance of work in mathematics due to its logical nature, we present a survey of strategies utilizing feedback at the step and outcome levels to enhance multi-step math reasoning for LLMs. As multi-step reasoning emerges a crucial component in scaling LLMs, we hope to establish its foundation for easier understanding and empower further research. 1 Introduction Large language models (LLMs) have made significant progress and demonstrated improvements on many benchmarks ( Hendrycks et al., 2021 ) . Traditional methods to improve the model performance include increasing the size of the dataset and enhancing its quality to approximate the population distribution. However, humans usually do not require as many practice problems as a training set to be able to solve the same type of problems. Based on this motivation, researchers have turned to enhancing LLMs through multi-step reasoning. Instead of learning from a vast amount of training data through optimizing a training objective, reasoning decomposes a complex problem into multiple simpler problems and makes decisions based on the context and their patterns. On the other hand, chain-of-thought (CoT) Figure 1: Our survey coverage on feedback-based multi-step reasoning. We focus on strategies that utilize feedback for enhancing the LLM performance across multiple steps on math problems. ( Wei et al., 2022 ) as a pioneering work, has found great success through prompting LLMs to output step-by-step responses, establishing the paradigm of multi-step reasoning ( Chu et al., 2024 ) and opening new avenues for training LLMs to produce such structured outputs, which showed better performance than prompting alone ( Uesato et al., 2022 ) . This is empowered by step-level feedback (process rewards) and outcome-level feedback (outcome rewards) which can be seen as a special case of step-level feedback with no intermediate rewards. Step-level feedback encourages correct reasoning steps and sets the solution on the right track, and such feedback can be provided by training-based models, training-free LLM prompting, or external tools, as shown in Figure 1 . Due to the logical nature of mathematics, lots of research apply reasoning on LLMs to solve math problems, creating the need for a survey. The specific focus on multi-step reasoning is due to the trend of solving problems by guiding LLMs through multiple steps with feedback, which is different from traditional methods (CoT and supervised fine-tuning (SFT)) where there is no step-level feedback. In this paper, we present a survey on not only fine-tuning approaches but also training-free strategies that guide LLMs through multiple steps to solve math problems. Previous survey papers for LLMs in reasoning tend to stay on the general domain. While Huang and Chang (2023) discussed an overview of LLM reasoning and Qiao et al. (2023) focused on prompting strategies for general reasoning, Guan et al. (2024) proposed verifier engineering. On math, Liu et al. (2023b) addressed math language models and Ahn et al. (2024) discussed the datasets and general methods such as prompting frozen LLMs and fine-tuning. Close to our work, Plaat et al. (2024) covered multi-step math reasoning but with a focus on prompt-based strategies, overlooking a mainstream of fine-tuning strategies utilizing feedback that can lead to better performance ( Uesato et al., 2022 ) . In this survey, we present training-free and training-based strategies utilizing feedback to guide LLMs in multi-step reasoning on math. Figure 2: Illustrative example. Outcome reward evaluates the outcome (GSM8K dataset ( Cobbe et al., 2021 ) ), while process reward evaluates each reasoning step (Math-Shepherd dataset ( Wang et al., 2024b ) ). 2 Background 2.1 Problem Statement Given a question Q , an LLM generates the m^{th} solution consisting of many reasoning steps denoted as r^{1}_{m},\dots,r^{n}_{m} and a final answer \hat{a}_{m} , resulting in a set of solution paths \{r^{1}_{m},\dots,r^{n}_{m},\hat{a}_{m}\}_{m=1}^{M} where n and M are the total number of steps and solutions. Each final answer \hat{a}_{m} has a correctness label l_{m} , and the final prediction a is selected from the set of final answer(s) \{\hat{a}_{m}\}_{m=1}^{M} and compared against the correct final answer A for correctness which is the objective to maximize. 2.2 Preliminaries To integrate multi-step reasoning into LLMs, Uesato et al. (2022) conducted SFT on a pre-trained LLM by treating the question as input tokens and the multi-step solution as the target tokens. They annotated each step whether the m^{th} solution has been correct so far, resulting in step-label pairs \{(r^{i}_{m},s^{i}_{m})\}_{i=1}^{n} , where s is a binary label. Process reward models (PRMs) utilize this additional supervision for each reasoning step and the training combines with the language model objective. In Cobbe et al. (2021) , a scalar head implemented as a bias parameter and a gain parameter operates on the logit of the special token for binary classification. To save annotation cost, all reasoning steps can instead be labeled as whether the final answer is correct, resulting in outcome reward models (ORMs). Figure 2 illustrates the difference between process and outcome rewards with an example. Though Uesato et al. (2022) found comparable performance between PRMs and ORMs, Lightman et al. (2023) conducted a similar experiment and found PRMs to outperform ORMs by a large margin, attributing the success to the larger size of the dataset. {forest} Figure 3: Taxonomy of multi-step math reasoning through feedback. Training-based approaches leverage step- or outcome-level feedback. Step-level feedback can be aggregated for voting or best-of-N, or assist search or refinement. Outcome-level feedback can train discriminative or generative ORMs, or be determined by rules. Training-free techniques evaluate each step by direct LLM responses, LLM token logits, or an external tool to obtain step-level feedback. While the LLM generation process typically follows greedy decoding to output tokens with the highest probability, Self-consistency ( Wang et al., 2022 ) found success by sampling 40 solutions \{\hat{a}_{m}\}_{m=1}^{40} and selecting the most consistent answer as the final answer. Consistency can be measured through majority voting or weighted voting. Majority voting selects the most common answer: a=\arg\max_{a}\sum_{i=1}^{m}\mathbf{1}(\hat{a}_{i}=a) (1) while weighted voting selects the final answer with the largest sum of weights: a=\arg\max_{a}\sum_{i=1}^{m}w_{i}\cdot\mathbf{1}(\hat{a}_{i}=a) (2) where w_{i} is the weight for \hat{a}_{i} . Alternatively, best-of-N ( Cobbe et al., 2021 ; Lightman et al., 2023 ) selects the solution with the highest weight: a=\hat{a}_{\arg\max_{1\leq i\leq m}w_{i}} (3) These strategies can directly follow a trained ORM by setting w_{i} as the model’s scalar output in probability. For PRMs, the model predicts a probability score for each reasoning step, and these scores are aggregated by either taking their product, their minimum, or the last value, to represent w_{i} . In addition, stepwise scores can assist beam search in finding the best solution path (