Error Reflection Prompting: Can Large Language Models Successfully Understand Errors?
Jason Li Lauren Yraola Kevin Zhu Sean O’Brien · 2025
Read Error Reflection Prompting: Can Large Language Models Successfully Understand Errors? by Jason Li Lauren Yraola Kevin Zhu Sean O’Brien (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-16729.txt.
Introduction
Jason Li, Lauren Yraola, Kevin Zhu, and Sean O'Brien test a prompting trick in a 2025 paper on language model reasoning. Step-by-step prompting helps models solve problems, but it cannot spot or fix its own slips, so early errors carry through. Their method adds a reflection stage built around a worked wrong answer.
The model is shown a wrong answer, asked to name the error and its type, then given the right answer. It can also build these examples itself, which keeps the method scalable. Tests show the approach strengthens plain step-by-step prompting across tasks. Models learn which steps to avoid, not just which to take. The authors also note gains in how clearly the model's path to an answer can be read.
Catalog record
| CREATOR | Jason Li Lauren Yraola Kevin Zhu Sean O’Brien |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-16729
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Error Reflection Prompting: Can Large Language Models Successfully Understand Errors? Jason Li Lauren Yraola Kevin Zhu Sean O’Brien Abstract Prompting methods for language models, such as Chain-of-thought (CoT), present intuitive step-by-step processes for problem solving. These methodologies aim to equip models with a better understanding of the correct procedures for addressing a given task. Despite these advancements, CoT lacks the ability of reflection and error correction, potentially causing a model to perpetuate mistakes and errors. Therefore, inspired by the human ability for said tasks, we propose Error Reflection Prompting (ERP) to further enhance reasoning in language models. Building upon CoT, ERP is a method comprised of an incorrect answer, error recognition, and a correct answer. This process enables the model to recognize types of errors and the steps that lead to incorrect answers, allowing the model to better discern which steps to avoid and which to take. The model is able to generate the error outlines itself with automated ERP generation, allowing for error recognition and correction to be integrated into the reasoning chain and produce scalability and reliability in the process. The results demonstrate that ERP serves as a versatile supplement to conventional CoT, ultimately contributing to more robust and capable reasoning abilities along with increased interpretability in how models ultimately reach their errors. 1 Introduction Large Language Models (LLMs) have demonstrated remarkable performance across numerous NLP tasks OpenAI 2023 . Yet scaling them is costly, prompting interest in more efficient methods like Chain-of-Thought (CoT) Wei et al. 2023 , which guides step-by-step reasoning. A key shortcoming of CoT, however, is that it lacks a mechanism to recognize and correct its own mistakes Turpin et al. 2023 . This can lead models to persist with errors, as they lack explicit guidance to reflect on and revise flawed reasoning. Figure 1: Example of Error Reflection Prompting with a focus on misinterpretation. Drawing on the human ability to self-reflect and amend mistakes Huang and Chang 2023 , we propose Error Reflection Prompting (ERP). ERP supplies an incorrect answer alongside an explanation of the errors, then presents the correct reasoning. By exposing common pitfalls and clarifying why they are wrong, ERP steers models away from these errors in future reasoning. Evaluations on multiple benchmarks with GPT-3.5 and GPT-4 confirm that ERP consistently outperforms standard CoT, notably lowering specific error types (e.g., algebraic and calculation mistakes) and balancing the overall error distribution. Thus, ERP adds a complementary layer of self-correction to CoT, enhancing both accuracy and transparency in model outputs. Dataset Type of Reasoning Test GSM8K Arithmetic 1319 AQuA Arithmetic 254 MATH Arithmetic 1000 CSQA Commonsense 1221 StrategyQA Commonsense 2290 Table 1: Problems used from each dataset. Note that CSQA and StrategyQA’s test set answers are not publicly available, so we follow Wei et al. 2023 and evaluate performance on development (dev) datasets. We propose Error Reflection Prompting (ERP) , which prompts an LLM with an incorrect answer, explicit errors, and a corrected solution. This strategy helps the model identify and avoid known pitfalls: 1. Incorrect Answer. Provide a solution with specific errors—ones the model commonly makes Gao et al. 2022 —ensuring the prompt focuses on these weaknesses. 2. Error Reflection. Explain each error and why it is incorrect, offering rationales to help the model understand and avoid repeating them. 3. Correct Answer. Present the corrected chain of reasoning, reinforcing the proper approach. Formally, let E be a set of errors identified during analysis, and Q\subseteq E be a subset of n errors. For each example, we embed Q into the prompt and provide corresponding explanations and the correct solution. In contrast to a single, general CoT prompt, ERP may need unique prompts for each problem, which can be time-consuming. To make ERP scalable, we automate the process: 1. Prompt the model to generate n plausible errors for a given question. 2. Construct an incorrect solution incorporating those errors. 3. Incorporate the question, incorrect solution, errors, and correct solution into the final prompt. By letting the model generate its own error outlines, ERP integrates recognition and correction into the reasoning chain, making it more practical for large-scale tasks. 2 Experiments 2.1 Experimental Setup We focus our analysis on two types of reasoning: commonsense and arithmetic. By using random sampling, we ran tests on all shuffled datasets. Prompting Method Arithmetic Reasoning GSM8K Accuracy AQuA Accuracy MATH Accuracy GPT-3.5 with 5-shot Auto ERP 79.8 % 48.0% ——– GPT-3.5 with 4-shot CoT 74.6% 54.3% 40.2% GPT-3.5 with 4-shot ERP 77.8% 58.7% 41.1% GPT-4 with 4-shot CoT 95.4% 75.9% 54.1% GPT-4 with 4-shot ERP 95.7% 76.7% 54.8% Table 2: Results from a variety of math-based datasets Prompting Method Commonsense Reasoning StrategyQA Acc. CSQA Acc. GPT-3.5 with 4-shot CoT 65.3% 75.8% GPT-3.5 with 4-shot ERP 66.1% 77.3% GPT-3.5 with 4-shot ERP w/ Categories 62.9% —— GPT-4 with 4-shot CoT 79.9% 85.3% GPT-4 with 4-shot ERP 80.0% 86.3% GPT-4 with 4-shot ERP w/ Categories 76.0% —— Table 3: Results from commonsense-based datasets Dataset details are included in Table 1 . Arithmetic Dataset For arithmetic reasoning, we consider three of the following datasets: GSM8K, a dataset for grade school math word problems Cobbe et al. 2021 , AQuA, a dataset consisting of algebraic word problems Ling et al. 2017 , and MATH, a dataset with challenging competition math problems Hendrycks et al. 2021 . All datasets have been run on their respective test sets except for MATH, for which we have limited to 1000 test samples for a reasonable computing limit. Commonsense Datasets In regards to commonsense reasoning, we conduct tests on StrategyQA, which requires models to infer solutions to questions with implicit reasoning steps Geva et al. 2021 , and CSQA, a dataset for commonsense question answering Talmor et al. 2019 . Models We use the OpenAI Chat Completions API, one of the most widely known and used frameworks for accessing language models Ge et al. 2023 . We use GPT-3.5 and GPT-4 for all tests. Specifically, we use gpt-3.5-turbo-0613 and gpt-4-1106-preview respectively for all tests. In addition, we employ 4-shot prompts for each dataset, and each prompt is derived from previous works in Wei et al. 2023 . All errors were created from human annotators or generated using GPT-4 with some slight modifications. Full prompts can be found in Appendix A . 2.2 Analysis Evaluation After collecting the results, we analyze the distribution of errors according to different categories. To accomplish our analysis of the model’s errors, GPT-4 was used as an annotator. A sample of 100 errors from each dataset was pulled to evaluate the effectiveness of GPT-4 as an error detector. To categorize each error and understand the underlying distribution, we separated errors into five and four different classes for arithmetic and commonsense reasoning datasets respectively. For the arithmetic datasets, we have Misinterpretation of Question , Missed Steps , Logical/Commonsense Error , Wrong Selection of Answer , and Algebraic/Calculation Error . For more details on each of these categories, please see Table B in the Appendix. As for the commonsense datasets, we have taken a majority of our errors from the categories presented in Dou et al. 2022 being Commonsense Error , Encyclopedic Error , and Self-Contradiction Error . We have also added an Assumption Error