Are They All Good? Evaluating the Quality of CoTs in LLM-based Code Generation

Binquan Zhang, Li Zhang, Zhiwen Luo, Yuxin Du, Fang Liu, Song Wang, Lin Shi · 2025

Read Are They All Good? Evaluating the Quality of CoTs in LLM-based Code Generation by Binquan Zhang, Li Zhang, Zhiwen Luo, Yuxin Du, Fang Liu, Song Wang, Lin Shi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06980.txt.

Introduction

Binquan Zhang and six co-authors grade the reasoning steps that language models produce before writing code in 2025. These chains of thought act like design notes, so their quality should drive the code's quality. The team dissected over a thousand failed solutions from two standard benchmarks.

Outside factors, unclear demands and missing context, caused over half the bad reasoning; the rest came from the models misreading prompts. Even with sound reasoning, nearly a fifth of the code still failed from poor instruction following, while some correct code rode on flawed reasoning. Feeding detailed problem statements let models repair weak chains, sketching a path to more trustworthy code generation.

Catalog record

CREATORBinquan Zhang, Li Zhang, Zhiwen Luo, Yuxin Du, Fang Liu, Song Wang, Lin Shi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-06980

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Are They All Good? Evaluating the Quality of CoTs in LLM-based Code Generation Binquan Zhang, Li Zhang, Zhiwen Luo, Yuxin Du, Fang Liu, Song Wang, Lin Shi Abstract Large language models (LLMs) have demonstrated impressive performance in code generation, particularly when augmented with chain-of-thought (CoT) prompting techniques. They break down requirements into intermediate reasoning steps, which act as design rationales to guide LLMs in writing code like human programmers. Thus, the quality of these steps is crucial for ensuring the correctness and reliability of the generated code. However, little is known about the quality of CoT generated by LLMs. To what extent can we trust the thoughts generated by LLMs? How good are they? This paper empirically explores the external and internal factors of why LLMs generate unsatisfactory CoTs by analyzing 1,023 failed code samples on two widely used code generation benchmarks. We also evaluate their impact on code generation performance by analyzing 210 CoT-code pairs and refining the unsatisfied CoTs by prompting LLMs. Our study yields the following findings: 1) Among the factors affecting CoT quality, external factors account for 53.60%, primarily including unclear requirements and lack of contextual information. Internal factors make up 40.10%, mainly due to inconsistencies between CoT and prompts caused by LLMs’ misunderstanding of the instructions. 2) Despite CoT being correct, 18.5% of the generated code still contains errors. This is primarily due to LLMs failing to follow instructions, leading to inconsistencies between CoT and the code. Additionally, we found that even when the code is correct, there is an 11.90% chance that the CoT contains errors. 3) Our further research on refining the low-quality CoTs reveals that LLMs can improve CoT, especially when providing detailed CoT problem information. Our findings shed light on the underlying issues that hinder the effectiveness of CoT in LLM-based code generation, offering valuable insights for enhancing both the reasoning process and the overall reliability of code generation. I Introduction Large Language Models (LLMs) have recently made remarkable advancements in their reasoning capabilities, a trend exemplified by the emergence of reasoning models like OpenAI’s o1 [ 2 ] , DeepSeek-R1 [ 11 ] , and Gemini-2.0-Flash-Thinking [ 1 ] . Chain of Thought (CoT) [ 43 ] represents one of the most pivotal breakthroughs in eliciting these reasoning abilities. This approach guides models to systematically articulate their thought processes before delivering a final answer, effectively bridging the gap between “fast thinking” and “slow thinking”. In complex domains like code generation, this step-by-step reasoning is particularly valuable. It emulates the cognitive strategies of human programmers by “forcing” LLMs to break down complex task requirements into a series of intermediate reasoning steps [ 22 , 30 , 13 ] . These steps serve as a design foundation, guiding LLMs to write code like human programmers. Extensive research has been done to generate multi-step reasoning chains with LLMs, such as Tree-of-Thought(ToT) [ 49 ] , Structured Chain-of-Thought(SCoTs) [ 22 ] , Reasoning-via-Planning(RAP) [ 12 ] , among others [ 47 , 53 , 18 ] . While these methods have improved the performance of code generation, little is known about the quality of the intermediate reasoning steps. To what extent can we trust the “thoughts” generated by LLMs? How good are they? Therefore, we conduct a large-scale empirical study that evaluates the quality of CoTs and explores automated methods for their detection and repair. We first collect CoTs generated by LLMs and the corresponding code produced based on these CoTs from two code generation benchmarks(i.e., CoderEval [ 52 ] and SWE-bench [ 17 ] ), resulting in a total of 1,023 CoT-code pairs. Among these pairs, we then identify 813 code samples that fail to pass the tests and conduct an in-depth analysis of the associated CoTs. Since CoTs are generated by LLMs in response to a given requirement, we categorize the root causes of low-quality CoTs into two main aspects: external and internal factors, where external factors refer to issues arising from the input requirement during CoT generation, while internal factors are relate to the content of the CoT itself. Our empirical study identifies external factors such as unclear implementation details and missing contextual information in the prompt. Internal factors include misunderstanding explicit requirements , incomprehension of implicit requirements , and incorrect planning within the CoT. Upon evaluation, we find that out of the 813 LLM-generated CoTs, 53.6% of the quality issues stem from external factors, while 40.1% are attributed to internal factors. Beyond analyzing the 813 CoTs, we also extend our investigation to the remaining 210 passing code samples and their associated CoTs. This additional analysis provides a more comprehensive understanding of the impact of LLM-generated CoTs on code quality. Our findings reveal that even when the CoT is correct, 18.5% of the generated code still contains errors . This issue arises mainly because LLMs sometimes fail to adhere to the steps in the CoT, resulting in discrepancies between the CoT and the code. Furthermore, our analysis reveals that even when the generated code is correct, there is still an 11.9% chance of an incorrect CoT . This occurs because LLMs possess strong adaptive capabilities. Even if certain steps in the CoT are flawed, LLMs can rely on its inherent knowledge and contextual understanding to correct these errors and ultimately generate accurate code. Finally, we investigate the self-detection and self-repair capabilities of LLMs. For the detection of faulty CoTs, we employ a Multi-Agent Debate (MAD) [ 24 ] framework. The experiments demonstrate that the MAD framework significantly outperforms a single LLM. Regarding the repair of faulty CoTs, we examine the impact of feedback information at different granularities (ranging from simple feedback to error type information and detailed error descriptions) on the self-repair capabilities of LLMs. Our findings reveal a clear positive correlation: the more detailed the provided feedback, the more effective the model’s repair performance. While this is a preliminary study, we believe that LLMs hold significant potential in detecting and repairing erroneous CoTs. To summarize, this paper makes the following contributions: • To the best of our knowledge, we conduct the first in-depth examination of the factors influencing the quality of LLM-generated CoTs and propose a comprehensive taxonomy of CoT errors, encompassing both external and internal factors. • We further investigate the relationship between CoT quality and the correctness of the generated code. Our findings indicate that even correct CoTs can sometimes lead to faulty code, while, conversely, a small portion of correct code can still be generated from incorrect CoTs. • We conduct a preliminary exploration of faulty CoT detection and repair strategies across different LLMs, highlighting that more detailed error information improves LLMs’ performance in refining faulty CoTs. • Our findings shed light on the underlying issues that hinder the effectiveness of CoT in LLM-based code generation, offering valuable insights for enhancing both the reasoning process and the overall reliability of generated code. II Background & Related Work II-A Code Generation with LLMs LLMs have recently gained prominence in Software Engineering (SE) for automatic code generation. Among the most notable is OpenAI’s CodeX [ 8 ] , a GPT-based auto-regressive model with up to 12 billion parameters, fine-tuned on 54 million public GitHub repositories. CodeX powers GitHub Copilot

Read the complete text