An Empirical Study of Reasoning Steps in Thinking Code LLMs Journal: TOSEM CCS: Software and its engineering Source code generation CCS: Software and its engineering Empirical software validation CCS: Computing methodologies Natural language generation

Haoran Xue, Gias Uddin, Song Wang · 2025

Read An Empirical Study of Reasoning Steps in Thinking Code LLMs Journal: TOSEM CCS: Software and its engineering Source code generation CCS: Software and its engineering Empirical software validation CCS: Computing methodologies Natural language generation by Haoran Xue, Gias Uddin, Song Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-05874.txt.

Introduction

Haoran Xue, Gias Uddin, and Song Wang inspect the visible reasoning traces of thinking language models on coding tasks. These models write out intermediate steps before their final code, which promises clearer and better answers. Whether those step chains are actually good had not been studied closely.

Six leading reasoning models were run on a hundred coding tasks of varied difficulty. The team counted steps, adjusted step budgets up and down, and had twenty-one people rate traces for efficiency, logic, and completeness. More steps helped some models, while small cuts rarely hurt easy tasks. The main failure mode was incompleteness, and hard tasks suffered it far more. Traces kept a steady logical shape even as effort levels changed.

Catalog record

CREATORHaoran Xue, Gias Uddin, Song Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE112 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-05874

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An Empirical Study of Reasoning Steps in Thinking Code LLMs Journal: TOSEM CCS: Software and its engineering Source code generation CCS: Software and its engineering Empirical software validation CCS: Computing methodologies Natural language generation Haoran Xue, Gias Uddin, Song Wang Abstract . Thinking Large Language Models (LLMs) generate explicit intermediate reasoning traces before final answers, potentially improving transparency, interpretability, and solution accuracy for code generation. However, the quality of these reasoning chains remains underexplored. We present a comprehensive empirical study examining the reasoning process and quality of thinking LLMs for code generation. We evaluate six state-of-the-art reasoning LLMs (DeepSeek-R1, OpenAI-o3-mini, Claude-3.7-Sonnet-Thinking, Gemini-2.0-Flash-Thinking, Gemini-2.5-Flash, and Qwen-QwQ) across 100 code generation tasks of varying difficulty from BigCodeBench. We quantify reasoning-chain structure through step counts and verbosity, conduct controlled step-budget adjustments, and perform a 21-participant human evaluation across three dimensions: efficiency, logical correctness, and completeness. Our step-count interventions reveal that targeted step increases can improve resolution rates for certain models/tasks, while modest reductions often preserve success on standard tasks, rarely on hard ones. Through systematic analysis, we develop a reasoning-problematic taxonomy, identifying completeness as dominant failure mode. Task complexity significantly impacts reasoning quality; hard problems are substantially more prone to incompleteness than standard tasks. Our stability analysis demonstrates that thinking LLMs maintain consistent logical structures across computational effort levels and can self-correct previous errors. This study provides new insights into the strengths and limitations of current thinking LLMs in software engineering. 1. Introduction LLMs have achieved remarkable progress in software engineering tasks such as code generation ( Mu et al. 2024 ; Jiang et al. 2024a ) , program translation ( Ibrahimzada et al. 2024 ; Luo et al. 2024 ; Bairi et al. 2024 ) , and bug repair ( Fan et al. 2023 ; Xia et al. 2023 ) . Despite achieving strong pass@1 accuracy on many benchmarks ( et al. 2021 ; Austin et al. 2021 ) , conventional (non-thinking) LLMs face structural limitations in complex coding tasks. First, their reasoning is not effectively transparent: even when correct code is produced, the intermediate hypotheses and decision points remain latent, preventing independent verification of the solution path and inhibiting process-level auditing ( Turpin et al. 2023 ) . Prompted rationales such as chain-of-thought can be elicited, but they are not guaranteed to be faithful to the model’s internal computation ( Turpin et al. 2023 ) . Second, these models exhibit fragility on multi-step programming tasks that demand principled decomposition (from informal requirements to subproblems), systematic edge-case coverage, and the coordination of multiple concepts (APIs, state, concurrency, resource constraints); performance often depends sensitively on prompts and decoding heuristics rather than robust, controllable planning ( Lightman et al. 2023 ) . Third, when generation fails, developers receive little process signal, i.e., only an incorrect final artifact, making it difficult to localize the breakdown (e.g., misread specification vs flawed algorithmic plan vs implementation slip), which slows debugging and hinders iterative improvement ( Liao and Vaughan 2023 ) . Recent thinking LLMs are models that explicitly externalize intermediate reasoning traces before producing final solutions (Fig. 1 ). These advanced LLMs represent a fundamental architectural evolution from the black-box reasoning of traditional models to transparent, step-by-step cognitive processes. Thinking LLMs, exemplified by models such as OpenAI’s o1 and o3 series ( OpenAI et al. 2024 ; OpenAI 2025 ) , DeepSeek-R1 ( et al. 2025c ) , and Claude’s reasoning variants ( Anthropic 2025 ) , generate explicit chains of thought that articulate their problem understanding, solution strategy, implementation approach, and consideration of constraints and edge cases. By articulating their reasoning, these models are expected not only to generate correct code but also to provide explanations that align with program semantics, thereby improving trustworthiness and robustness in practical use ( Kojima et al. 2022 ) . Figure 1 . Processing flow comparison: Non-thinking vs. LLMs with CoT vs. Thinking LLMs Despite these promises, the reliability and robustness of LLM-generated reasoning remain underexplored. Prior work on evaluating LLMs for code tasks has largely focused on the correctness of final outputs, e.g., pass@k for code generation ( et al. 2021 ; Austin et al. 2021 ; Jain et al. 2024 ) . While these studies reveal performance differences across different LLM models, they offer little insight into the quality of the reasoning process itself. As a result, we analyzed the step count and verbosity of the reasoning chains by different models, and then we ask in this paper: How good is the reasoning quality produced by thinking LLMs as perceived by humans while using the LLMs for coding tasks? To answer this question, we have conducted an empirical study to assess reasoning quality in thinking LLMs for code generation through the lenses of developers as human participants in our study. We frame our study around two research questions. RQ1 (Reasoning Process Analysis) examines how thinking LLMs structure their chains. We explore this question from three sub-questions. First, we investigate how many steps thinking LLMs typically produce per task and how this differs between successful versus failed cases and between Hard versus Full tasks (RQ1.1). Second, we investigate whether changes in step count have a causal effect on resolution rates, specifically whether increasing step count improves a model’s resolution rate (RQ1.2.1) and whether reducing step count degrades it (RQ1.2.2). Third, we examine whether verbosity in the reasoning chain is associated with resolution success (RQ1.3). RQ2 (Reasoning Quality Analysis) investigates the reasoning contents that identify patterns that characterize problematic reasoning (RQ2.1), and evaluates reasoning traces through three criteria—efficiency (RQ2.2), logical consistency (RQ2.3), and completeness (RQ2.4). We also examine how task complexity ( Hard vs. Full ) affects both reasoning quality and solution correctness across models (RQ2.5). In our study, we quantify the reasoning chains by the six state-of-the-art reasoning LLMs: DeepSeek-R1, OpenAI-o3-mini, Claude-3.7-Sonnet-Thinking, Gemini-2.0-Flash-Thinking, Gemini-2.5-Flash, and Qwen-QwQ, via step count and verbosity. We record the average steps produced by the thinking models on the success tasks and failed tasks, and to further explore the effect of step counts on the resolution rate changes of solving the code generation tasks, we run two controlled interventions experiments on the six reasoning LLMs: (1) think-deeper by prompting the models to "think deeper" in expanding their reasoning chains uniformly increase from the original step count (from 10% to 100%) on prior failures to measure the resolution rate changes; and (2) reduce-step by reducing steps progressively in the models’ reasoning chains on prior successes to explore the maximum survival space. To assess the reasoning quality from the developers’ perspectives, we conduct a human evaluation study in which participants evaluate the six reasoning LLMs on 100 diverse code generation tasks of varying difficulty levels drawn from BigCodeBench ( et al. 2024 ) . Unlike prior work that only measures the correctness of final code outputs, we ask participants to analyze the reasoning traces themselves to answer important questions about how different LLMs “think” when approaching programming problems, potentially identifying strengths and weaknesses in their reasoning patterns. A total of 21 participants were recruited to label the reasoning steps generated in 600 coding tasks. They evaluated the reasoning traces along three key dimensions, i.e., efficiency ( Xia et al. 2025 ) , logic consistency ( Lee and

Read the complete text