Think Like Human Developers: Harnessing Community Knowledge for Structured Code Reasoning

Chengran Yang, Zhensu Sun, Hong Jin Kang, Jieke Shi, David Lo · 2025

Read Think Like Human Developers: Harnessing Community Knowledge for Structured Code Reasoning by Chengran Yang, Zhensu Sun, Hong Jin Kang, Jieke Shi, David Lo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14838.txt.

Introduction

Chengran Yang and co-authors mine developer talk to teach code reasoning in this 2025 paper. Language models write code well but stumble on tasks that need many logic steps. Good reasoning data are scarce, and machine-made chains often hold errors.

Their framework, SVRC, pulls reasoning chains from community forums where developers solve real problems. It reshapes messy threads along software life cycle lines and checks each chain. A model called CodeThinker was trained on more than twelve thousand such samples. On a live coding benchmark it beat its base model by a wide margin on medium tasks. It also passed two strong commercial models by large gaps.

Catalog record

CREATORChengran Yang, Zhensu Sun, Hong Jin Kang, Jieke Shi, David Lo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-14838

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Think Like Human Developers: Harnessing Community Knowledge for Structured Code Reasoning Chengran Yang, Zhensu Sun, Hong Jin Kang, Jieke Shi, David Lo Abstract . Large Language Models (LLMs) have significantly advanced automated code generation, yet they struggle with complex coding tasks requiring multi-step logical reasoning. High-quality reasoning data is crucial for improving LLMs’ reasoning capabilities, but such datasets remain scarce. Existing approaches either rely on computationally expensive reinforcement learning (RL) or error-prone reasoning chains synthesized by LLMs, posing challenges in scalability and accuracy. To address this challenge, we propose SVRC (Structured and Validated Reasoning Chains for Code Generation), a novel framework that mines, restructures, and enriches reasoning chains from community-driven discussions on software engineering platforms. SVRC refines unstructured and incomplete discussions of coding problems by aligning them with Software Development Life Cycle (SDLC) principles, ensuring that reasoning chains capture real-world problem-solving strategies and support iterative refinement. To evaluate the effectiveness of SVRC, we introduce CodeThinker, an LLM fine-tuned on 12,444 reasoning-augmented samples generated by SVRC. Experiments on LiveCodeBench show that CodeThinker surpasses its base model by 42.86% on medium-level code problems in terms of pass@1 and outperforms GPT-4o-mini and GPT-4o by 73.14% and 115.86%, respectively. Our ablation study further highlights that each component of SVRC contributes to the reasoning capabilities of CodeThinker. 1. Introduction Large Language Models (LLMs) have revolutionized software engineering with exceptional performance across a spectrum of coding tasks ( 39 , 24 , 16 , 42 ) , ranging from code summarization ( 7 , 8 , 48 ) , vulnerability detection ( 52 , 50 ) , to program repair ( 30 , 31 , 34 ) . Among these, code generation ( 29 , 35 , 46 ) —the process of generating code from natural language requirements—has been one of the most impactful applications of LLMs, enabling automatic tools like GitHub Copilot ( 17 ) and Cursor Editor ( 14 ) to assist developers. However, despite their ability to generate syntactically correct code, recent studies indicate that LLMs often struggle to fully understand the natural language task descriptions and fall short in constructing solutions that require advanced logical reasoning ( 43 , 15 ) , highlighting an urgent need to improve their code reasoning capabilities. Being data-driven, improving LLMs’ capacity for code reasoning typically requires training with high-quality datasets that explicitly encode reasoning steps for complex coding problems. However, manually creating such datasets is time-consuming and costly ( 21 , 12 , 22 ) , demanding advanced programming expertise, which has resulted in a scarcity of large-scale code reasoning datasets. To address this, researchers have explored alternative training methods that leverage synthetic reasoning chains generated by LLMs. For example, OpenAI’s o1 ( 27 ) and DeepSeek’s R1 ( 19 ) utilize reward optimization to encourage LLMs to autonomously produce step-by-step reasoning. However, their reliance on large-scale base models and associated high computational costs significantly restrict their scalability and broader adoption. Other approaches involve fine-tuning LLMs using reasoning traces from math and science problems generated by larger LLMs ( 41 , 40 , 33 ) . Yet, these synthetic reasoning chains can be error-prone since they heavily depend on the intrinsic reasoning abilities of LLMs, which may not always generate reliable reasoning steps comparable to those produced by humans. This becomes especially problematic when flawed reasoning still results in a correct final answer ( 45 ) , as it can mislead the fine-tuning process. Thus, there is a pressing need for a scalable as well as reliable method to construct code-generation datasets enriched with accurate reasoning chains. Figure 1. An example of community discussions on LeetCode. To address this need, we turn to software engineering community platforms such as LeetCode ( 2 ) and Stack Overflow ( 5 ) , which host extensive collections of coding problems, corresponding solutions, and detailed discussions. As illustrated in Figure 1 , these discussions offer authentic, peer-reviewed insights into developers’ reasoning and problem-solving strategies, making them a valuable yet underutilized resource for mining human reasoning patterns. Such reasoning patterns hold great potential to enhance LLMs’ capabilities for human-like code reasoning. However, harnessing community discussions on coding problems to enhance LLMs’ code reasoning ability is challenging because they are often unstructured, scattered, and incomplete. We show that directly applying disorganized community discussions to fine-tune LLMs prevents models from effectively understanding and learning reasoning patterns (as discussed in Section 5.3 ). Moreover, human developers rarely arrive at a perfect solution on their first attempt; instead, they follow an iterative refinement process—reconsidering requirements, debugging, and optimizing their code in cycles. However, most community discussions focus only on the final correct solution, omitting the trial-and-error reasoning that led to it, which is crucial for LLMs to capture and mimic the iterative problem-solving process. This motivates us to develop a method that not only extracts and structures reasoning chains from successful solutions but also enriches them with the iterative refinement process that reflects real-world coding practices. In this paper, we propose SVRC (Structured and Validated Reasoning Chains for Code Generation), a novel method for extracting and augmenting code generation datasets with structured reasoning chains mined from community platforms. SVRC transforms unstructured problem-solving insights into well-formed, iterative reasoning steps aligned with Software Development Life Cycle (SDLC) principles ( 37 ) —a systematic software development process encompassing phases such as problem revisiting, planning, design, coding, testing, and refinement. Specifically, SVRC begins by collecting coding problems along with their discussions, applying heuristic rules to retain only high-quality content. These discussions are then processed to fill in missing sections of reasoning chains using state-of-the-art LLMs while also perturbing code formats to enhance dataset diversity and ensure adherence to a structured reasoning format. At this stage, the extracted reasoning chains provide coherent steps for a code solution but fail to capture the iterative refinement process of human developers. To address this, we draw inspiration from SDLC principles and employ an LLM to refine the raw reasoning chains. This refinement process mirrors key SDLC stages, including requirement analysis, planning, solution design, and solution development with testing. Additionally, we expand the reasoning chains by explicitly incorporating iterative solution refinement, enabling them to systematically identify issues, analyze their root causes, and progressively optimize solutions. The resulting structured reasoning chains are comprehensive and well-organized, making them suitable for fine-tuning LLMs to emulate human-like reasoning in code-generation tasks. To evaluate the effectiveness of SVRC, we fine-tune Qwen2.5-Instruct-32B and release CodeThinker, an LLM with enhanced code reasoning capabilities, as a proof of concept. The fine-tuning process utilizes 12,444 data samples synthesized from SVRC and sourced from LeetCode discussions. We evaluate the performance of CodeThinker on LiveCodeBench, which continuously aggregates the latest coding problems from diverse competitive programming platforms such as LeetCode, Codeforces, and AtCoder. To ensure a fair evaluation, we exclude all LeetCode problems used during fine-tuning. The experimental results show that, by fine-tuning with only 12,444 data samples, CodeThinker enables advanced code reasoning and outperforms its base model on medium-level problems by 42.86% in terms of pass@1 and achieves on-par performance on easy-level problems. Furthermore, CodeThinker achieves better performance than GPT-4o-mini and GPT-4o in addressing medium-level code problems by 73.14% and 115.86% in terms of pass@1, respectively. Our ablation study further illustrates that each stage

Read the complete text