Knowledge-Enhanced Program Repair for Data Science Code
Shuyin Ouyang, Jie M. Zhang, Zeyu Sun, Albert Merono Penuela · 2025
Read Knowledge-Enhanced Program Repair for Data Science Code by Shuyin Ouyang, Jie M. Zhang, Zeyu Sun, Albert Merono Penuela (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-09771.txt.
Introduction
Shuyin Ouyang and co-authors present a software engineering paper, published in 2025, on fixing faulty code written by language models for data science work. Their tool, DSrepair, repairs broken snippets by first gathering the right knowledge. It searches a knowledge graph built from popular data science libraries for correct API use, and it locates the fault inside the code's syntax tree.
With that knowledge added to its prompts, the repair model fixed far more faulty snippets than five rival methods across four advanced language models. It also used fewer tokens per task, making each repair cheaper to run. Tests used the DS-1000 benchmark, a standard set of data science coding problems, giving the gains a shared yardstick.
Catalog record
| CREATOR | Shuyin Ouyang, Jie M. Zhang, Zeyu Sun, Albert Merono Penuela |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-09771
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Knowledge-Enhanced Program Repair for Data Science Code Shuyin Ouyang, Jie M. Zhang, Zeyu Sun, Albert Merono Penuela Abstract This paper introduces DSrepair, a knowledge-enhanced program repair approach designed to repair the buggy code generated by LLMs in the data science domain. DSrepair uses knowledge graph based RAG for API knowledge retrieval and bug knowledge enrichment to construct repair prompts for LLMs. Specifically, to enable knowledge graph-based API retrieval, we construct DS-KG (Data Science Knowledge Graph) for widely used data science libraries. For bug knowledge enrichment, we employ an abstract syntax tree (AST) to localize errors at the AST node level. We evaluate DSrepair’s effectiveness against five state-of-the-art LLM-based repair baselines using four advanced LLMs on the DS-1000 dataset. The results show that DSrepair outperforms all five baselines. Specifically, when compared to the second-best baseline, DSrepair achieves substantial improvements, fixing 44.4%, 14.2%, 20.6%, and 32.1% more buggy code snippets for each of the four evaluated LLMs, respectively. Additionally, it achieves greater efficiency, reducing the number of tokens required per code task by 17.49%, 34.24%, 24.71%, and 17.59%, respectively. † † aftertitle: I Introduction Data science is crucial in driving innovation and decision-making across various domains [ 1 , 2 ] , leveraging data to uncover insights and inform strategic actions. Nevertheless, the complexity of data science libraries and the expertise required to use them can pose significant barriers to lay users. Large Language Models (LLMs) have emerged as powerful tools to generate data science code automatically [ 3 , 4 , 5 ] , democratizing access and accelerating development processes. Despite their potential, the widely acknowledged shortcomings with LLMs, such as hallucination and the lack of specialized knowledge of certain domains (e.g. long-tail API usage) [ 6 , 7 ] and specific code context [ 8 ] , remain significant obstacles. These issues are particularly critical in the data science domain, where the code heavily relies on libraries for accurate and efficient data processing and analysis, making precision and contextual accuracy essential for robust outcomes. Existing studies have applied feedback-based iterative self repair [ 9 , 10 , 11 , 12 , 13 ] to improve the reliability of LLM-generated code. Nevertheless, these approaches are not designed for data science code. Recently, Retrieval-Augmented Generation (RAG) [ 14 ] has also emerged as a widely-adopted technique to inject external knowledge into LLMs to facilitate more coherent code generation. RAG combines the strengths of information retrieval and LLMs to enhance code generation. Existing RAG-based code generation studies [ 15 , 16 , 17 , 18 ] commonly follow a standard RAG architecture, where the “retriever” component retrieves relevant plain text from a vast corpus or database using a vector similarity search. This retrieved information is then fed into an LLM, which uses this context to produce more accurate and relevant code [ 19 , 20 , 21 , 22 ] . However, these text-based RAG approaches are not well-suited for code generation tasks, as they rely on unstructured or semi-structured plain text, which lacks the semantic relationships and structured representation needed for complex code understanding and generation. Specifically, 1) text-based retrieval relies on vector similarity search, which often retrieves irrelevant or loosely related information due to ambiguities in natural language; 2) plain text does not explicitly represent the relationships between APIs, their dependencies, or their attributes (e.g., parameters, return types). As a result, text-based RAG approaches may fail to provide the comprehensive contextual knowledge required for resolving an issue; 3) retrieved plain text often contains redundant descriptions or ambiguities, which can confuse large language models (LLMs) or lead to suboptimal code generation. This paper introduces DSrepair, a knowledge-enhanced approach for repairing incorrect data science code produced by LLMs through knowledge graph-based RAG and bug information enrichment. We construct DS-KG ( D ata S cience K nowledge G raph), a set of knowledge graphs for the seven most widely adopted data science libraries (i.e., NumPy, Pandas, SciPy, Scikit-learn, Matplotlib, PyTorch, and TensorFlow) [ 23 , 24 , 5 , 3 , 4 ] . For the buggy code generated by an LLM, DSrepair uses the API name appeared in the code as the query, to obtain the correct usage of corresponding API functions by accessing DS-KG. It then uses the returned result to guide the LLM in repairing code. Compared to text-based RAG, our KG-based RAG naturally captures more complex relationships and dependencies within API documents, which is essential in the data science domain. The rich semantic relationships stored in knowledge graphs enhance the reliability and efficiency of code generation and repair. For instance, in Matplotlib’s API document, an API named matplotlib.pyplot.subplots , has a parameter called gridspec_kw . The value of gridspec_kw must be passed to another API object, called matplotlib.gridspec.GridSpec . If an error occurs in gridspec_kw , querying knowledge from matplotlib.gridspec.GridSpec is more helpful than querying only from matplotlib.pyplot.subplots . A well-designed KG can infer such dependency naturally. DSrepair also uses enhanced bug information to improve the program repair effectiveness. Data science code often contains multiple function calls and data operations within a single line. Therefore, to obtain fine-grained bug information, DSrepair uses Abstract Syntax Tree (AST) and test case execution information to localize errors at the AST node level. We evaluate DSrepair on two widely used general-purpose LLM (i.e., GPT-3.5-turbo and GPT-4o-mini [ 25 ] ) and two state-of-the-art coding LLMs (i.e., DeepSeek-Coder [ 26 ] and Codestral [ 27 ] ) based on DS-1000 [ 5 ] , the data science code generation benchmark spanning seven data science libraries. Our results show that DSrepair outperforms all the five baseline LLM-based repairing approaches . In particular, compared to the second-best baseline, DSrepair correctly fixes 44.4%, 14.2%, 20.6%, and 32.1% more buggy code snippets for the four LLMs, respectively, while reducing token usage per code task by 17.49%, 34.24%, 24.71%, and 17.59%. To summarize, this paper makes the following contributions: • We present DSrepair, a novel LLM-based program repair approach for data science code, leveraging knowledge graph-based RAG and enriched bug information. • We construct and release DS-KG (Data Science Knowledge Graph), a comprehensive set of knowledge graphs tailored to the seven most widely used data science libraries. • We propose an AST-based bug information enriching approach that can pinpoint errors at the AST node level. • We conduct an empirical study using four LLMs and five baselines, demonstrating that DSrepair significantly outperforms all baselines in repairing data science code. We release our data, code, KG data dump, and results at our homepage [ 28 ] . The rest of the paper is organized as follows. Section II outlines our methodology. Section III describes the design of the experiments, including research questions, benchmarks, baselines, selected models, and measurements. Section IV presents the results and highlights notable findings based on our empirical results. Section V discusses the threat to validity, the limitation, and the generalizability of our work. Section VI introduces the related work of our study. Section VII concludes. II Method Fig 1 shows an overview of DSrepair. Given a code problem description, we first let LLM (i.e. GPT-3.5-turbo) generate code. If the code fails to pass the test cases, DSrepair constructs a repair prompt and requests LLM to regenerate the