Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA

Jun Zhang, JianYing Qu, Hanwen Du, Zhongkai Sun, Yehua Yang, Qiao Zhao · 2026

Read Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA by Jun Zhang, JianYing Qu, Hanwen Du, Zhongkai Sun, Yehua Yang, Qiao Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29277.txt.

Introduction

Jun Zhang and colleagues present Code-QA-Bench, a framework for testing whether code question-answering reflects genuine comprehension or documentation recall. An answer-first pipeline has a tool-equipped agent explore code to produce verified gold answers first. This grounds every task in real code structure. A three-condition design compares closed-book, code-only, and documented settings across hundreds of tasks from ten Python repositories. Code access proves dominant, documentation adds only modest benefit, and code-only nearly matches the documented setting on code-derivable tasks.

Catalog record

CREATORJun Zhang, JianYing Qu, Hanwen Du, Zhongkai Sun, Yehua Yang, Qiao Zhao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56525
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.29277

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA Jun Zhang, JianYing Qu, Hanwen Du, Zhongkai Sun, Yehua Yang, Qiao Zhao Abstract We present Code-QA-Bench , a fully automated framework for synthesizing repository-level code understanding benchmarks that separates genuine code comprehension from documentation recall and pretraining memorization. The framework makes two methodological contributions: (1) an answer-first generation pipeline where a tool-equipped agent explores source code to produce verified gold answers before deriving questions, ensuring every task is grounded in real code structure; and (2) a three-condition experimental design evaluating agents under closed-book (no repository), code-only (documentation removed), and documented (full repository) conditions, with deltas directly quantifying documentation utility and memorization. We generate 528 code-derivable and 100 doc-dependent tasks across 10 Python repositories from SWE-Bench, scored by an LLM judge on accuracy, completeness, and specificity. Experiments on four frontier models reveal that code access is the dominant factor (+0.23 mean gain over closed-book), documentation provides modest additional benefit (+0.071 on doc-dependent tasks, p < 0.003 p<0.003 ), and code-only ≈ \approx documented on code-derivable tasks ( Δ < 0.01 \Delta<0.01 ), validating the design. The framework is open-source and applicable to any well-documented Python repository. Keywords: code comprehension, repository-level QA, benchmark, documentation utility, LLM evaluation, code reasoning 1 Introduction The dominant use of AI coding assistants in practice is not writing code from scratch, but understanding existing codebases [ 27 ] . Code understanding is broad: it includes reasoning about the input/output behavior of a function, locating the code responsible for a given feature, tracing control flow across modules, and building enough context to confidently propose a bug fix. Yet the most prominent benchmarks for AI coding agents (HumanEval [ 4 ] , MBPP [ 2 ] , and SWE-Bench [ 12 ] ) focus on code generation or issue resolution, leaving code comprehension under-evaluated. This gap matters. An agent that can generate correct patches may still fail to explain why a function behaves a certain way, or to locate the right file for a given concept in a 500-file repository. Code understanding is a prerequisite for safe and reliable code modification, and measuring it independently provides signal that generation benchmarks miss. CRUXEval [ 7 ] highlighted this by showing that code generation ability does not imply code reasoning ability, but operates only at the function level. RepoReason [ 17 ] extended reasoning evaluation to the repository level, revealing that aggregating information across files is the primary cognitive bottleneck for frontier models. Recent work on repository-level code QA [ 24 , 3 ] has made progress on question taxonomies and difficulty calibration, but evaluates agents on repositories with full documentation intact, making it difficult to separate genuine code reading from documentation recall or pretraining memorization. We introduce Code-QA-Bench , a benchmark designed around four principles: (1) a three-condition experimental design evaluating each task under closed-book (no repository), code-only (documentation removed), and documented (full repository) conditions, with deltas directly quantifying documentation utility and memorization; (2) documentation removal as an environment-level control , removing all docstrings, comments, and documentation files to force code-structural reasoning while retaining semantic signal from identifiers, imports, and type annotations (Section 8 ); (3) answer-first task generation where a tool-equipped agent explores source code to produce verified gold answers before deriving questions, ensuring every task is grounded in real code structure; and (4) unified continuous scoring via an LLM judge on three axes (accuracy, completeness, specificity), each 0–5, normalized to a single 0–1 score. Code-QA-Bench uses 10 Python repositories from SWE-Bench [ 12 ] (Table 1 ), pinned to specific commits for reproducibility. Tasks follow a four-type taxonomy (What/Why/Where/How) with 12 subtypes from SWE-QA [ 24 ] , balanced per repository. The entire pipeline is fully automated and repo-agnostic : it can be applied to any well-documented Python repository without modification. 2 Related Work Function-level and repository-level code benchmarks. Early code understanding benchmarks operate at the function level: CRUXEval [ 7 ] , CodeQA [ 21 ] , CS1QA [ 13 ] , LiveCodeBench [ 10 ] , and EvalPlus [ 19 ] . None capture the cross-file reasoning demands of real-world development. At the repository level, SWE-Bench [ 12 ] and its extensions [ 1 , 29 , 5 ] evaluate code generation via patches, while others address cross-file completion [ 6 , 22 ] , lifecycle tasks [ 14 , 15 ] , and agent architectures [ 30 , 28 ] . In all cases, code understanding is a means to generation, not the measured outcome. Repository-level code QA. SWE-QA [ 24 ] is the first large-scale repository-level code QA benchmark (576 questions, 12 repositories, 4-type taxonomy). SWE-QA-Pro [ 3 ] improves on it by selecting 26 long-tail repositories and applying difficulty calibration to filter questions answerable without tools, demonstrating that many existing items do not require repository interaction. Other work evaluates code reasoning through assertion verification [ 17 ] , execution prediction [ 20 ] , multi-dimensional frameworks [ 32 ] , and automated benchmark synthesis [ 11 ] . Our work addresses the memorization problem from a different angle: rather than filtering easy questions or selecting unfamiliar repositories, we directly measure memorization via three evaluation conditions and adopt SWE-QA’s taxonomy with balanced distribution and an answer-first generation pipeline. Evaluation methodology. We use rubric-guided LLM-as-judge evaluation [ 33 , 18 ] with a 3-axis rubric emphasizing concreteness. Our documentation-removal approach to contamination control relates to CodeSearchNet’s [ 8 ] separation of code from natural-language descriptions, and complements temporal methods [ 10 ] and difficulty calibration [ 3 ] by removing the natural-language layer most susceptible to memorization. 3 Benchmark Design 3.1 Repository Selection and Pinning We select 10 of the 12 SWE-Bench Python repositories [ 12 ] (Table 1 ), excluding Flask and Requests due to insufficient documentation volume. Unlike SWE-QA-Pro [ 3 ] , which avoids popular repositories, we retain well-known projects and address memorization via the three-condition design (Section 5.1 ). Each repository is pinned to a specific git commit SHA for reproducibility. Task counts are allocated by size tier: large ( > 1,500 .py files) receive 72 tasks, medium (200–1,000) receive 48, and small ( < 200) receive 24, yielding 528 total. The pipeline makes no assumptions specific to these repositories and can be applied to any well-documented Python project. Table 1: Benchmark repositories, drawn from SWE-Bench [ 12 ] . Task counts are allocated by repository size tier. Repository Domain .py Files Commit SHA Tasks django Web framework 2,894 856c9153... 72 pylint Static code analysis 2,366 b715af6c... 72 sympy Symbolic mathematics 1,589 693a559a... 72 scikit-learn Machine learning 993 f3182980... 48 astropy Astronomy library 987 30686862... 48 matplotlib Plotting / visualization 913 cc6cead6... 48 sphinx Documentation generator 774 cc7c6f43... 48 pytest Testing framework 262 8ecf49ec... 48 xarray N-D labeled arrays 237 92601de1... 48 seaborn Statistical visualization 151 32088bbc... 24 Total 528 3.2 Documentation Removal To isolate code comprehension from documentation recall, we create a code-only version of each repository that removes three categories of natural-language content: 1. Docstrings. We use Python’s Abstract Syntax Tree (AST) module to identify docstrings (string literals that are the first statement in a module, class, or function body). These are removed from the source. When removing a

Read the complete text