Defective Task Descriptions in LLM-Based Code Generation: Detection and Analysis
Amal Akli, Mike Papadakis, Maxime Cordy, Yves Le Traon · 2026
Read Defective Task Descriptions in LLM-Based Code Generation: Detection and Analysis by Amal Akli, Mike Papadakis, Maxime Cordy, Yves Le Traon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-24703.txt.
Introduction
Akli, Papadakis, Cordy and Le Traon address defective task descriptions in language-model code generation, where vague or underspecified prompts hurt code correctness. They build SpecValidator, a lightweight classifier based on a small fine-tuned model, to detect lexical vagueness, under-specification and syntax-formatting defects. On three benchmarks it reaches an F1 score of 0.804, significantly beating larger models. It also generalizes to unseen defects in the original benchmark descriptions.
Catalog record
| CREATOR | Amal Akli, Mike Papadakis, Maxime Cordy, Yves Le Traon |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50850 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.24703
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Defective Task Descriptions in LLM-Based Code Generation: Detection and Analysis Amal Akli, Mike Papadakis, Maxime Cordy, Yves Le Traon Abstract . Large language models are widely used for code generation, yet they rely on an implicit assumption that the task descriptions are sufficiently detailed and well-formed. However, in practice, users may provide defective descriptions, which can have a strong effect on code correctness. To address this issue, we develop SpecValidator , a lightweight classifier based on a small model that has been parameter-efficiently finetuned, to automatically detect task description defects. We evaluate SpecValidator on three types of defects, Lexical Vagueness, Under-Specification and Syntax-Formatting on 3 benchmarks with task descriptions of varying structure and complexity. Our results show that SpecValidator achieves defect detection of F1 = 0.804 and MCC = 0.745, significantly outperforming GPT-5-mini (F1 = 0.469 and MCC = 0.281) and Claude Sonnet 4 (F1 = 0.518 and MCC = 0.359). Perhaps more importantly, our analysis indicates that SpecValidator can generalize to unseen issues and detect unknown Under-Specification defects in the original (real) descriptions of the benchmarks used. Our results also show that the robustness of LLMs in task description defects depends primarily on the type of defect and the characteristics of the task description, rather than the capacity of the model, with Under-Specification defects being the most severe. We further found that benchmarks with richer contextual grounding, such as LiveCodeBench, exhibit substantially greater resilience, highlighting the importance of structured task descriptions for reliable LLM-based code generation. 1. Introduction Large language models (LLMs) such as Codex ( 5 ) , AlphaCode ( 21 ) , Code Llama ( 35 ) , and Qwen2.5 Coder ( 14 ) , along with related LLM agents ( 31 ; 1 ) , increasingly support developers across a broad spectrum of programming tasks, such as code generation ( 18 ) . Previous work demonstrates their practical impact: AI-based coding tools boost developer productivity ( 32 ) . In parallel, the evaluation methodologies for these systems have evolved substantially. Benchmarks such as HumanEval ( 5 ) and MBPP ( 2 ) established the unit-test pass rate as a standard evaluation paradigm. However, subsequent work has revealed important limitations. EvalPlus ( 22 ) shows that increasing the test coverage significantly reduces the reported performance, highlighting the fragility of previous assessments. More recently, LiveCodeBench ( 16 ) addressed additional concerns by continuously updating problem sets to mitigate data contamination. In addition, it introduces richer problem specifications, incorporating explicit numerical constraints, formal input–output contracts, and illustrative examples alongside natural language descriptions. Interestingly, the above studies evaluate the generation of code on well-formed and sufficiently described task descriptions. However, in practice, natural language descriptions are inherently imprecise and may be subject to several description issues, similar to specification defects that appear in software specifications ( 28 ; 41 ; 19 ; 42 ) . Therefore, task descriptions can include vague terminology, omit edge-case constraints, or may contain surface-level noise, such as typographical and formatting errors, all of which can drastically affect the correctness of the generated code ( 19 ; 17 ; 48 ; 45 ) . To address this issue, we developed SpecValidator , a simple classifier to automatically detect such task description defects prior to code generation. SpecValidator is a lightweight LoRA-fine-tuned predictor trained on a set of artificially injected defects. In particular, we define and apply three defect types — Lexical Vagueness (LV), Under-Specification (US), and Syntax and Formatting (SF) — reflecting the key issues task descriptions may have, based on which we fine-tune our predictor and check its ability to detect them. We evaluate 10 LLMs spanning from small (6–7B) to large open-source (15–34B) and state-of-the-art reasoning models across 3 benchmarks and found significant degradation in the models’ code correctness. Interestingly, our results show that the structural characteristics of the task descriptions play an important role on the correctness of the generated code. Under-Specification causes the most significant drops, of up to 15.3%, with no model being robust, while benchmarks with richer contextual grounding, such as LiveCodeBench, are substantially more resilient. Lexical Vagueness produces moderate and benchmark-dependent degradation, whereas surface formatting issues have a negligible effect. SpecValidator shows strong defect detection performance with F1 = 0.804 and MCC = 0.745, outperforming significantly few-shot GPT-5-mini (F1 = 0.462 and MCC = 0.276) and Claude Sonnet 4 (F1 = 0.557 and MCC = 0.412). Perhaps more importantly, by applying SpecValidator on the original set of task descriptions, we reveal that these surprisingly contain Under-Specification defects (confirmed via thorough manual inspection). This demonstrates the ability of SpecValidator to detect real specification defects. Thus, in a sense we show that SpecValidator generalizes beyond the artificial defects we used for the fine-tuning, with a precision of 72%, correctly identifying 72% of the real unseen descriptions as defective. The main contributions of this paper are: • A large-scale analysis of 10 models and 3 benchmarks, showing that defects’ impact is shaped both by defect type and task description structure. • SpecValidator A lightweight defect detector, a Qwen2.5-Coder-1.5B LoRA fine-tune model, achieving F1 = 0.804 and MCC = 0.745, outperforming both GPT-5 mini and Claude Sonnet 4. SpecValidator is capable of detecting real description defects in the benchmarks we use. • A publicly available dataset of 6,573 defective task description instances, supporting replication and future research on task description and benchmark design. 2. Related Work 2.1. LLM-based code generation and evaluation The HumanEval ( 6 ) , MBPP ( 2 ) and EvalPlus ( 22 ) benchmarks evaluate functional correctness using test pass rates (e.g. pass@k ). LiveCodeBench ( 16 ) mitigates benchmark contamination by continuously introducing new competitive-programming problems, while BigCodeBench ( 49 ) , and Siddiq et al. ( 38 ) extend evaluation to more complex, library-intensive tasks that require diverse API usage and more realistic programming workflows. Open-weight models include Code Llama ( 35 ) , StarCoder2 ( 24 ) , DeepSeek-Coder ( 10 ) , and Qwen2.5-Coder ( 14 ) , while proprietary systems such as GPT-5 ( 31 ) and Claude Code ( 1 ) represent the state of the art in commercial deployments. Despite the growing diversity of benchmarks and models, evaluation practices remain largely focused on correctness under well-specified task descriptions, leaving model behavior under realistic description defects unexplored. 2.2. Prompt engineering for code tasks Li et al. ( 20 ) proposed Structured Chain-of-Thought (SCoT) prompting, which decomposes generation through program-structure-aware intermediate steps ( 44 ) , yielding consistent improvements over standard prompting on HumanEval and MBPP. Test-driven feedback loops ( 9 ) ask users to refine the task description using feedback from tests generated from the original description, using a multi-turn LLM dialog. Self-Debugging ( 7 ) teaches models to self-identify and correct errors in their own outputs via execution-guided feedback. These studies focus on the generation of correct code from well-specified, sufficiently detailed descriptions and excluding potential defects. 2.3. Detecting description defects ClarifyGPT ( 30 ) and Larbi et al. ( 19 ) , and Wu et al. ( 45 ) use LLMs to detect defective task descriptions. These approaches have been evaluated on relatively simple cases, such as HumanEval and MBPP, and on a limited number of defects and types. The baselines