How Natural Language Proficiency Shapes GenAI Code for Software Engineering Tasks
Ruksit Rojpaisarnkit 1 , Youmei Fan 1 , Kenichi Matsumoto 1 · 2025
Read How Natural Language Proficiency Shapes GenAI Code for Software Engineering Tasks by Ruksit Rojpaisarnkit 1 , Youmei Fan 1 , Kenichi Matsumoto 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04115.txt.
Introduction
Ruksit Rojpaisarnkit, Youmei Fan, Kenichi Matsumoto, and Raula Gaikovina Kula ask whether a developer's English proficiency in a prompt changes the code a language model writes. Prompt structure has been studied heavily, but the language level itself is an overlooked factor as model tools spread through software work.
Using one hundred sixty-four HumanEval tasks, prompts were rewritten from basic to advanced English while the task stayed fixed. Models defaulted to an intermediate language level in output. Effects on code proficiency varied by model, yet higher-proficiency prompts consistently produced more correct code across all models tested. Prompt language, the authors conclude, is a real lever for reliability.
Catalog record
| CREATOR | Ruksit Rojpaisarnkit 1 , Youmei Fan 1 , Kenichi Matsumoto 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-04115
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How Natural Language Proficiency Shapes GenAI Code for Software Engineering Tasks Ruksit Rojpaisarnkit 1 , Youmei Fan 1 , Kenichi Matsumoto 1 Abstract With the widespread adoption of Foundation Model (FM)-powered tools in software engineering, the natural language prompt has become a critical interface between developers and Large Language Models (LLMs). While much research has focused on prompt structure, the natural language proficiency is an underexplored factor that can influence the quality of generated code. This paper investigates whether the English language proficiency itself independent of the prompting technique affects the proficiency and correctness of code generated by LLMs. Using the HumanEval dataset, we systematically varied the English proficiency of prompts from basic to advanced for 164 programming tasks and measured the resulting code proficiency and correctness. Our findings show that LLMs default to an intermediate (B2) natural language level. While the effect on the resulting code proficiency was model-dependent, we found that higher-proficiency prompts consistently yielded more correct code across all models. These results demonstrate that natural language proficiency is a key lever for controlling code generation, helping developers tailor AI output and improve the reliability of solutions. and Raula Gaikovina Kula 2 I Introduction Foundation Model (FM)-powered coding assistants are disrupting the software development landscape. Developers are utilizing AI technologies like GitHub Copilot 1 1 1 https://github.com/features/copilot , ChatGPT 2 2 2 https://openai.com/index/chatgpt/ , and Gemini 3 3 3 https://gemini.google.com/app to generate code and solve software engineering tasks. In recent years, Large Language Models (LLMs) have become powerful coding assistants, particularly for code generation [ 1 , 2 ] . The efficacy of these models heavily relies on the natural language prompts they receive, making prompt quality a key determinant of output quality [ 3 ] . However, while much research has focused on prompt engineering and model fine-tuning to improve results [ 4 , 5 ] , less attention has been paid to how the natural language complexity of the prompt itself affects the generated code. The proficiency of the generated code is a critical, yet overlooked factor. A mismatch between the code complexity and the user’s skill level can create significant issues. This can lead to novices being overwhelmed by overly complex code or experts being frustrated by simplistic, inefficient solutions. Consider a developer who is highly proficient at describing tasks in natural language (e.g., C1-level proficiency) but has only a novice-level understanding of the programming language (e.g., A1-level proficiency). If a LLMs respond to the sophisticated prompt and generates code that is also at a C1-level proficiency, this developer may face significant challenges in understanding, debugging, and safely integrating the code. Recent studies show that students and practitioners often struggle to understand LLM-generated code due to unfamiliar code styles, an overestimation of its correctness, and their own limited programming proficiency [ 6 ] . Furthermore, other research has found that beginning programmers struggle to write effective prompts and often misunderstand the resulting code, revealing a fundamental disconnect between novice users and LLMs [ 7 ] . Prompt Sample AI-Generated Response Translate the following software engineering problem into natural language including function signature, docstring, body, and unit testcases without code. Make sure it fits into one paragraph. The function ‘has_close_elements’ takes a list of floating-point numbers, ‘numbers’, and a floating-point ‘threshold’. It checks if there are any two numbers in the list that are closer to each other than the specified threshold. The function returns ‘True’ if such a pair of numbers exists, and ‘False’ otherwise. For example, calling ‘has_close_elements’ with the list ‘[1.0, 2.0, 3.0]’ and a threshold of ‘0.5’ returns ‘False’ because no two numbers in the list are within ‘0.5’ units of each other. Conversely, calling it with ‘[1.0, 2.8, 3.0, 4.0, 5.0, 2.0]’ and a threshold of ‘0.3’ returns ‘True’ because at least one pair of numbers is closer than ‘0.3’ units apart. TABLE I: AI-Generated Descriptions of Software Engineering Problems This challenge extends to various settings. In a collaborative industry environment, it can create significant friction and increase review overhead. For instance, if a senior developer uses LLMs to produce highly complex, idiomatic code, junior team members may struggle to understand, review, and later maintain it, impacting team velocity. Furthermore, modern software teams often operate in environments where developers have varied expertise across multiple languages. Ensuring AI-generated code aligns with their specific proficiency in the target language is crucial for preventing subtle bugs [ 8 ] . Therefore, tailoring the proficiency of generated code to user expertise is a critical factor for the successful adoption of LLMs. In this paper, we conduct an empirical study using several leading LLMs (GPT-4o, Gemini 2.5 Pro, and Claude Sonnet 4) and the HumanEval dataset [ 9 ] to investigate whether the natural language proficiency of a prompt affects the proficiency and correctness of the generated code. Briefly, the proficiency of an artifact (a prompt or generated code) refers to the proficiency level required by a person to author or comprehend it. We modified the English proficiency level of each problem description based on the Common European Framework of Reference for Languages (CEFR), a robust international standard for defining language proficiency. By prompting we align the CEFR [ 10 ] according to the following levels and example words: • CEFR A1 - Beginner (i.e. about, above) • CEFR A2 - Elementary (i.e. ability, access, acceptance) • CEFR B1 - Intermediate (i.e. accidental, accompany) • CEFR B2 - Upper Intermediate (i.e. absurd, actual) • CEFR C1 - Advanced (i.e.complexity, reluctantly) • CEFR C2 - Proficient (i.e. eccentricity, deliberate) Our study is based on the software engineering tasks dataset from HumanEval, which contains 164 hand-written programming problems in Python. Each problem includes a prompt (doc-string), a canonical solution, and test cases. To address known issues with the original test cases and ensure the reliability of our pass@1 metric, we incorporate the more robust test suite from HumanEvalPlus [ 9 ] . To measure code proficiency, we adapt the levels defined by PYCEFR [ 11 ] , which assigns proficiency ratings to Python elements based on their conceptual difficulty. For example, print() and return are considered A1 (Beginner), if/else and assert are B1 (Intermediate), and zip() and map() are C2 (Proficient). We then ask the following research questions: 1. RQ1: What is the baseline English proficiency (CEFR level) of problem descriptions generated by LLMs? While the challenges of code comprehension are known, the causal link from natural language prompt proficiency to code proficiency is not well understood. This research question aims to investigate the proficiency level of LLMs-generated descriptions based on the CEFR based proficiency. 2. RQ2: Does the English proficiency (CEFR level) of a natural language prompt influence the code proficiency and correctness of the solutions generated by LLMs? With the baseline established from RQ1, we can then systematically explore the core of the problem. This research question investigates whether the natural language proficiency of problem descriptions influences the code proficiency of solutions generated by LLMs. Our findings reveal that LLMs tend to generate problem descriptions at a B2 (Upper Intermediate) level or higher, suggesting a baseline English proficiency is required to understand their output. Furthermore, a clear correlation exists for some LLMs: adjusting the natural language