More code, less validation: Risk factors for over-reliance on AI coding tools among scientists
Gabrielle O’Brien, Alexis Parker, Nasir U. Eisty, Jeffrey Carver · 2025
Read More code, less validation: Risk factors for over-reliance on AI coding tools among scientists by Gabrielle O’Brien, Alexis Parker, Nasir U. Eisty, Jeffrey Carver (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-19644.txt.
Introduction
Gabrielle O'Brien, Alexis Parker, Nasir Eisty, and Jeffrey Carver surveyed 868 scientists who write code, asking how generative AI has entered their working lives. Programming is essential to modern research, yet most scientists feel undertrained for it, and earlier studies warn that AI coding help invites over-reliance, especially among beginners. The survey maps adoption, favorite tools, and what tracks with feeling more productive.
Adoption runs highest among students and less experienced programmers, and general chatbots like ChatGPT beat developer-specific tools by a wide margin. Feeling productive correlates with inexperience and with skipping practices like testing and code review, though the two interact. The strongest single predictor is how many generated lines a respondent typically accepts at once, suggesting productivity is being judged by code produced, not code checked.
Catalog record
| CREATOR | Gabrielle O’Brien, Alexis Parker, Nasir U. Eisty, Jeffrey Carver |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-19644
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
More code, less validation: Risk factors for over-reliance on AI coding tools among scientists Gabrielle O’Brien, Alexis Parker, Nasir U. Eisty, Jeffrey Carver Abstract Programming is essential to modern scientific research, yet most scientists report inadequate training for the software development their work demands. Generative AI tools capable of code generation may support scientific programmers, but user studies indicate risks of over-reliance, particularly among inexperienced users. We surveyed 868 scientists who program, examining adoption patterns, tool preferences, and factors associated with perceived productivity. Adoption is highest among students and less experienced programmers, with variation across fields. Scientific programmers overwhelmingly prefer general-purpose conversational interfaces like ChatGPT over developer-specific tools. Both inexperience and limited use of development practices (like testing, code review, and version control) are associated with greater perceived productivity—but these factors interact, suggesting formal practices may partially compensate for inexperience. The strongest predictor of perceived productivity is the number of lines of generated code typically accepted at once. These findings suggest scientific programmers using generative AI may gauge productivity by code generation rather than validation, raising concerns about research code integrity. keywords generative AI, scientific software, research software, productivity 1 Introduction Programming is essential to modern scientific research 14 ; 50 , yet many scientists report inadequate training and support for the software development their work demands 6 ; 26 . The recent emergence of generative AI (genAI)-based tools capable of code generation offers both promise and risk. While some studies of software developers report perceptions of increased productivity, 52 ; 21 , user studies indicate risks for over-reliance—uncritically accepting generations without understanding or taking steps to verify them 36 ; 24 ; 25 ; 43 ; 4 ; 29 . This risk may be especially worrisome in scientific settings 30 , where there is already inconsistent use of industry-standard development practices like code testing or review that provide feedback on code quality 6 ; 49 ; 28 ; 33 . Furthermore, seemingly minor and easily overlooked mistakes in scientific code can have radical consequences for how results are understood and interpreted 38 ; 19 ; 23 . Despite the potential threats to research software integrity, empirical data on genAI adoption among scientists for programming remains sparse. This study addresses three aspects of adoption using survey methods: first, who uses genAI tools most in their research-related programming, and which tools do they prefer? Second, how do users’ programming experience and software development practices relate to perceived productivity with genAI tools? We hypothesize that a lack of programming experience and limited adoption of development practices (such as version control, code review, and testing) will be associated with greater perceived productivity gains, as these factors may indicate fewer scaffolds for assessing code quality and correctness. Finally, why do some scientists reject genAI tools for programming? We next review relevant literature to contextualize these questions and hypotheses. 1.1 Background While studies directly targeting scientific programmers’ usage of genAI tools are limited, there is literature addressing how programmers broadly work with genAI tools, programming practices in scientific settings, and the adoption of genAI in their day-to-day research activities. 1.2 Programming with genAI tools Since genAI tools like ChatGPT and GitHub Copilot became widely available, numerous studies have attempted to characterize how they are changing the work of software development, though mostly in non-scientific contexts. Some studies suggest that perceptions of increased developer productivity with genAI tools correspond to measurable increases in objective measures of programming activities 47 ; 21 , though this finding is inconsistent—other studies have failed to find meaningful productivity increases or even observed reductions 5 ; 40 . Increased developer velocity, even when observed, may be accompanied by a rise in code quality issues 13 . Notably, a large study of GitHub Copilot users cross-referencing survey data and activity logs found that less experienced developers attributed the greatest productivity gains to the tool 52 . Perceived productivity was also strongly related to how often a user accepted code suggestions. While this could indicate that novices genuinely benefit more from assistance, it may also reflect a lack of critical inspection or understanding of generated code. Users who frequently accept generated code may be making conscious choices to delay the work of verification 25 . However, some studies suggest that programmers—particularly inexperienced ones—may rely heavily on superficial forms of “checking” , or may even believe it is not necessary 10 ; 29 ; 27 ; 35 . In a study of novice programmers working with a genAI tool, Prather et al. 36 described the “illusion of competence” , in which novice coders greatly overestimated their understanding of generated code solutions and their own programming ability. Notably, this phenomenon has antecedents in previous eras of programming tools—spreadsheet programmers have been observed to overestimate the correctness of values produced automatically 32 ; 20 . To date, user studies about genAI programming tools have generally not focused on scientific contexts. It is not obvious which findings will generalize. However, if there is an emerging pattern that less experienced developers are prone to being less critical of generated code suggestions, this should be concerning for scientific code production broadly: because many scientists report inadequate development training, they may carry a latent “risk factor” for over-reliance on genAI tools 6 ; 14 ; 26 . 1.3 Scientific programming practices Although there have been tremendous strides in recent years in professionalizing scientific software 41 ; 50 ; 49 , there is still considerable variability in where infrastructure and expertise are available to support practices that are common guardrails on code quality in industry settings 6 ; 49 ; 28 ; 33 . For example, industry-standard practices like code testing 9 ; 18 ; 45 , code review 2 , and version control 16 are used inconsistently in research settings—if at all. Without mechanisms to manage the complexity of research code and support activities to verify its correctness, errors may remain undetected for extended periods of time (including long after publication). Considering that genAI tools may make different kinds of errors than human programmers (fewer syntax errors that prevent code from running, but more semantic errors—subtle mistakes in logic or implementation that are considerably harder to detect 17 ; 46 ), and that genAI may produce code that is exceptionally complex and challenging to maintain 13 , verification work may be even more important for developers than ever. 1.4 Adoption of genAI tools in science research Empirical data on genAI adoption among scientists for programming is sparse. In a 2023 Nature survey, over 40% of researchers agreed genAI would make programming faster 44 , but this survey did not ask about their actual uses. A more recent survey of a nationally representative sample of academic scientists indicated 65% of respondents used AI tools in their research at least some of the time, but did not explicitly address programming 3 . Additionally, because this survey recruited career scientists from a panel cohort, junior scientists and trainees (who may do the bulk of programming work) were not targeted. A 2025 survey of researchers at one large institute reported that 43.2% of respondents used AI to “help write code” , but programming was not specifically addressed further 7 . This study reported a