Non-programmers Assessing AI-Generated Code: A Case Study of Business Users Analyzing Data

Yuvraj Virk, Dongyu Liu · 2025

Read Non-programmers Assessing AI-Generated Code: A Case Study of Business Users Analyzing Data by Yuvraj Virk, Dongyu Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06484.txt.

Introduction

Yuvraj Virk and Dongyu Liu present a 2025 case study of business users judging AI written data analysis. Staff with no coding background now lean on chatbots to run numbers for real choices. The models err often, so the human check is the last guard.

Marketing and sales staff were shown plain word walk throughs of AI code, told the AI errs often, and asked to hunt the flaws. Many still missed faults that could warp a choice, even ones needing no tech skill to spot. A retry split the answer into clear steps and gave a rival path for each call. Scores rose somewhat, but staff still strained to weigh the steps. The authors warn that firms cannot count on this check as it stands.

Catalog record

CREATORYuvraj Virk, Dongyu Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-06484

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Non-programmers Assessing AI-Generated Code: A Case Study of Business Users Analyzing Data Yuvraj Virk, Dongyu Liu Abstract Non-technical end-users increasingly rely on AI code generation to perform technical tasks like data analysis. However, large language models (LLMs) remain unreliable, and it is unclear whether end-users can effectively identify model errors — especially in realistic and domain-specific scenarios. We surveyed marketing and sales professionals to assess their ability to critically evaluate LLM-generated analyses of marketing data. Participants were shown natural language explanations of the AI’s code, repeatedly informed the AI often makes mistakes, and explicitly prompted to identify them. Yet, participants frequently failed to detect critical flaws that could compromise decision-making, many of which required no technical knowledge to recognize. To investigate why, we reformatted AI responses into clearly delineated steps and provided alternative approaches for each decision to support critical evaluation. While these changes had a positive effect, participants often struggled to reason through the AI’s steps and alternatives. Our findings suggest that business professionals cannot reliably verify AI-generated data analyses on their own and explore reasons why to inform future designs. As non-programmers adopt code-generating AI for technical tasks, unreliable AI and insufficient human oversight poses risks of unsafe or low-quality decisions. I Introduction Developers widely use LLMs for code generation. However, LLMs make mistakes that developers must recognize and correct. Increasingly, however, end-user non-programmers also use AI-generated code, interacting through natural language or other intuitive methods. In our study, 18 of 26 marketing/sales professionals reported using tools like ChatGPT to analyze data daily or weekly (which often involves code generation). Prior work explores code generation for non-programmers across domains such as UI generation [ 1 , 2 ] , data analysis [ 3 , 4 , 5 , 6 ] , and robot programming [ 7 , 8 ] . Yet, end-user non-programmers’ current and potential code generation use is concerning: can they be reasonably expected to catch an LLM’s mistakes without inspecting generated code ? Programs encode high-level approaches to achieve a programmer’s goals; non-programmers might identify flaws in high-level approaches, but end-users we consider also lack relevant technical expertise e.g. data science expertise for assessing data science techniques. Our study evaluates their ability to assess AI-generated code through natural language explanations of the AI’s approach and results (Figure 1 ). End-users we consider possess domain expertise and critical thinking ability, which they can use to assess an AI’s technical output — our study investigates to what extent and for realistic, domain-specific tasks. If they cannot reliably catch an AI’s mistakes, and if the AI makes mistakes often enough, then our end-users can not safely complete technical tasks under our setup. While AI code generation holds promise for end-users, it remains unclear how reliable AI must be to ensure safe use or whether they can complement its deficiencies. Fig. 1: The end-user describes a task, the AI generates and runs code, then explains its approach and results. The end-user only views the natural language explanation. To simulate a realistic use case, we focus on marketing and sales professionals analyzing data. These users commonly work with spreadsheets, but advanced analysis is outside their capabilities. Without access to dedicated analysts for day-to-day needs, business professionals resort to ad-hoc analyses [ 9 ] . While access to advanced analysis capabilities is important to business professionals, state-of-the-art LLMs remain unreliable for complex data analysis [ 10 , 11 ] . We first survey professionals to identify the types of flaws they detect in data analysis solutions. A second survey further explores why participants fail at assessing AI outputs. We hypothesize that AI decisions are not salient in outputs so restructured responses into clearly delineated steps. Second, we hypothesize AI decisions lack perceived alternatives so we present two plausible alternatives for each step. Fig. 2: When prompted to select a model for predicting sales, the LLM generates and executes the code on the left. It generates the explanation on the right for participants. The highlighted code contains a hallucinated API name, a mistake non-programmers can not detect. The highlighted text on the right exhibits a flaw they can detect: MSE value is hard to interpret. In summary, our goal is understand whether business end-users can safely use code generating AI for advanced data analysis and discover techniques to improve safety. Our study makes the following contributions: 1. We conduct a survey of skeptical and motivated marketing/sales professionals (n=10) assessing 8 realistic, high-level data analysis tasks. We evaluate their ability to identify issues in the AI’s response requiring domain expertise, technical expertise, or general critical thinking. We find they can identify valid flaws in each category but, overall, cannot be relied upon to detect flaws in data analysis through our natural language explanations. 2. Our second survey (n=18) evaluates the impact of explanation structure and presenting alternative steps on users’ ability to evaluate solution quality. These have a positive but inconsistent effect. Participants have a limited ability to assess alternatives and a low tolerance for deeply engaging with technical approaches. II Related Work Studies of Programmers using Code LLMs Detecting errors in LLM-generated code is difficult for both novices and experts [ 12 , 13 ] . Feldman and Anderson [ 14 ] studied non-expert programmers’ prompting and code-editing behaviors, completing the code evaluation step on their behalf. In contrast, we evaluate non-programmers’ ability to judge natural language explanations of code. Non-expert programmers can effectively use natural language representations to understand and fix AI-generated code, especially when explanations clarify jargon and reveal execution behavior [ 15 , 16 ] . Our study controls explanation quality (see Section III ) to isolate non-programmer evaluation ability — distinct from non-expert programmer abilities [ 14 ] . Other work explores non-programmers using LLMs in domains like robot programming [ 7 ] and spreadsheet programming [ 17 ] . Node-based visual editors and decomposing AI-generated code into helper and summary columns in spreadsheets enhance diagnosis abilities [ 8 , 17 ] . Our findings align with this, suggesting that representations other than natural language may support more reliable evaluations. Unlike these prior works that focus on low-level tasks, we study realistic high-level domain tasks. Our study isolates verification capability and controls for user over-reliance on AI outputs (see Section IV-A ). AI Reliance Explanations, even when incorrect, increase reliance on AI recommendations [ 18 ] . To promote appropriate reliance, prior work proposes interventions such as verification reminders [ 19 ] . Rather than calibrating trust, we investigate whether non-technical users can detect flaws in technical outputs at all. We explicitly prime and incentivize users to distrust the AI. TABLE I: The table lists the task descriptions given to the LLM and participant, as well as an example of an author-identified shortcoming for the task. A flaw is safety critical if it directly causes a poor or unsound decision when gone unnoticed. Task Example of Flaw in AI’s Response Safety Critical Technical Domain Neither 1) Using the given data, add and test new variables that are important in helping predict sales. Does not add variables for lagged effect of marketing efforts X ✓ ✓ X 2) Using the data with newly added factors, select a model that

Read the complete text