How do Humans and LLMs Process Confusing Code? Conference: ; April 12–18, 2026; Rio De Janeiro, Brazil CCS: General and reference Empirical studies CCS: General and reference Metrics CCS: Computing methodologies Natural language generation
Youssef Abdelsalam, Norman Peitek, Anna-Maria Maurer, Mariya Toneva, Sven Apel · 2025
Read How do Humans and LLMs Process Confusing Code? Conference: ; April 12–18, 2026; Rio De Janeiro, Brazil CCS: General and reference Empirical studies CCS: General and reference Metrics CCS: Computing methodologies Natural language generation by Youssef Abdelsalam, Norman Peitek, Anna-Maria Maurer, Mariya Toneva, Sven Apel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-18547.txt.
Introduction
Youssef Abdelsalam and co-authors present a study from two thousand twenty-five asking whether humans and language models stumble over the same confusing code. Programmers now work daily with AI assistants, so a mismatch in what confuses each side could cause bugs and wasted effort. The team set out to compare the two directly.
They measured model confusion by perplexity, the model's surprise at the code. Human confusion was read from brain signals tied to eye movements while programmers read clean and confusing code. The model's surprise spikes matched the human brain responses in both place and size. That likeness suggests both sides trip on the same spots. The authors then built a tool that flags code regions likely to confuse people.
Catalog record
| CREATOR | Youssef Abdelsalam, Norman Peitek, Anna-Maria Maurer, Mariya Toneva, Sven Apel |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-18547
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How do Humans and LLMs Process Confusing Code? Conference: ; April 12–18, 2026; Rio De Janeiro, Brazil CCS: General and reference Empirical studies CCS: General and reference Metrics CCS: Computing methodologies Natural language generation Youssef Abdelsalam, Norman Peitek, Anna-Maria Maurer, Mariya Toneva, Sven Apel Abstract . Already today, humans and programming assistants based on large language models (LLMs) collaborate in everyday programming tasks. Clearly, a misalignment between how LLMs and programmers comprehend code can lead to misunderstandings, inefficiencies, low code quality, and bugs. A key question in this space is whether humans and LLMs are confused by the same kind of code. This would not only guide our choices of integrating LLMs in software engineering workflows, but also inform about possible improvements of LLMs. To this end, we conducted an empirical study comparing an LLM to human programmers comprehending clean and confusing code. We operationalized comprehension for the LLM by using LLM perplexity , and for human programmers using neurophysiological responses (in particular, EEG-based fixation-related potentials). We found that LLM perplexity spikes correlate both in terms of location and amplitude with human neurophysiological responses that indicate confusion. This result suggests that LLMs and humans are similarly confused about the code. Based on these findings, we devised a data-driven, LLM-based approach to identify regions of confusion in code that elicit confusion in human programmers. 1. Introduction Large language models (LLMs) have rapidly advanced and demonstrated remarkable performance in many programming-related tasks ( Hou et al. 2024 ) , including code completion ( Husein et al. 2025 ; Izadi et al. 2022 ) , synthesis ( Nguyen and Nadi 2022 ) , and repair ( Hossain et al. 2024 ; Zubair et al. 2025 ) . Trained on massive corpora of data, LLMs are able to generate syntactically correct and semantically plausible code across all areas of software engineering, from education ( Raihan et al. 2025 ; Rasnayaka et al. 2024 ; Hellas et al. 2023 ; Kazemitabaar et al. 2024 ) and research ( de Martino et al. 2024 ; Felizardo et al. 2024 ; Steinmacher et al. 2024 ) to practice ( Khojah et al. 2024 ; Hao et al. 2024 ) . Already today, LLM-based coding assistants and human programmers work in concert to produce substantial amounts of source code ( Daniotti et al. 2025 ) . Clearly, a misalignment between LLM and human programmers in such a collaborative setting can lead to misunderstandings, inefficiencies, low code quality, and bugs ( Ziegler et al. 2022 ; Vaithilingam et al. 2022 ; Siddiq et al. 2024 ) . To maximize the potential of LLMs, we are well advised to critically assess the alignment ( Moussa et al. 2023 ; AlKhamissi et al. 2025 ; Sergeyuk et al. 2024 ; Barke et al. 2023 ) of how LLMs and humans comprehend source code. Figure 1. A comparison of confusing and corresponding clean code (a) as well as effects on LLM perplexity (b) and neurophysiological responses (c): The confusing code snippet includes an atom of confusion (highlighted in red). The corresponding clean snippet avoids this atom by extracting the decrement-expression into a separate statement (a). The token-level perplexity from an LLM increases for the confusing code region (b). Corresponding EEG fixation-related potentials exhibit increased amplitude during the comprehension of the confusing snippet (c). The premise of our investigation is that understanding (the presence and extent of) this alignment would allow us to better understand possible risks and pitfalls of integrating LLMs into software engineering workflows and to improve the capabilities of LLMs in collaborating with humans in programming-related tasks. Specifically, we are interested in the reaction of LLMs and humans to confusing code. Prior research has dedicated considerable attention to problematic or confusing code ( Dijkstra 1968 ; Gopstein et al. 2017 ) , even offering validated stimuli that elicit distinct neurophysiological responses in humans ( Yeh et al. 2017 ; Yeh et al. 2022 ; Bergum et al. 2025 ) . Thus, confusing code provides an empirically grounded and well-studied setting to evaluate the alignment between human and LLM comprehension. Research questions and methodology. Our investigation is guided by three research questions: RQ 1 : How does confusing code affect LLM confusion? RQ 2 : How does LLM confusion about code relate to human confusion about the same code? RQ 3 : How well is a data-driven, LLM-based approach able to identify code regions that confuse humans? For our investigation, we rely on a well-studied set of syntactically valid and semantically correct code idioms that are known to confuse human programmers, called atoms of confusion ( Gopstein et al. 2017 ) . Figure 1 (a) displays an example pair of an atom of confusion in a confusing and a corresponding clean (i.e., less confusing) code snippet. Atoms of confusion lead to longer reading times, higher error rates, and lower comprehension accuracy across programming languages ( Langhout and Aniche 2021 ; da Costa et al. 2023 ; Gopstein et al. 2017 ) . Studies involving visual attention ( de Oliveira et al. 2020 ; da Costa et al. 2023 ) and neurophysiological measures using electroencephalography (EEG) ( Yeh et al. 2017 ; Yeh et al. 2022 ) further corroborate these results. To analyze how humans comprehend code, we rely on EEG data from prior work. EEG has been a well-established method for measuring programmers’ neurophysiological responses to source code ( Yeh et al. 2017 ; Peitek et al. 2022 ; Ishida and Uwano 2019 ; Lee et al. 2016 ; Crk and Kluthe 2014 ; Medeiros et al. 2021 ; Lin et al. 2021 ) . In a study on atoms of confusion, Bergum et al. 2025 combined EEG with eye tracking to measure fixation-related potentials (FRPs). They found that confusing code reliably elicits a so-called late frontal positivity in the EEG signal of human programmers (see Figure 1 (c)), which is a neurophysiological correlate of human confusion ( Kuperberg et al. 2020 ) . Therefore, we also measure human confusion using fixation-related potentials with the focus on the amplitude of late frontal positivity. To analyze how LLMs comprehend code, we use the measure of perplexity . Perplexity reflects how unexpected a token is to the LLM given its context (see Figure 1 (b)). In natural language processing and psycholinguistics, the unexpectedness of natural language text serves as a well-established proxy for human confusion and cognitive load ( Levy 2008 ) , which has been confirmed also for source code ( Hindle et al. 2012 ; Casalnuovo et al. 2020 ) . Combining the pieces, we have devised a methodology that: (1) computes token- and snippet-level perplexity values with state-of-the-art LLMs, (2) extends this analysis to calculate perplexity at predefined areas of interest (AOIs), (3) aligns these with fixation-related potentials in EEG signals, and (4) detects perplexity peaks within code snippets to define new AOIs, extract corresponding FRPs, and align these measures accordingly. We leverage the dataset of Bergum et al. 2025 ( Bergum et al. 2025 ) for fixation-related potentials and the state-of-the-art LLM Qwen2.5-Coder-32B ( Hui et al. 2024 ) for perplexity measurement. Results. Our results demonstrate that the presence of atoms of confusion consistently triggers a spike in LLM