Sense and Sensitivity: Examining the Influence of Semantic Recall on Long Context Code Understanding
Adam Štorek, Mukur Gupta, Samira Hajizadeh, Prashast Srivastava, Suman Jana · 2025
Read Sense and Sensitivity: Examining the Influence of Semantic Recall on Long Context Code Understanding by Adam Štorek, Mukur Gupta, Samira Hajizadeh, Prashast Srivastava, Suman Jana (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13353.txt.
Introduction
Adam Štorek and co-authors probe, in a 2025 paper, whether language models truly grasp what long code does, or merely match its surface patterns. Developers increasingly hand whole codebases to these models. Verbatim recall and real understanding are differing skills.
Ten leading models were evaluated. Word-perfect recall stayed near flawless regardless of position in the file. Grasp of operational meaning, though, collapsed when the key code sat mid-context. The authors built a counterfactual test that separates the two skills. Results show heavy reliance on shortcuts that pattern matching affords. Benchmarks that reward recitation overstate readiness for the maintenance work programmers actually delegate.
Catalog record
| CREATOR | Adam Štorek, Mukur Gupta, Samira Hajizadeh, Prashast Srivastava, Suman Jana |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-13353
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sense and Sensitivity: Examining the Influence of Semantic Recall on Long Context Code Understanding Adam Štorek, Mukur Gupta, Samira Hajizadeh, Prashast Srivastava, Suman Jana Abstract Large language models (LLMs) are increasingly deployed for understanding large codebases, but whether they understand operational semantics of long code context or rely on pattern matching shortcuts remains unclear. We distinguish between lexical recall (retrieving code verbatim) and semantic recall (understanding operational semantics). Evaluating 10 state-of-the-art LLMs, we find that while frontier models achieve near-perfect, position-independent lexical recall, semantic recall degrades severely when code is centrally positioned in long contexts. We introduce semantic recall sensitivity to measure whether tasks require understanding of code’s operational semantics vs. permit pattern matching shortcuts. Through a novel counterfactual measurement method, we show that models rely heavily on pattern matching shortcuts to solve existing code understanding benchmarks. We propose a new task SemTrace , which achieves high semantic recall sensitivity through unpredictable operations; LLMs’ accuracy exhibits severe positional effects, with median accuracy drops of 92.73% versus CRUXEval ’s 53.36% as the relevant code snippet approaches the middle of the input code context. Our findings suggest current evaluations substantially underestimate semantic recall failures in long context code understanding. 1 1 1 Our code is available at https://github.com/adamstorek/long-context-code-understanding . 1 Introduction Large Language Models (LLMs) are increasingly applied to industry coding tasks ( Edwards, 2024 ) that demand understanding of large codebases ( Jimenez et al., 2024 ) . Recent advances ( Dao et al., 2022 ; Peng et al., 2024 ; Su et al., 2024 ) enable these models to process extremely long inputs, up to millions of tokens ( OpenAI, 2025 ) . However, a fundamental question remains unanswered: when models solve code understanding tasks, are they processing the specific code provided in context, or applying memorized patterns from pretraining? This distinction becomes critical as LLMs are deployed in production environments where they must handle novel, project-specific code that cannot be solved through pattern matching alone. Pattern matching shortcuts may also result in LLMs missing subtle vulnerabilities ( Ding et al., 2025 ) . We introduce a key distinction between two capabilities for code understanding in long contexts: lexical recall , meaning the ability to locate and reproduce code verbatim, and semantic recall , meaning the ability to remember what code does when it is run, i.e., its operational semantics ( Winskel, 1993 ) . These capabilities are distinct; models can have perfect lexical recall yet fail at semantic recall, demonstrating they can access relevant code but not understand its effects. While needle-in-the-haystack (NIAH) benchmarks ( Liu et al., 2024a ; Liu et al., 2024c ) measure lexical recall, the relationship between lexical and semantic recall is not well understood. Moreover, code understanding tasks like output prediction are designed to measure semantic recall, but can often be solved through pattern-matching shortcuts (recognizing familiar algorithms, applying memorized correlations) without requiring semantic recall of the specific implementation. This conflation poses a fundamental evaluation challenge: existing benchmarks may allow shortcuts that mask semantic recall failures. We introduce semantic recall sensitivity as a property of tasks: the degree to which solving the task requires semantically recalling specific code details rather than pattern matching. To investigate whether lexical and semantic recall rely on different mechanisms, we leverage positional variation in long contexts as a diagnostic lens for how models integrate information, rather than as an end in itself. Positional variation naturally arises as code length scales, and understanding of code should remain stable regardless of where relevant details occur. Prior work has shown that LLM performance varies based on where information appears in the input ( Liu et al., 2024b ; Lu et al., 2022 ) , but here we treat position as a controlled perturbation to probe representational differences between lexical and semantic recall. Our evaluation across 10 state-of-the-art LLMs reveals a clear dissociation: frontier models achieve near-perfect, position-independent lexical recall of relevant code, yet semantic recall on the same code degrades when centrally positioned within the code context. On CRUXEval ( Gu et al., 2024 ) , a popular input-output prediction benchmark, semantic recall shows moderate position-dependent degradation (median accuracy drop of 53.36%) compared to negligible lexical recall degradation (2.39%). However, we hypothesize this substantially underestimates semantic recall fragility due to low semantic recall sensitivity, as CRUXEval permits pattern matching that compensates for semantic recall failures. To test this, we propose a novel counterfactual measurement method: systematically removing lines from code and measuring the LLMs’ performance degradation. Code that requires semantic recall should cause the LLMs’ performance to fall sharply (like the Python interpreter), since the code’s operational semantics eventually changes; LLMs using pattern matching degrade gradually. CRUXEval shows gradual degradation (only 44.15-59.74% accuracy loss at 50% line removal), confirming low sensitivity. To isolate semantic recall, we introduce SemTrace , an output prediction task that achieves high semantic recall sensitivity through unpredictable operations. SemTrace reveals dramatically more severe degradation: median accuracy drops of 92.73% versus CRUXEval ’s 53.36%, with some frontier models reaching zero accuracy when code is centrally positioned. Even GPT-4.1 shows clear position-dependence when scaled to higher-digit arithmetic that cannot be memorized, confirming that moderate degradation on existing benchmarks masks severe semantic recall fragility. Contributions. (1) We introduce the distinction between lexical recall (accessing code verbatim) and semantic recall (understanding operational semantics), demonstrating they dissociate in long contexts: models maintain near-perfect, position-independent lexical recall while semantic recall exhibits severe position-dependent failures. We validate this across multiple programming languages (Python, JavaScript, PHP). (2) We propose semantic recall sensitivity as a task property with a counterfactual measurement method, revealing existing benchmarks like CRUXEval have low sensitivity, allowing pattern matching to mask failures. (3) We introduce SemTrace , an output prediction task isolating semantic recall via unpredictable operations, revealing 92.73% median accuracy degradation vs. 53.36% on CRUXEval , demonstrating current evaluations substantially underestimate semantic recall challenges. 2 Related Work Our work intersects a number of research areas, namely long context evaluation, positional effects in long contexts, and memorization. 2.1 Long Context Evaluation Significant effort has been devoted to long-context evaluation, expanding from needle-in-haystack retrieval ( Mohtashami and Jaggi, 2023 ; Shaham et al., 2023 ; Kamradt, 2023 ) to comprehensive benchmarks ( Bai et al., 2024 ; Bai et al., 2025 ) , extended context lengths ( Zhang et al., 2024a ) , and complex reasoning tasks ( Kuratov et al., 2024 ; Hsieh et al., 2024 ; Levy et al., 2024 ; Modarressi et al., 2025 ) . For code specifically, prior work has focused on repository-level retrieval and code completion ( Guo et al., 2023 ; Liu et al., 2024c ) , semantic search ( Liu et al., 2024a ) , and code repository QA ( Bai et al., 2025 ) . However, these tasks primarily evaluate code retrieval (locating relevant code given queries) rather than understanding operational semantics (predicting execution behavior or input-output mappings). While prior work evaluates retrieval and understanding as separate tasks, we test both capabilities on identical code in long contexts to understand their relationship. We also introduce semantic recall sensitivity as a property for characterizing code understanding benchmarks. 2.2 Position Effects in Long Context Prior work has identified positional biases in LLMs, ranging from long context scenarios