Learned or Memorized ? Quantifying Memorization Advantage in Code LLMs Conference: 2026 IEEE/ACM 48th International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE ’26), April 12–18, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3744916.3764554 ISBN: 979-8-4007-2025-3/26/04 CCS: General and reference Empirical studies CCS: General and reference Evaluation CCS: Computing methodologies Natural language generation CCS: General and reference Performance
Djiré Albérick Euraste, Kaboré Abdoul Kader, Jordan Samhi, Earl T. Barr, Jacques Klein, Tegawendé F. Bissyandé · 2026
Read Learned or Memorized ? Quantifying Memorization Advantage in Code LLMs Conference: 2026 IEEE/ACM 48th International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE ’26), April 12–18, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3744916.3764554 ISBN: 979-8-4007-2025-3/26/04 CCS: General and reference Empirical studies CCS: General and reference Evaluation CCS: Computing methodologies Natural language generation CCS: General and reference Performance by Djiré Albérick Euraste, Kaboré Abdoul Kader, Jordan Samhi, Earl T. Barr, Jacques Klein, Tegawendé F. Bissyandé (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13997.txt.
Description
Euraste and colleagues measure how much code language models lean on memorization rather than learning. Their perturbation method scores the gap between likely seen and unseen inputs across 8 open-source models and 19 benchmarks. StarCoder shows sensitivity up to 0.8 on some benchmarks, while QwenCoder stays lower, and suspected-leak benchmarks CVEFixes and Defects4J show low advantage. Memorization risk varies sharply by task and model.
Catalog record
| CREATOR | Djiré Albérick Euraste, Kaboré Abdoul Kader, Jordan Samhi, Earl T. Barr, Jacques Klein, Tegawendé F. Bissyandé |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60105 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.13997