Learned or Memorized ? Quantifying Memorization Advantage in Code LLMs Conference: 2026 IEEE/ACM 48th International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE ’26), April 12–18, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3744916.3764554 ISBN: 979-8-4007-2025-3/26/04 CCS: General and reference Empirical studies CCS: General and reference Evaluation CCS: Computing methodologies Natural language generation CCS: General and reference Performance

Djiré Albérick Euraste, Kaboré Abdoul Kader, Jordan Samhi, Earl T. Barr, Jacques Klein, Tegawendé F. Bissyandé · 2026

Read Learned or Memorized ? Quantifying Memorization Advantage in Code LLMs Conference: 2026 IEEE/ACM 48th International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE ’26), April 12–18, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3744916.3764554 ISBN: 979-8-4007-2025-3/26/04 CCS: General and reference Empirical studies CCS: General and reference Evaluation CCS: Computing methodologies Natural language generation CCS: General and reference Performance by Djiré Albérick Euraste, Kaboré Abdoul Kader, Jordan Samhi, Earl T. Barr, Jacques Klein, Tegawendé F. Bissyandé (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13997.txt.

Description

Euraste and colleagues measure how much code language models lean on memorization rather than learning. Their perturbation method scores the gap between likely seen and unseen inputs across 8 open-source models and 19 benchmarks. StarCoder shows sensitivity up to 0.8 on some benchmarks, while QwenCoder stays lower, and suspected-leak benchmarks CVEFixes and Defects4J show low advantage. Memorization risk varies sharply by task and model.

Catalog record

CREATORDjiré Albérick Euraste, Kaboré Abdoul Kader, Jordan Samhi, Earl T. Barr, Jacques Klein, Tegawendé F. Bissyandé
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60105
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.13997

Browse the full library · About the Tower of Babel