Are Large Language Models Truly Smarter Than Humans? Benchmark Contamination, Surface-Pattern Reliance, and Behavioral Memorization Across Six Frontier Models

Eshwar Reddy M Applied AI Scientist, Health Vectors [email protected] Sourav Karmakar Senior AI Scientist, Intuit India [email protected] · 2026

Read Are Large Language Models Truly Smarter Than Humans? Benchmark Contamination, Surface-Pattern Reliance, and Behavioral Memorization Across Six Frontier Models by Eshwar Reddy M Applied AI Scientist, Health Vectors [email protected] Sourav Karmakar Senior AI Scientist, Intuit India [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16197.txt.

Introduction

Reddy and Karmakar audit six frontier language models for benchmark contamination. Public test sets are mirrored online, so models may train on their own exam. The team runs three checks on MMLU questions. One check found a 13.8% contamination rate across 513 questions. Reworded questions cut accuracy by 7 points on average. Guessing probes flagged memorization signals in 72.5% of cases. All three checks rank STEM fields as most contaminated.

Catalog record

CREATOREshwar Reddy M Applied AI Scientist, Health Vectors [email protected] Sourav Karmakar Senior AI Scientist, Intuit India [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35190
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.16197

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Are Large Language Models Truly Smarter Than Humans? Benchmark Contamination, Surface-Pattern Reliance, and Behavioral Memorization Across Six Frontier Models Eshwar Reddy M Applied AI Scientist, Health Vectors [email protected] Sourav Karmakar Senior AI Scientist, Intuit India [email protected] Abstract Public leaderboards increasingly suggest that large language models (LLMs) surpass human experts on benchmarks spanning academic knowledge, law, and programming. Yet most benchmarks are fully public, their questions widely mirrored across the internet, creating systematic risk that models were trained on the very data used to evaluate them. This paper presents three original, complementary experiments that together form a rigorous multi-method contamination audit of six frontier LLMs—GPT-4o, GPT-4o-mini, DeepSeek-R1, DeepSeek-V3, Llama-3.3-70B, and Qwen3-235B—conducted entirely using public APIs and open benchmarks. Experiment 1 applies a lexical contamination detection pipeline to 513 sampled MMLU test questions across all 57 subject categories, finding an overall contamination rate of 13.8% (18.1% in STEM, up to 66.7% in Philosophy) and estimated performance gains (EPG) of + + 0.030– + + 0.054 accuracy points by category. Experiment 2 applies a paraphrase and indirect-reference diagnostic to 100 MMLU questions across six subjects and all six models, finding that model accuracy drops by an average of 7.0 percentage points when surface wording changes to indirect reference—rising to 19.8 pp in both Law and Ethics, precisely the domains most heavily contaminated in Experiment 1. Experiment 3 applies TS-Guessing behavioral probes (Option Mask and Word Mask) to all 513 sampled questions and all six models, finding that 72.5% of questions trigger memorization signals far above chance baselines, with DeepSeek-R1 displaying an anomalous distributed memorization signature (76.6% partial reconstruction, 0% verbatim recall) that directly explains its uniquely brittle accuracy pattern in Experiment 2. Across all three experiments, STEM is consistently the most contaminated category and the methods converge on the same subject-category ranking, providing independent convergent evidence that contamination is pervasive, structurally non-uniform, and not adequately captured by any single detection approach alone. 1 Introduction Public benchmarks and leaderboards have become the primary currency of progress in large language models. New systems are announced with headline claims such as “model X outperforms most humans on the bar exam” or “model Y surpasses human experts on MMLU,” shaping perceptions among researchers, policymakers, investors, and the broader public [ 9 ] . These claims typically rest on performance on static, open-access benchmarks whose complete test sets—including question–answer pairs and often detailed solutions—have been available online for years. As training corpora for LLMs have expanded to include vast slices of the internet, it has become increasingly plausible that benchmark items, or close variants of them, are present in pretraining data. If the model has effectively “seen the exam” during training, a high score does not demonstrate the same kind of general intelligence that a human’s first-time performance would. Instead, it reflects a mixture of memorization, pattern completion, and interpolation within a highly familiar distribution. This paper addresses the central question through three original, mutually reinforcing experiments on six frontier models, using only public APIs and open benchmarks, making the entire evaluation pipeline independently reproducible. Contributions • Experiment 1 (Section 4 ): A full lexical contamination scan of MMLU across all 57 subjects (513 questions), yielding an overall contamination rate of 13.8% and subject-level rates as high as 66.7% (Philosophy). Estimated performance gains from contamination range from + 0.030 to + 0.054 accuracy points by category. • Experiment 2 (Section 5 ): A paraphrase and indirect-reference diagnostic on 100 MMLU questions across six subjects and all six models, quantifying accuracy degradation under surface-form change. Average drop of 7.0 pp on indirect-reference variants across all models, with Law ( - 19.8 pp) and Ethics ( - 19.8 pp) showing the largest degradation—directly matching their high Experiment 1 contamination rates. • Experiment 3 (Section 6 ): A TS-Guessing behavioral contamination probe on all 513 sampled questions and all six models, finding a 72.5% average combined flagging rate and revealing a distinctive DeepSeek-R1 distributed memorization signature (76.6% partial reconstruction, 0% exact recall) that explains its anomalous Experiment 2 profile. • A structured synthesis showing that all three methods converge on the same category-level contamination ranking, providing multi-method independent evidence for the pervasiveness and structure of LLM benchmark contamination. 2 Background: Benchmarks, Contamination, and Intelligence 2.1 Benchmarks and evaluation culture Modern LLM benchmarks cover a wide spectrum of tasks. Massive Multitask Language Understanding (MMLU) [ 9 ] aggregates thousands of multiple-choice questions across dozens of academic and professional subjects. Code benchmarks such as HumanEval [ 10 ] present short programming problems with unit tests, while TruthfulQA [ 11 ] probes susceptibility to common misconceptions. Many commercial and open-source model releases report performance on such benchmarks alongside comparisons to human baselines, often suggesting the model matches or exceeds average college graduates or domain experts. Leaderboards amplify this culture, incentivizing score improvements over methodological transparency. 2.2 Mechanisms of evaluation data contamination Evaluation data contamination occurs when benchmark examples—or their paraphrases, translations, or derivative discussions—appear in training data. Common mechanisms include: (a) direct inclusion of benchmark repository files scraped into pretraining corpora; (b) indirect inclusion via blog posts, teaching materials, and solution write-ups that preserve problem structure; (c) fine-tuning contamination through instruction-tuning datasets that incorporate benchmark-like Q&A; and (d) deliberate benchmaxxing , where practitioners intentionally include benchmark items to maximize leaderboard scores. Critically, contamination is not limited to exact string matches. Even when n-gram deduplication is applied, models may be exposed to paraphrased or near-duplicate versions that preserve underlying problem structure while modifying surface form—a key motivation for the behavioral detection methods in Experiments 2 and 3. 2.3 Intelligence, generalization, and memorization In cognitive science and machine learning, intelligence is linked to generalization: applying finite learned experience to structurally novel situations. Memorization can produce impressive interpolation on similar tasks, but differs from the robust, causal reasoning that characterizes expert human performance in law, medicine, and science [ 8 ] . Evaluations that fail to separate memorization from generalization risk overstating what these systems can safely do in deployment. 3 Prior Empirical Work on Contamination 3.1 Retrieval-based detection One line of work investigates contamination using retrieval-based and behavioral methods [ 1 ] . For open models, search pipelines find benchmark items in training data. For closed-source models, behavioral Testset Slot Guessing (TS-Guessing) protocols mask elements of benchmark questions and check whether models reconstruct them—the basis for our Experiment 3. Empirical results show that leading commercial models reconstruct missing MMLU options at rates far above chance, indicating prior exposure. 3.2 Output-distribution methods A second line detects contamination from model outputs alone [ 2 ] . Contaminated items produce “peaked” output distributions: a memorizing model assigns very high probability to the correct continuation. Contamination Detection via output Distribution (CDD) uses this signal; the Memorization Generalization Index (MGI) separates memorization-driven from generalization-driven performance. 3.3 Impact metrics and contamination-free benchmarks The Contamination Threshold Analysis Method (ConTAM) [ 3 ] introduces the Estimated Performance Gain (EPG) metric, applied directly in Experiment 1. MMLU-CF [ 5 ] reconstructs MMLU with strict decontamination; absolute scores drop and model rankings change. MMLU-Pro [ 4 ] adds reasoning-intensive questions and similarly reduces reported scores. These results motivate our Experiment 2 paraphrase diagnostic: if performance is inflated by surface familiarity, accuracy should degrade as surface form

Read the complete text