Gaussian Match-and-Copy A Minimalist Benchmark for Studying Transformer Induction

Antoine Gonon, Alexandre Cordonnier, Nicolas Boumal, Institute of Mathematics, EPFL, Switzerland · 2026

Read Gaussian Match-and-Copy A Minimalist Benchmark for Studying Transformer Induction by Antoine Gonon, Alexandre Cordonnier, Nicolas Boumal, Institute of Mathematics, EPFL, Switzerland (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-07562.txt.

Description

Language models retrieve a matching token from context and copy what follows, but on natural data retrieval and memorization are tangled. Antoine Gonon and two co-authors isolate the retrieval skill with Gaussian Match-and-Copy, a minimalist benchmark built on pure second-order correlation signals.

Numerical studies show the task preserves key features of how transformers develop match-and-copy circuits, separates architectures by retrieval ability. Reveals an implicit-bias regime where gradient descent drives weights toward retrieval solutions in a simplified attention setting. Interpretability teams get a clean laboratory for a core inference primitive.

Catalog record

CREATORAntoine Gonon, Alexandre Cordonnier, Nicolas Boumal, Institute of Mathematics, EPFL, Switzerland
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE144 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-07562

Browse the full library · About the Tower of Babel