Transformers converge to invariant algorithmic cores

Joshua S. Schiffman · 2026

Read Transformers converge to invariant algorithmic cores by Joshua S. Schiffman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-22600.txt.

Description

Joshua S. Schiffman argues that training selects what a model does, not its wiring, so any single trained transformer may show quirks of one run rather than the task itself. His remedy is Algorithmic Core Extraction, a method that isolates compact subspaces needed for a task and recurring across models trained apart.

Markov-chain transformers hid three-dimensional cores in near-orthogonal subspaces yet got back the same transition spectra. Modular-addition models built compact cyclic cores at grokking that later swelled under regularization, spreading the same work widely, which sped the leap from rote memory to broad use. Across six large models, subject-verb agreement rode a single steerable axis, and flipping it swapped singular and plural in open-ended text.

Catalog record

CREATORJoshua S. Schiffman
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-22600

Browse the full library · About the Tower of Babel