To See the Unseen: on the Generalization Ability of Transformers in Symbolic Reasoning
Nevena Lazić, Liam Fowl, András György, Csaba Szepesvári · 2026
Read To See the Unseen: on the Generalization Ability of Transformers in Symbolic Reasoning by Nevena Lazić, Liam Fowl, András György, Csaba Szepesvári (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21632.txt.
Description
Nevena Lazić and three colleagues study transformers for in-context propositional logic reasoning. Prior work showed models fail on unseen names, partly from copying trouble. Collapse also matters: unseen-token unembeddings collapse to nearly one vector, making unseen variables hard to tell apart when embeddings share settings. This explains active forgetting, which resets embeddings over time. They combine copying tweaks, varied data, and embedding resets, generalizing to unseen tokens. Tests support the claims, and Gemma 3 models show embedding collapse, including 99 unused tokens starting finetuning poorly.
Catalog record
| CREATOR | Nevena Lazić, Liam Fowl, András György, Csaba Szepesvári |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45338 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.21632