Emergent Stack Representations in Modeling Counter Languages Using Transformers
Utkarsh Tiwari, Aviral Gupta, Michael Hahn · 2025
Read Emergent Stack Representations in Modeling Counter Languages Using Transformers by Utkarsh Tiwari, Aviral Gupta, Michael Hahn (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-01432.txt.
Description
Utkarsh Tiwari, Aviral Gupta, and Michael Hahn probe in 2025 what transformer models learn inside when trained on counter languages, formal tongues whose grammar needs a running count. Such toy tasks expose how the models compute. The team trained models on four counter languages, then restated those languages with stacks, where stack depth plays the role of the count.
By probing the models' inner states at each word, they found stack like codes for depth. Models trained just to predict the next word had built a data structure on their own. The result brings circuit hunting in these models a step closer, showing that neat algorithmic tricks can emerge from plain training.
Catalog record
| CREATOR | Utkarsh Tiwari, Aviral Gupta, Michael Hahn |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-01432