Position-aware Automatic Circuit Discovery
Tal Haklay, Hadas Orgad, David Bau, Aaron Mueller, Yonatan Belinkov · 2025
Read Position-aware Automatic Circuit Discovery by Tal Haklay, Hadas Orgad, David Bau, Aaron Mueller, Yonatan Belinkov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-04577.txt.
Description
To understand how a language model performs a task, researchers carve out the circuit: the smallest part of its internal wiring that does the job. Tal Haklay and co-authors found a blind spot in the standard carving tools. They assume a component matters equally wherever it sits in the input, so mechanisms that depend on position, or span across positions, slip past them.
The team fixed both gaps. They upgraded a gradient-based discovery method so it scores components separately at each token position. They also introduced dataset schemas, templates that mark which spans of text play the same role across examples of different lengths. A language model pipeline writes the schemas automatically. The result is fully automated discovery of position-sensitive circuits that stay smaller while remaining more faithful to the model's real behavior.
Catalog record
| CREATOR | Tal Haklay, Hadas Orgad, David Bau, Aaron Mueller, Yonatan Belinkov |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-04577