Architecture, Not Scale: Circuit Localization in Large Language Models
Sohan Venkatesh · 2026
Read Architecture, Not Scale: Circuit Localization in Large Language Models by Sohan Venkatesh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08853.txt.
Description
Sohan Venkatesh says a model's design for focus, not its size, sets how easy its circuits are to find. Work on model guts tends to assume big means hard. Venkatesh checks three circuit types in two model lines. Grouped query focus gives far tighter, more stable circuits than the plain multi head kind. The scale is much the same. The edge holds for object picks, for copy heads, and for fact recall.
In one line, fact recall circuits snap at a key scale. They crash to one tight neck, not a slow fade. Design picks can thus make large models more open to study. How hard a model is to read is a design trait, not a size fate.
Catalog record
| CREATOR | Sohan Venkatesh |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08853