Architecture, Not Scale: Circuit Localization in Large Language Models

Sohan Venkatesh · 2026

Read Architecture, Not Scale: Circuit Localization in Large Language Models by Sohan Venkatesh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08853.txt.

Description

Sohan Venkatesh says a model's design for focus, not its size, sets how easy its circuits are to find. Work on model guts tends to assume big means hard. Venkatesh checks three circuit types in two model lines. Grouped query focus gives far tighter, more stable circuits than the plain multi head kind. The scale is much the same. The edge holds for object picks, for copy heads, and for fact recall.

In one line, fact recall circuits snap at a key scale. They crash to one tight neck, not a slow fade. Design picks can thus make large models more open to study. How hard a model is to read is a design trait, not a size fate.

Catalog record

CREATORSohan Venkatesh
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08853

Browse the full library · About the Tower of Babel