Position-aware Automatic Circuit Discovery

Tal Haklay, Hadas Orgad, David Bau, Aaron Mueller, Yonatan Belinkov · 2025

Read Position-aware Automatic Circuit Discovery by Tal Haklay, Hadas Orgad, David Bau, Aaron Mueller, Yonatan Belinkov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-04577.txt.

Description

To understand how a language model performs a task, researchers carve out the circuit: the smallest part of its internal wiring that does the job. Tal Haklay and co-authors found a blind spot in the standard carving tools. They assume a component matters equally wherever it sits in the input, so mechanisms that depend on position, or span across positions, slip past them.

The team fixed both gaps. They upgraded a gradient-based discovery method so it scores components separately at each token position. They also introduced dataset schemas, templates that mark which spans of text play the same role across examples of different lengths. A language model pipeline writes the schemas automatically. The result is fully automated discovery of position-sensitive circuits that stay smaller while remaining more faithful to the model's real behavior.

Catalog record

CREATORTal Haklay, Hadas Orgad, David Bau, Aaron Mueller, Yonatan Belinkov
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-04577

Browse the full library · About the Tower of Babel