Directional Routing in Transformers

Kevin Taylor · 2026

Read Directional Routing in Transformers by Kevin Taylor (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14923.txt.

Description

Taylor adds directional routing to transformer attention heads at small parameter cost. A shared router steers learned suppression directions for each head. A mid-size model trains beside an identical baseline in one run. Routing becomes the dominant compute path in the routed model. Killing it drops factual recall near zero on eight test prompts. Induction accuracy falls from over 93 percent to zero as well. Single head knockouts barely matter by contrast. Perplexity drops by about a third to a half versus baseline, Taylor reports.

Catalog record

CREATORKevin Taylor
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29149
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.14923

Browse the full library · About the Tower of Babel