Signed-Permutation Coordinate Transport for RMSNorm Transformers

John Sweeney · 2026

Read Signed-Permutation Coordinate Transport for RMSNorm Transformers by John Sweeney (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-31963.txt.

Description

Sweeney shows that moving coordinate-indexed objects between model checkpoints needs the right symmetry group, and that common practice uses the wrong one. Steering vectors, sparse autoencoders and neuron lists all attach meaning to fixed coordinates of a transformer's residual stream. LayerNorm models allow coordinate permutations, up to a global sign flip. RMSNorm models, including Llama and Qwen, also allow each coordinate its own sign flip.

Permutation-only matching therefore has a structural accuracy ceiling, which sign-marginalized Hungarian matching removes in full. Putting recovered gauges together along a fine-tuning path restores 91.1 percent of cross-run coordinates at 1500 steps, against 60.3 percent for endpoint matching in his experiments. Sparse autoencoder reconstruction and sentiment steering survive the recovered gauge, while permutation-only transport breaks or even reverses them.

Catalog record

CREATORJohn Sweeney
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE117 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.31963

Browse the full library · About the Tower of Babel