Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks

Tejas Pradeep Shirodkar · 2026

Read Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks by Tejas Pradeep Shirodkar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29176.txt.

Introduction

Shirodkar builds an optimiser that respects the hidden symmetries of deep networks. A network's loss does not change under certain reparameterisations: shifting logits, rescaling ReLU units, scaling normalisation layers, or rotating attention heads. Standard optimisers like Adam drift along these flat symmetry directions, wasting motion and blurring measurements taken at the solution.

His Dead-Direction Conditioner lifts a base optimiser into a symmetry-respecting one, conditioning its state within the orbit structure of an invariant metric. The construction handles four gauge families and is proved exactly equivariant on an Adam base. In tests, the conditioned Adam resisted the over-training collapse that plain AdamW suffers, and read a key geometric rate in far more layers. A vision transformer trained from scratch reached lower validation loss, and with a Muon base the method solved a grokking task on ten of eleven seeds where plain Muon never succeeded.

Catalog record

CREATORTejas Pradeep Shirodkar
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE186 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29176

Browse the full library · About the Tower of Babel