Traversing the solution space of neural networks with Hessian Null Space Continuation

Ann Huang, Mitchell Ostrow, Zhouyang Lu, William T Redman, Leo Kozachkov, Kanaka Rajan · 2026

Read Traversing the solution space of neural networks with Hessian Null Space Continuation by Ann Huang, Mitchell Ostrow, Zhouyang Lu, William T Redman, Leo Kozachkov, Kanaka Rajan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/PMC13648821.txt.

Description

Ann Huang and five colleagues explore the hidden variety inside trained neural networks, showing that many different internal mechanisms can sit within one connected region of equally good solutions. They introduce a method that walks through weight space along directions that leave network behaviour unchanged, using local curvature to guide the path.

Applied to recurrent networks, image-recognition transformers, and game-playing agents, the method found representations more different from the original than separately trained models, and it exposed a reward-hacking strategy in one agent. It also measured how model size and task complexity shape the solution region. The authors argue this diversity matters for understanding, merging, and editing models.

Catalog record

CREATORAnn Huang, Mitchell Ostrow, Zhouyang Lu, William T Redman, Leo Kozachkov, Kanaka Rajan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE2 KB
LICENSEhttps://creativecommons.org/licenses/by/4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/article/MED/PMC13648821

Browse the full library · About the Tower of Babel