The Geometry of Updates: Fisher Alignment at Vocabulary Scale

John Sweeney · 2026

Read The Geometry of Updates: Fisher Alignment at Vocabulary Scale by John Sweeney (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27242.txt.

Introduction

Sweeney studies how to pick source data for adapting language models that share one vocabulary. In string domains like chemical SMILES, protein, and genome sequences, candidate corpora share a tokenizer but aim at different targets. Similarity of internal features can then go dark, saying nothing about transfer. A key identity shows head Fisher alignment equals a cosine between kernel mean maps in joint activation-error space. That identity exposes activity, error, and coupling factors separately, in a transfer learning paper appearing in 2026.

A streaming estimator, FisherSketch, computes that cosine in one pass. Task signatures stay tiny, small enough to store beside a model hash. Experiments with shared-parameter and layer checks, plus verbalizer-shift tests on a Llama model, keep the score informative. Representation similarity fails to separate tasks in those same tests. Update geometry, not feature resemblance, carries the transfer signal. Source selection for huge vocabularies becomes practical.

Catalog record

CREATORJohn Sweeney
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE234 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.27242

Browse the full library · About the Tower of Babel