How Transformers Reject Wrong Answers: Rotational Dynamics of Factual Constraint Processing
Javier Marín · 2026
Read How Transformers Reject Wrong Answers: Rotational Dynamics of Factual Constraint Processing by Javier Marín (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13259.txt.
Description
Javier Marín studies how decoder-only transformers process correct and incorrect factual continuations. When forced to carry a wrong token, a model's hidden states rotate away from the correct path. Angular separation grows through mid-depth, then late layers settle on an asymmetric outcome. Marín reads this pattern as geometric, and stresses that the evidence is observational, not causal. The pattern held across six models from four architecture families, with one flat exception. Probes and patching tests suggest the effect is spread across the trajectory, not one layer.
Catalog record
| CREATOR | Javier Marín |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36195 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.13259