Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers
Hao Yan, Haolin Yang, Yiqiao Zhong · 2026
Read Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers by Hao Yan, Haolin Yang, Yiqiao Zhong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-03780.txt.
Description
Hao Yan, Haolin Yang, and Yiqiao Zhong study how transformers infer a latent task from context. That behavior has two modes: recognizing a task seen in training, and adapting to a novel one. Interpretability work has found task vectors, directions in middle-layer representations that steer behavior, but lacked a rigorous basis linking that geometry to the training distribution. The authors train small transformers from scratch on synthetic latent-task sequences.
Both inference modes coexist in a single model. In-distribution behavior follows Bayesian task retrieval, implemented internally through convex combinations of learned task vectors. Out-of-distribution behavior instead uses extrapolative task learning, whose representations sit in a subspace nearly at right angles to the task-vector subspace. Task-vector geometry thus helps explain how models generalize beyond the training tasks.
Catalog record
| CREATOR | Hao Yan, Haolin Yang, Yiqiao Zhong |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 165 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.03780