Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers

Hao Yan, Haolin Yang, Yiqiao Zhong · 2026

Read Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers by Hao Yan, Haolin Yang, Yiqiao Zhong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-03780.txt.

Description

Hao Yan, Haolin Yang, and Yiqiao Zhong study how transformers infer a latent task from context. That behavior has two modes: recognizing a task seen in training, and adapting to a novel one. Interpretability work has found task vectors, directions in middle-layer representations that steer behavior, but lacked a rigorous basis linking that geometry to the training distribution. The authors train small transformers from scratch on synthetic latent-task sequences.

Both inference modes coexist in a single model. In-distribution behavior follows Bayesian task retrieval, implemented internally through convex combinations of learned task vectors. Out-of-distribution behavior instead uses extrapolative task learning, whose representations sit in a subspace nearly at right angles to the task-vector subspace. Task-vector geometry thus helps explain how models generalize beyond the training tasks.

Catalog record

CREATORHao Yan, Haolin Yang, Yiqiao Zhong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE165 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.03780

Browse the full library · About the Tower of Babel