Bi-Orthogonal Factor Decomposition for Vision Transformers

Fenil R. Doshi, Thomas Fel, Talia Konkle, George A. Alvarez, [0.5em] Dept. of Psychology, Harvard University Kempner Institute, Harvard University fenildoshi.com/bfd-transformers · 2026

Read Bi-Orthogonal Factor Decomposition for Vision Transformers by Fenil R. Doshi, Thomas Fel, Talia Konkle, George A. Alvarez, [0.5em] Dept. of Psychology, Harvard University Kempner Institute, Harvard University fenildoshi.com/bfd-transformers (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05328.txt.

Introduction

Fenil R. Doshi and 3 co-authors dissect what vision transformers actually exchange between image tokens in a 2026 paper. Self-attention is the transformer's central mechanism, yet attention maps only show where weight piles up. They say nothing about what queries and keys actually pass between tokens: position, content, or both. That gap leaves the field without a principled account of what attention communicates inside an image model.

Their bi-orthogonal factor decomposition works in two stages: a statistical decomposition first separates token activations into positional and content factors, then a matrix decomposition exposes the modes of the query-key interaction. Knowing what attention trades could guide leaner architectures that keep only the channels carrying real information.

Catalog record

CREATORFenil R. Doshi, Thomas Fel, Talia Konkle, George A. Alvarez, [0.5em] Dept. of Psychology, Harvard University Kempner Institute, Harvard University fenildoshi.com/bfd-transformers
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-05328

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bi-Orthogonal Factor Decomposition for Vision Transformers Fenil R. Doshi, Thomas Fel, Talia Konkle, George A. Alvarez, [0.5em] Dept. of Psychology, Harvard University Kempner Institute, Harvard University fenildoshi.com/bfd-transformers Abstract Self-attention is the central computational primitive of Vision Transformers, yet we lack a principled understanding of what information attention mechanisms exchange between tokens. Attention maps describe where weight mass concentrates; they do not reveal whether queries and keys trade position, content, or both. We introduce Bi-orthogonal Factor Decomposition (BFD), a two-stage analytical framework: first, an ANOVA-based decomposition statistically disentangles token activations into orthogonal positional and content factors; second, SVD of the query-key interaction matrix 𝑸 ​ 𝑲 𝖳 ~\bm{QK}^{\mathsf{T}} exposes bi-orthogonal modes that reveal how these factors mediate communication. After validating proper isolation of position and content, we apply BFD to state-of-the-art vision models and uncover three phenomena. (i) Attention operates primarily through content. Content-content interactions dominate attention energy, followed by content-position coupling. DINOv2 allocates more energy to content-position than supervised models and distributes computation across a richer mode spectrum. (ii) Attention mechanisms exhibit specialization: heads differentiate into content-content, content-position, and position-position operators, while singular modes within heads show analogous specialization. (iii) DINOv2’s superior holistic shape processing emerges from intermediate layers that simultaneously preserve positional structure while contextually enriching semantic content. Overall, BFD exposes how tokens interact through attention and which informational factors – positional or semantic – mediate their communication, yielding practical insights into vision transformer mechanisms. Figure 1 : Content and Position Interactions in DINOv2. Representative bi-orthogonal modes in DINOv2 whose activation patterns illustrate three different interaction. Left: Content–content modes activate on semantic interactions such as object parts, revealing part-to-part or part-to-whole correspondences. Middle: Content–position modes show localization-aware semantic interactions, where semantic features are modulated by spatial context. Right: Position–position modes activate on purely spatial interaction, like left–right flows, top–bottom variations, and Fourier-like patterns, highlighting geometric organization without semantic selectivity. These examples show how different interaction types contribute to information flow in attention. † † footnotetext: ⋆ Denotes equal contribution. 1 Introduction Vision Transformers (ViTs) 81 ; 27 ; 62 ; 22 have become the workhorses of modern computer vision, powering recognition systems deployed in medical and other safety-critical settings 83 ; 4 and serving as the backbone of large-scale generative models 94 ; 92 used by millions 16 . Understanding these systems is therefore both a practical imperative and a scientific opportunity 51 ; 70 . From a scientific perspective, ViTs are among the most scalable 3 ; 90 ; 23 and capable vision models available 91 ; 41 ; 77 ; 13 ; it is natural to ask whether their internal computations share structure with human visual processing 34 ; 76 ; 57 ; 47 ; 50 ; 17 and, if so, where such similarities emerge 12 ; 54 ; 19 ; 46 ? To answer those question, we turn to the interpretability field 26 ; 42 ; 39 , which has proposed a range of theories 29 ; 65 ; 21 and methods 89 ; 44 ; 10 for understanding model internals. Early interpretability efforts centered on attribution 89 ; 36 ; 74 ; 69 ; 33 ; 58 ; 66 . These methods primarily aim to reveal where a model attends or which regions of an input most influence its prediction. While such tools offer valuable intuition, they face well-documented limitations 1 ; 45 ; 18 ; 72 ; 73 ; 59 . They often highlight broad image regions without clarifying what visual patterns drive the decision, nor how these signals are combined internally. In response, concept-based methods emerged 44 ; 38 ; 35 ; 32 ; 93 ; 82 ; 71 ; 64 ; 28 ; 48 , shifting focus from spatial attribution to semantic characterization. Rather than showing where the model looks, these approaches aim to identify what it has learned – isolating human-interpretable “visual atoms” 79 ; 70 or latent concepts within its representations 10 . Concept activation vectors 44 , network dissection 8 or feature visualization 60 ; 31 ; 40 exemplify this paradigm. Such analyses have revealed, for instance, that deep networks spontaneously form units selective for texture, object parts, or higher-order scene semantics 30 ; 49 . Yet, despite their insights, these techniques remain representation-centric: they explain what information is encoded, not how that information arises. They stop short of elucidating the circuits 52 or computations 67 that produce these representations – a question that has become central to modern interpretability. To uncover these internal computations, attention mechanisms present a natural entry point. Self-attention is a core component of ViTs as it mediates communication between tokens and thus provides an explicit interface for information exchange. Existing analyses have mostly interpreted attention through its spatial patterns – visualizing which tokens attend to which others across layers 87 ; 14 . However, these maps reflect only the surface structure of interactions: they indicate the amount of attention exchanged, not the content of that exchange or the reasons for it 84 ; 7 ; 2 . This gap has fueled growing skepticism over whether raw attention maps genuinely constitute explanations 9 ; 78 . Recently, however, a more mechanistic turn has emerged. Rather than treating attention weights as explanatory endpoints, researchers probe the underlying query-key interactions directly via spectral structure 63 . These methods aim to identify dominant interaction modes and relate them to concrete behaviors, echoing circuit-level analyses pioneered in large language models 80 ; 75 ; 11 ; 15 (e.g., induction heads 61 , copying mechanisms 55 , compositional reasoning 5 ; 53 ). Yet, even when dominant modes are identified, their nature often remains ambiguous: are these modes driven primarily by positional geometry, by semantic content, or by global trends such as the mean activation direction? Without a principled factor attribution framework 75 (one that disentangles these potential sources) it remains unclear whether the apparent contextual integration observed in deeper layers reflects genuinely content-driven computation or merely the imprint of positional structure. This motivates our first question. Question 1 . What informational factors – positional, semantic, or global – mediate token communication through attention, and how does this composition vary across layers and between supervised vs. self-supervised models? Understanding what information flows between tokens naturally leads to asking how this flow is organized at the network level. The architecture of multi-head attention introduces a second source of ambiguity here as each head could, in principle, implement a distinct computational role (tracking spatial relations, aggregating semantics, …) supporting a modular view of attention. Alternatively, information may be distributed across heads, with individual heads contributing only partial, entangled signals to a collective computation. These contrasting modular versus distributed hypotheses carry deep implications for interpretability: the former supports circuit-level mechanistic explanations at the head level, while the latter implies that per-head analyses may be fundamentally incomplete. This leads to our second question. Question 2 . Do attention heads and their constituent singular modes exhibit functional specialization into distinct informational operators (content-content, content-position, position-position), and is this specialization consistent across layers and models? Understanding the organization of attention naturally raises a further question: what determines it? If head specializations exist, are they intrinsic to the architecture or induced by the learning paradigm? Evidence suggests

Read the complete text