Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models

Arnas Uselis, Andrea Dittadi, Seong Joon Oh · 2026

Read Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models by Arnas Uselis, Andrea Dittadi, Seong Joon Oh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-24264.txt.

Description

Arnas Uselis and two co-authors argue that compositional generalization in vision embedding models requires linear, orthogonal representations. The 2026 paper defines compositional generalization as recognizing familiar parts in novel combinations.

The authors formalize, for compositional generalization under standard training, three desiderata: divisibility, transferability, and stability. They show necessary geometric constraints follow: per-concept components must combine linearly to form representations. That result grounds the linear representation hypothesis. Tests across modern vision models find factors that are partly linear, low-rank, and near-orthogonal per concept. Dimension bounds link the number of composable concepts to embedding geometry.

Catalog record

CREATORArnas Uselis, Andrea Dittadi, Seong Joon Oh
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE168 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-24264

Browse the full library · About the Tower of Babel