SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models
Olaf Dünkel ⋆ \star, Basavaraj Sunagad ⋆ \star, Haoran Wang, David T. Hoffmann, Christian Theobalt, Adam Kortylewski · 2026
Read SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models by Olaf Dünkel ⋆ \star, Basavaraj Sunagad ⋆ \star, Haoran Wang, David T. Hoffmann, Christian Theobalt, Adam Kortylewski (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-31597.txt.
Introduction
Dünkel and team present SOCO, a benchmark for semantic object correspondence. Semantic correspondence tests if object parts match across instances and categories despite varied looks and viewpoints. SOCO gives consistent keypoint annotations across 100 categories. It also includes language descriptions to test vision-language models. Tests reveal vision backbones encode strong meaning structure yet transfer matches poorly across kinds. Vision-language models beat visual matching on text-prompted part finding, and match performance predicts downstream scores more strongly than ImageNet scores.
Catalog record
| CREATOR | Olaf Dünkel ⋆ \star, Basavaraj Sunagad ⋆ \star, Haoran Wang, David T. Hoffmann, Christian Theobalt, Adam Kortylewski |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77338 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.31597
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models Olaf Dünkel ⋆ \star, Basavaraj Sunagad ⋆ \star, Haoran Wang, David T. Hoffmann, Christian Theobalt, Adam Kortylewski Abstract Measuring structured object understanding in vision foundation models remains challenging due to inconsistent evaluation protocols and limited part-level supervision. Semantic correspondence (SC) evaluates this capability by testing whether object parts can be matched across instances and categories under large variations in appearance, viewpoint, and geometry. To enable a systematic SC evaluation, we introduce SOCO, a new benchmark for Semantic Object Correspondence that introduces a taxonomy of correspondence types and provides consistent, functionally meaningful keypoint annotations across 100 categories and over 1M correspondence pairs. In addition, SOCO includes keypoint language descriptions, enabling the evaluation of large vision–language models (LVLMs) and their fine-grained part-level understanding. Comprehensive experiments reveal that (i) vision foundation backbones encode strong semantic structure but transfer correspondences poorly across related categories and only partially capture object-part position, (ii) LVLMs are stronger at text-prompted part localization than at visual-reference cross-image matching, exposing a gap between language-grounded localization and fine-grained visual correspondence, and (iii) correspondence performance predicts dense downstream tasks—segmentation, tracking, 3D pose estimation, and 3D detection—more strongly than ImageNet classification. Together, these findings position SOCO as a benchmark for structured, part-level representation quality in vision and multimodal foundation models. Dataset and code are available at https://genintel.github.io/SOCO/ . $\star$ $\star$ footnotetext: Equal contribution. Figure 1 : SOCO provides the first taxonomy-driven, language-grounded formulation of Semantic Object Correspondence (SOC), enabling structured, semantically coherent, and cross-category part annotations across 100 diverse categories, which allows evaluating semantic and structured object understanding in vision foundation models (VFMs) and large vision language models (LVLMs). 1 Introduction Visual representations form the foundation of visual intelligence. Evaluating their quality has long been central to progress in computer vision. Existing benchmarks probe distinct aspects of visual understanding, ranging from category-level recognition benchmarks such as ImageNet [ 16 ] to spatial localization tasks including detection, segmentation, and pose estimation [ 36 , 15 , 2 ] . However, they provide limited insight into whether a representation captures structured object understanding , i.e., the ability to relate semantically corresponding parts across different object instances and categories. Recently, semantic correspondence (SC) has become increasingly important for evaluating self-supervised and foundation models [ 60 , 66 , 51 , 24 ] , as it measures a model’s ability to establish correspondences between object parts across different instances of a category—a capability that requires consistently capturing object structure under substantial variation in appearance, viewpoint, and geometry. The ability to establish such correspondences is crucial for transferring knowledge across related objects, for example when adapting affordances, recognition, pose estimation, or reconstruction to unseen categories, which is important for embodied and robotic systems [ 23 , 68 ] . However, despite this growing adoption, progress in SC research has been constrained by the lack of a clear task definition and by the limitations of existing datasets [ 45 , 63 ] . Current benchmarks conflate two distinct abilities in a single within-category score—recognizing the same local concept (e.g. a wheel center) and identifying its correct repeated instance within an object (front-left vs. rear-right wheel)—and do not evaluate transfer across related categories (a wheel center on a car, bus, or tractor) at all. This ambiguity limits current evaluations of modern foundation models. We therefore propose Semantic Object Correspondence (SOC) , a taxonomy-driven formulation of semantic correspondence that disentangles these three abilities. SOC explicitly models the relationship between object part semantics and the overall object structure, providing a clearer separation between local concept recognition, object-relative identity, and cross-category transfer. Concretely, the taxonomy distinguishes concept correspondence (CC, matching the same local concept), semantic object correspondence (SOC, matching the same concept with the same object-relative identity), and cross-category SOC (matching object-relative keypoints across related categories through shared taxonomy concepts). This decomposition reduces annotation ambiguity, standardizes what constitutes a valid correspondence across object categories and viewpoints, and makes distinct model failure modes separately measurable. Building on this definition, we present SOCO , a S emantic O bject CO rrespondence dataset that measures SOC with taxonomy-driven keypoint annotations across 100 object categories organized into four super-classes. Unlike prior datasets, SOCO emphasizes semantic consistency and cross-category matching, enabling structured evaluation of correspondence across varying geometry and appearance for a diverse range of man-made object and animal categories. Across a broad family of vision foundation models—including self-supervised and vision–language models such as DINO [ 47 , 60 , 13 ] , CLIP [ 48 ] , Stable Diffusion [ 53 ] , and I-JEPA [ 3 ] —the decomposed evaluation reveals distinct failure modes: strong VFMs recognize local concepts but exhibit large CC \to SOC drops (repeated-part confusion) and further SOC \to Cross-SOC drops (limited category-level abstraction). Moreover, SOC is a practical zero-shot diagnostic of representation quality: it correlates with dense downstream tasks—segmentation, tracking, 3D pose, and 3D detection—more strongly than ImageNet classification accuracy. As connecting vision and language modalities becomes increasingly important, benchmarks for structured object understanding should not only evaluate visual representations but also large vision–language models (LVLMs) . To support this, we extend SOCO with language descriptions of correspondence keypoints , creating a comprehensive benchmark for studying the interplay between visual correspondences and natural language in multimodal foundation models. LVLM evaluations reveal a complementary failure mode: current LVLMs are substantially stronger at text-prompted part localization within a single image than at visual-reference correspondence across images, exposing a gap between language-grounded localization and visual matching. Together, the results position SOCO as a unified benchmark for analyzing fine-grained visual reasoning and multimodal representation quality in the era of large foundation models. In summary, our main contributions are: • Task formulation. We introduce Semantic Object Correspondence (SOC) as a taxonomy-driven decomposition of semantic correspondence into concept correspondence, structured object understanding, and cross-category transfer. • Dataset. We present SOCO , a large-scale benchmark built on this taxonomy, featuring 100 diverse categories, semantically grounded keypoint annotations, and over 1 M correspondence pairs, with provided language descriptions that enable joint study of visual correspondence and language understanding in multimodal models. • Vision-model analysis. Across a broad family of vision foundation models, the SOC decomposition exposes repeated-part confusion and limited cross-category abstraction even in strong dense self-supervised backbones. • LVLM analysis. On the same taxonomy, current LVLMs are stronger at text-prompted part localization than at visual-reference cross-image matching, revealing a gap between language-grounded localization and fine-grained visual correspondence. • SOC as a representation diagnostic. We conduct extensive experiments across a broad family of vision models, demonstrating that SOC correlates more strongly than ImageNet k NN with dense downstream tasks, positioning SOC as a practical zero-shot diagnostic of representation quality. 2 Related work Semantic Correspondence Benchmarks. Finding correspondences is a fundamental task in computer vision, ranging from geometric [ 37 , 52 ] and stereo matching [ 55 , 44 ] to optical flow [ 11 ] and tracking [ 71 ] , which are typically constrained to the same instance or scene. In contrast, semantic correspondence aims to establish correspondences between object parts across different instances of the same category. Early datasets such as PF-PASCAL and PF-WILLOW [ 27 ] , TSS [ 65 ] , and Freiburg-Cars [ 56 ]