A Data-driven Typology of Vision Models from Integrated Representational Metrics
Jialin Wu, Shreya Saha, Yiqing Bo, Meenakshi Khosla · 2025
Read A Data-driven Typology of Vision Models from Integrated Representational Metrics by Jialin Wu, Shreya Saha, Yiqing Bo, Meenakshi Khosla (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-21628.txt.
Introduction
Whether two neural systems use equivalent internal representations is measured in many incompatible ways. Jialin Wu and 3 co-authors compare those similarity metrics in a 2025 paper. Single metric studies dominate prior work, though each metric stresses a distinct facet of correspondence. Artificial vision models and real neural recordings both serve as test beds. Known structure supplies the ground truth for scoring metrics.
For models, procedurally matched pairs should score higher than dissimilar pairs. For neural data, same region responses should align across subjects while distinct regions separate. Metrics preserving representational geometry performed strongly in their tests. Tuning sensitive measures also featured. The study guides how to pick a similarity metric for biological and artificial systems alike.
Catalog record
| CREATOR | Jialin Wu, Shreya Saha, Yiqing Bo, Meenakshi Khosla |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-21628
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Data-driven Typology of Vision Models from Integrated Representational Metrics Jialin Wu, Shreya Saha, Yiqing Bo, Meenakshi Khosla Abstract Large vision models differ widely in architecture and training paradigm, yet we lack principled methods to determine which aspects of their representations are shared across families and which reflect distinctive computational strategies. We leverage a suite of representational similarity metrics, each capturing a different facet—geometry, unit tuning, or linear decodability—and assess family separability using multiple complementary measures. Metrics preserving geometry or tuning (e.g., RSA, Soft Matching) yield strong family discrimination, whereas flexible mappings such as Linear Predictivity show weaker separation. These findings indicate that geometry and tuning carry family-specific signatures, while linearly decodable information is more broadly shared. To integrate these complementary facets, we adapt Similarity Network Fusion (SNF), a method inspired by multi-omics integration. SNF achieves substantially sharper family separation than any individual metric and produces robust composite signatures. Clustering of the fused similarity matrix recovers both expected and surprising patterns: supervised ResNets and ViTs form distinct clusters, yet all self-supervised models group together across architectural boundaries. Hybrid architectures (ConvNeXt, Swin) cluster with masked autoencoders, suggesting convergence between architectural modernization and reconstruction-based training. This biology-inspired framework provides a principled typology of vision models, showing that emergent computational strategies—shaped jointly by architecture and training objective—define representational structure beyond surface design categories. 1 Introduction The rapid proliferation of vision models—spanning diverse architectures from CNNs to Vision Transformers, and training paradigms from supervised to self-supervised learning—has created a rich ecosystem of computational approaches to visual processing. Yet we lack principled methods to understand which aspects of learned representations are universally shared across this diverse landscape and which are specific signatures of particular model families. Do all vision models converge on similar geometric organizations of their representation spaces? Is linearly accessible information a common currency across architectures, or do different model families encode information in fundamentally distinct ways? These questions are critical for understanding the computational principles underlying successful vision models and for predicting how different models will behave on novel tasks. Current approaches to model comparison rely heavily on individual similarity metrics—Representational Similarity Analysis (RSA) ( Kriegeskorte et al. ) , Centered Kernel Alignment (CKA) ( Kornblith et al. 2019 ) , Linear Predictivity ( Yamins et al. 2014 ) , and others—each capturing a specific facet of representational structure. However, this fragmented methodology obscures a crucial insight: different representational facets (geometry, tuning properties, linearly accessible information) may vary in their universality across model families. Some aspects might reflect convergent computational solutions shared broadly across architectures, while others might constitute distinctive signatures of specific model families. Understanding this landscape requires not just comparing models, but systematically evaluating which representational properties are conserved versus specialized. In this work, we introduce a framework that addresses these challenges through two key contributions. First, we systematically evaluate how different representational facets discriminate between model families, revealing that geometry or tuning-preserving metrics (RSA, Soft Matching ( Khosla & Williams 2024 ) ) strongly separate families while metrics capturing linearly accessible information show weaker discrimination—suggesting that linearly accessible information represents a more universal computational solution while geometric organization or tuning of individual neurons is more family-specific. Second, inspired by multi-omics approaches in biology where diverse molecular signatures are integrated to reveal cell types, we employ Similarity Network Fusion (SNF) ( Wang et al. 2014 ) to combine these complementary perspectives into unified model signatures that provide clearer family identity than any single metric alone. This integrated approach enables us to construct a data-driven typology of vision models —a novel contribution that moves beyond surface-level architectural categories to reveal how models actually organize information. Our typology does not rely on a priori assumptions about which models should group together based on architecture or training method. Instead, following empirical traditions in psychology, neuroscience, and genetics ( Wang et al. 2014 ; Letwin et al. 2006 ; Echtermeyer et al. 2011 ; Mukamel & Ngai 2019 ) where researchers identify clusters of individuals through correlations across multiple behavioral or molecular indices, we discover natural groupings based on how models process visual information. While typologies could alternatively be defined from theoretical perspectives emphasizing explicit model properties like architecture or training data, our empirical approach reveals surprising patterns: all self-supervised models form a unified cluster that transcends architectural boundaries, with self-supervised ResNets grouping more closely with self-supervised ViTs than with their supervised architectural siblings. Similarly, hybrid architectures (ConvNeXt, Swin) ( Liu et al. 2022 ; Liu et al. 2021 ) cluster with MAE models, suggesting that architectural modernization and masked reconstruction converge on similar computational strategies despite different design origins. This data-driven typology provides researchers with a reference framework for understanding where any model instance sits within the broader landscape of vision models. Just as biological taxonomies help scientists understand relationships between species based on genetic and phenotypic traits, our representational typology reveals the “species” of vision models based on their computational strategies. By developing methods to systematically integrate different facets of representation in comparative analyses of models, we provide practical tools for navigating the expanding universe of vision models—enabling researchers to understand model relationships, predict transfer learning compatibility, and make informed choices about which models will exhibit similar behaviors on novel tasks. Figure 1: Top left: Each representational metric defines a pairwise similarity matrix over models. Bottom left: Each matrix is visualized as an affinity graph, with nodes representing models and edge widths reflecting pairwise similarity strength; weak similarities below a threshold are omitted for clarity. Right: A consensus matrix obtained via Similarity Network Fusion (SNF) highlights relations consistently supported across metrics while leveraging complementary signals. In the fused graph, solid edges denote agreement across all metrics, dotted edges indicate partial support; strong but uncorroborated edges may persist with reduced weight (e.g., edge 4–5); weak AND metric-specific connections are typically suppressed (e.g., edge 1–5). 2 Methods 2.1 Model Selection and Dataset We analyze 35 vision models across four primary categories: supervised Convolutional Neural Networks (CNNs), self-supervised CNNs, supervised Transformers, and self-supervised Transformers. We treat ConvNeXt ( Liu et al. 2022 ) and Swin ( Liu et al. 2021 ) as distinct families due to their hybrid nature—ConvNeXt incorporates Transformer-inspired design principles within a convolutional architecture, while Swin introduces CNN-like inductive biases into the Transformer framework. For datasets, we use the ImageNet-1k ( Deng et al. 2009 ) and Ecoset ( Mehrer et al. 2021 ) validation sets and the CIFAR10 ( Krizhevsky 2009 ) and CIFAR100 ( Krizhevsky 2009 ) test sets. Complete dataset and model details are provided in Appendix A and B . 2.2 Representational Metrics We evaluate widely used similarity metrics that differ in the flexibility of the mappings they permit—from permutation-based alignments (soft-matching) to rigid geometric transformations (Procrustes) to looser linear mappings (linear predictivity) as well as non-fitting approaches that compare representational geometry directly (RSA). Consider two representations \mathbf{X}_{i}\in\mathbb{R}^{M\times N_{i}} and \mathbf{X}_{j}\in\mathbb{R}^{M\times N_{j}} from different models, where M denotes the number of stimuli and N_{i},N_{j} denote the number of units. All representations are mean-centered along the sample dimension as required. Singular Vector Canonical Correlation Analysis ( Raghu et al. 2017 ) . SVCCA first applies singular value decomposition (SVD) to the representation matrices from two models or layers to