Unraveling the Geometry of Visual Relational Reasoning
Jiaqi Shang, Gabriel Kreiman, Haim Sompolinsky · 2025
Read Unraveling the Geometry of Visual Relational Reasoning by Jiaqi Shang, Gabriel Kreiman, Haim Sompolinsky (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-17382.txt.
Introduction
Jiaqi Shang, Gabriel Kreiman and Haim Sompolinsky wrote a 2025 paper on how machines and humans reason about visual relations. People easily spot sameness in shape or color across new scenes. Neural networks struggle with that kind of leap. The team built a simpler puzzle set to test such reasoning cleanly.
Human trials set a difficulty scale for direct comparison. Of four models tested, a compositional learner matched humans best. A geometric view of the models' inner states predicted who would generalize. New relations turned out to squeeze into subspaces shaped by training. The team adds a training goal that balances signal against spread.
Catalog record
| CREATOR | Jiaqi Shang, Gabriel Kreiman, Haim Sompolinsky |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-17382
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unraveling the Geometry of Visual Relational Reasoning Jiaqi Shang, Gabriel Kreiman, Haim Sompolinsky Abstract Humans readily generalize abstract relations, such as recognizing “constant” in shape or color, whereas neural networks struggle, limiting their flexible reasoning. To investigate mechanisms underlying such generalization, we introduce SimplifiedRPM , a novel benchmark for systematically evaluating abstract relational reasoning, addressing limitations in prior datasets. In parallel, we conduct human experiments to quantify relational difficulty, enabling direct model-human comparisons. Testing four models—ResNet-50, Vision Transformer, Wild Relation Network, and Scattering Compositional Learner (SCL)—we find that SCL generalizes best and most closely aligns with human behavior. Using a geometric approach, we identify key representation properties that accurately predict generalization and uncover a fundamental trade-off between signal and dimensionality: novel relations compress into training-induced subspaces. Layer-wise analysis reveals where relational structure emerges, highlights bottlenecks, and generates concrete hypotheses about abstract reasoning in the brain. Motivated by these insights, we propose SNRloss, a novel objective explicitly balancing representation geometry. Our results establish a geometric foundation for relational reasoning, paving the way for more human-like visual reasoning in AI and opening promising avenues for extending geometric analysis to broader cognitive tasks. Introduction Humans exhibit a remarkable ability to recognize and generalize abstract relations across diverse contexts. Once we understand an abstract relation like “constant,” we can flexibly apply it to different object attributes, such as a row of items sharing the same color, shape, or number. This generalization ability allows us to recognize patterns and make inferences in novel situations, supporting flexible reasoning and complex problem-solving. A central challenge in both cognitive science and artificial intelligence is to understand the mechanisms that enable this kind of generalization. Raven’s Progressive Matrices (RPM) task [ 1 ] has long served as a benchmark for abstract relational reasoning. In this task, participants view a 3\times 3 matrix of panels with the bottom-right panel missing. Participants must infer the relational rules governing the rows or columns, such as “constant shape across the row”, and use these inferred rules to select the missing panel from several alternatives. Decades of behavioral studies in cognitive science revealed consistent patterns of human performance on the task and have established RPM accuracy as a reliable predictor of general cognitive and problem-solving abilities [ 1 , 2 , 3 , 4 , 5 ] . Yet, the cognitive mechanisms that support this generalization ability remain unclear. Inspired by RPM, artificial intelligence research has developed large-scale tasks, such as PGM [ 6 ] and RAVEN [ 7 , 8 ] , to evaluate relational reasoning in neural networks. While models trained on these datasets [ 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 ] achieve high accuracy on learned relational rules, they generalize poorly to novel rules [ 19 ] . Recent evaluations of large-scale pre-trained vision-language models on the RAVEN task have shown that these models also struggle, despite employing advanced inference strategies such as in-context learning [ 20 ] . This limitation is further compounded by the lack of detailed human behavioral data on the same tasks, making it challenging to assess whether the model’s behavior aligns with human reasoning. While recent work has begun to explore mechanisms in simplified relational tasks, such as transitive inference [ 21 , 22 , 23 ] and same-different relations [ 24 , 25 , 26 ] , broad generalization across diverse relations and realistic image inputs remains an open problem. This gap motivates a fundamental question: What mechanisms enable or constrain the generalization of abstract relations? Specifically, how are relational rules represented in neural networks, and what geometric principles govern whether they can be flexibly applied to new contexts? To address these questions, we introduce SimplifiedRPM , a novel benchmark designed to systematically evaluate abstract relational generalization. We assess both human performance and a range of neural network models on this benchmark. We propose that a model’s ability to generalize can be understood in terms of the geometry of its emergent rule manifolds—the representations formed for different relational rules. To formalize this hypothesis, we extend a prior geometric framework developed for object classification [ 27 ] to the domain of relational reasoning. We show that generalization performance is accurately predicted by several measurable and interpretable rule manifold geometries. Our analysis reveals a core tradeoff: generalization is constrained by a compression mechanism that restricts both seen and unseen rule manifolds to a shared low-dimensional subspace shaped by the training rules. Motivated by these insights, we propose SNRloss, a novel training objective that directly shapes representational geometry to improve relational generalization. Together, our findings provide a principled geometric account of abstract relation generalization, offer testable hypotheses for abstract relation reasoning in the brain, and pave the way for applying geometric analysis to broader domains in cognitive science and artificial intelligence. Results Evaluating Deep Neural Networks in Relational Reasoning To systematically assess neural networks’ ability to generalize abstract relations, we developed the Simplified Raven’s Progressive Matrices ( SimplifiedRPM ) task (Fig. 1 a). Inspired by the original RPM task [ 1 ] , SimplifiedRPM requires models to infer a relational rule from a sample row (Fig. 1 a, top) and determine which of two choice rows (Fig. 1 a, bottom) follows the same rule. Each row consists of three panels with abstract objects, such as triangles or circles, arranged according to a relational rule. For example, the sample row in Fig. 1 a illustrates the “constant shape” rule, where all objects share the same shape while other attributes, such as number or position, vary. SimplifiedRPM refines the original RPM task [ 1 ] by reducing answer choices from eight to two and presenting a single example row instead of two to remove multi-row comparisons. These changes maintain RPM’s core challenge while enabling a more focused assessment of relational reasoning. We construct a dataset for evaluating abstract relation generalization in the SimplifiedRPM task, comprising rows governed by 40 distinct relational rules. Each rule applies an abstract relation (e.g., constant) to an object attribute (e.g., shape). The abstract relations and attributes are selected to be consistent with those used in human experiments [ 1 , 4 ] while maximizing the number of rules for comprehensive evaluation. Importantly, each row corresponds to a unique rule. We generate many rows per rule, resulting in a dataset of 400k training rows, 40k validation rows, and 40k test rows. To construct a SimplifiedRPM task trial, a sample row is selected that follows one rule. A correct choice is drawn from the same rule, while an incorrect choice is drawn from a different rule. Further details are provided in the Methods section. We assess models’ ability to generalize abstract relations using a held-out rule split, in which 5 out of 40 relational rules are excluded from training. During training, models learn relational structures from the remaining 35 rules. For instance, a model trained on "constant number" and "OR shape" may never encounter "constant shape." After training, their performance is evaluated on the five withheld rules using the SimplifiedRPM task. Instead of evaluating models pertained on large-scale datasets, where the overlap between their training distribution and the