CARV: A Diagnostic Benchmark for Compositional Analogical Reasoning in Multimodal LLMs

Yongkang Du, Xiaohan Zou, Minhao Cheng, Lu Lin · 2026

Read CARV: A Diagnostic Benchmark for Compositional Analogical Reasoning in Multimodal LLMs by Yongkang Du, Xiaohan Zou, Minhao Cheng, Lu Lin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27958.txt.

Introduction

Du and three co-authors present CARV, a test of analogical reasoning for multimodal language models. Unlike older tests, CARV asks a model to draw rules from several image pairs and combine them into new changes. The set holds 5,500 samples. Even the best model tested, Gemini-2.5 Pro, scored only 40.4%, while humans reached 100%. The authors trace the gap to weak rule extraction and poor robustness in complex settings.

Catalog record

CREATORYongkang Du, Xiaohan Zou, Minhao Cheng, Lu Lin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42176
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.27958

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CARV: A Diagnostic Benchmark for Compositional Analogical Reasoning in Multimodal LLMs Yongkang Du, Xiaohan Zou, Minhao Cheng, Lu Lin Abstract Analogical reasoning tests a fundamental aspect of human cognition: mapping the relation from one pair of objects to another. Existing evaluations of this ability in multimodal large language models (MLLMs) overlook the ability to compose rules from multiple sources, a critical component of higher-order intelligence. To close this gap, we introduce CARV ( C ompositional A nalogical R easoning in V ision), a novel task together with a 5,500-sample dataset as the first diagnostic benchmark. We extend the analogy from a single pair to multiple pairs, which requires MLLMs to extract symbolic rules from each pair and compose new transformations. Evaluation on the state-of-the-art MLLMs reveals a striking performance gap: even Gemini-2.5 Pro achieving only 40.4% accuracy, far below human-level performance of 100%. Diagnostic analysis shows two consistent failure modes: (1) decomposing visual changes into symbolic rules, and (2) maintaining robustness under diverse or complex settings, highlighting the limitations of current MLLMs on this task. 1 1 1 The dataset will be released in the camera-ready version. 1 Introduction Figure 1: Examples of single-step and compositional analogy. The single-step analogy (left) delivers one transformation ( color ) from input to target, while the compositional analogy (right) delivers the union of color (wood to red) and number (one to two) transformations. The development of multimodal large language models (MLLMs) ( 4 ) has significantly advanced visual reasoning, enabling models to tackle more complex tasks ( 32 ; 2 ) . As MLLMs approach human-level performance on many tasks, research focus has shifted toward evaluating their capacity for higher-order cognition, including complex processes such as perception ( 12 ) and logical reasoning ( 28 ) . Analogical reasoning, an essential part of human cognition, enables learning from diverse contexts in an adaptive and robust way, supporting decision making and out-of-distribution generalization ( 24 ; 38 ) . Current visual analogy benchmarks ( 38 ; 37 ) are mainly formulated as single-step analogy tasks (Figure 1 left), where models only need to identify and apply the transformation rule derived from a single input pair. However, human analogical reasoning is often sequential and constructive ( 30 ) . It involves combining experiential knowledge from multiple sources to solve unfamiliar problems, rather than focusing on a single relationship ( 8 ; 7 ) . Existing visual analogy evaluation paradigms are overly simplified and therefore neglect the ability to derive novel solutions by composing and synthesizing knowledge from multiple, distinct sources. In this paper, we introduce C ompositional A nalogical R easoning in V ision ( CARV ). We differ from existing visual analogy settings by evaluating a higher-order, often overlooked capability of MLLMs, namely structuring and combining knowledge from multiple sources in analogical reasoning. As shown in Figure 1 , existing single-step visual analogy (left) typically involves extracting and mapping a single transformation, whereas our CARV (right) goes further by demanding the ability to perform logical operations over multiple transformation sets derived from multiple image pairs. Furthermore, CARV is developed in a controlled visual domain. As testing in open-world scenarios often contains visual noise, controlled domain allows us to strictly isolate models’ reasoning capabilities from their perception limits, which enable us to thoroughly study the failure modes. Our contributions are summarized as follows: • Novel Task Formulation We formally define the Compositional Analogical Reasoning in Vision (CARV) task and construct a structured evaluation protocol to test a model’s ability to reconstruct and apply visual transformations learned from multiple sources. • New CARV Benchmark We introduce a comprehensive dataset spanning diverse levels of analogical reasoning complexity. CARV is carefully designed to cover orthogonal dimensions of visual transformations, including spatial and attribute transformations. • Detailed Diagnostic Analysis We conduct extensive experiments on the state-of-the-art MLLMs and find that our task is challenging for most models. To understand these limitations, we develop a fine-grained diagnosis pipeline that evaluates reasoning step by step. Our analysis reveals critical weaknesses of current MLLMs: they struggle to translate visual perceptions into the symbolic, structured rules and are sensitive to increased contextual complexity. 2 Related Work This work mainly explores whether MLLMs can perform compositional analogical reasoning. In light of this, we review two lines of research that form the basis of this work: analogical reasoning and visual composition. Analogical Reasoning Analogical reasoning has long been focused in the AI community. Traditional studies primarily focus on lexical analogy ( 33 ; 11 ; 29 ) , i.e., king to men as queen to women, examining word pairs that share similar semantic relations. Recently, as LLMs demonstrate advanced reasoning capabilities ( 22 ; 5 ) , research on analogical reasoning has extended beyond lexical tasks. For example, exploring how LLMs construct analogies in real-world situations ( 31 ) , applying analogical reasoning to storytelling ( 16 ) or scientific understanding ( 39 ) , and using analogical reasoning as prompting strategies ( 35 ; 27 ) . In the vision domain, analogical reasoning also gets significant attention ( 40 ; 18 ; 41 ; 14 ; 19 ) . The most recent benchmarks study analogical reasoning following the same format in lexical analogy, A to B as C to D , where the input image pair consists of A and B, the query image is C, and the desired output is D. KiVA ( 38 ) , draws inspiration from visual analogy tasks for children, employing basic image transformations (i.e., rotation and flipping) to construct the input image pair. VOILA ( 37 ) incorporates diverse action scenes (number, subject, and action) in the image where the transformation is derived from. Notably, both studies find that MLLMs struggle more with applying transformations than with identifying them. However, these prior approaches are limited by task formats that rely on a single source, a constraint that our work aims to address. Instead of revisiting issues in applying transformations, we reveal the bottleneck of composition ability of current MLLMs. Visual Composition Compositional ability is fundamental for MLLMs to understand complex scenes ( 36 ; 34 ) and create images with sophisticated instructions ( 26 ; 42 ) . The study on visual composition is first conducted on static scene. In CLEVR ( 17 ) and GQA ( 15 ) , models are trained to apply object attributes and spatial relations in question answering or caption generation. As vision language models become popular, the research steers to the image generation process ( 6 ; 23 ) , studying the composition of scene description ( 20 ; 21 ) or task instruction ( 13 ) . AVSD ( 1 ) studies decomposition ability over visual comprehension tasks. However, previous work mainly focuses on composition of object attributes and neglects the composition of abstract spaces or rules, which is found to be an essential cognitive process in human creation process ( 8 ; 9 ) . To fill this gap, our work studies composition of transformation rules observed from image pairs. 3 Compositional Analogical Reasoning In this section, we formally define compositional analogical reasoning and describe the tasks designed to evaluate MLLMs. 3.1 Problem Formulation Preliminary We first introduce the single-step

Read the complete text