MM-OPERA: Benchmarking Open-ended Association Reasoning for Large Vision-Language Models

Zimeng Huang, Jinxin Ke, Xiaoxuan Fan, Yufeng Yang, Yang Liu, Liu Zhonghan, Zedi Wang, Junteng Dai, Haoyi Jiang, Yuyu Zhou, Keze Wang, Ziliang Chen, [0.5em] Sun Yat-sen University · 2025

Read MM-OPERA: Benchmarking Open-ended Association Reasoning for Large Vision-Language Models by Zimeng Huang, Jinxin Ke, Xiaoxuan Fan, Yufeng Yang, Yang Liu, Liu Zhonghan, Zedi Wang, Junteng Dai, Haoyi Jiang, Yuyu Zhou, Keze Wang, Ziliang Chen, [0.5em] Sun Yat-sen University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26937.txt.

Introduction

Zimeng Huang, with eleven co-authors, benchmarks a basic human skill in vision-language models: association. Association underlies creative thinking and knowledge integration, yet most benchmarks use closed-ended tasks that miss open-ended associative reasoning. Their benchmark, MM-OPERA, holds over eleven thousand instances across two tasks.

Huang and the team set models a remote-item association task and an in-context association task, aligned with human psychometric principles. Answers are free-form, with explicit reasoning paths, and tailored judge strategies score them with process-aware precision. Extensive studies across abilities, domains, languages, and cultures map where current models fall short. Sensitivity and validity analyses check the benchmark itself. The authors offer the data set and code publicly, aiming at more human-like general AI.

Catalog record

CREATORZimeng Huang, Jinxin Ke, Xiaoxuan Fan, Yufeng Yang, Yang Liu, Liu Zhonghan, Zedi Wang, Junteng Dai, Haoyi Jiang, Yuyu Zhou, Keze Wang, Ziliang Chen, [0.5em] Sun Yat-sen University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE120 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-26937

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MM-OPERA: Benchmarking Open-ended Association Reasoning for Large Vision-Language Models Zimeng Huang, Jinxin Ke, Xiaoxuan Fan, Yufeng Yang, Yang Liu, Liu Zhonghan, Zedi Wang, Junteng Dai, Haoyi Jiang, Yuyu Zhou, Keze Wang, Ziliang Chen, [0.5em] Sun Yat-sen University Abstract Large Vision-Language Models (LVLMs) have exhibited remarkable progress. However, deficiencies remain compared to human intelligence, such as hallucination and shallow pattern matching. In this work, we aim to evaluate a fundamental yet underexplored intelligence: association, a cornerstone of human cognition for creative thinking and knowledge integration. Current benchmarks, often limited to closed-ended tasks, fail to capture the complexity of open-ended association reasoning vital for real-world applications. To address this, we present MM-OPERA, a systematic benchmark with 11,497 instances across two open-ended tasks: Remote-Item Association (RIA) and In-Context Association (ICA), aligning association intelligence evaluation with human psychometric principles. It challenges LVLMs to resemble the spirit of divergent thinking and convergent associative reasoning through free-form responses and explicit reasoning paths. We deploy tailored LLM-as-a-Judge strategies to evaluate open-ended outputs, applying process-reward-informed judgment to dissect reasoning with precision. Extensive empirical studies on state-of-the-art LVLMs, including sensitivity analysis of task instances, validity analysis of LLM-as-a-Judge strategies, and diversity analysis across abilities, domains, languages, cultures, etc., provide a comprehensive and nuanced understanding of the limitations of current LVLMs in associative reasoning, paving the way for more human-like and general-purpose AI. The dataset and code are available at https://github.com/MM-OPERA-Bench/MM-OPERA . 1 Introduction Recent advancements in Large Vision-Language Models (LVLMs) have significantly improved their ability to handle multi-modal inputs and address diverse tasks. Systems such as GPT-4 [ 69 ] , Gemini models [ 79 ] , and LLaVA [ 55 ] exhibit remarkable proficiency in visual understanding, language generation, and multi-step reasoning. These capabilities are driving transformative applications across fields such as education, design, scientific discovery, embodied intelligence, and so on [ 39 , 19 , 30 , 57 ] . Existing benchmarks for LVLMs [ 3 , 62 , 35 , 95 , 44 , 43 , 96 , 18 , 31 , 59 , 60 ] has facilitated systematic assessments of instruction-following and alignment tasks, focusing on recognition, comprehension, and reasoning. However, the evaluation of association intelligence in LVLMs remains underexplored. Association , a cornerstone of human cognition, enables creative thinking [ 64 ] , underpins the integration of fragmented information into coherent knowledge and supports critical cognitive processes such as memory, perception, and rule discovery [ 5 ] . We argue that LVLMs need to develop this core capability to move beyond shallow pattern matching toward true knowledge synthesis and reasoning. It is a prerequisite for many real-world applications such as scientific discovery, creative ideation and design, personalized education, innovative problem-solving and robot planning. Current efforts, such as the Labyrinth of Links [ 46 ] have begun to formalize association as an evaluation target, using closed-ended tasks with predefined options to probe associative memory. While this approach offers valuable insights, it falls short of capturing the full scope of association reasoning required for real-world AI applications. We argue that open-ended association reasoning is essential for two key reasons: (1) Closed-ended tasks with fixed options may introduce bias, subtly guiding the model’s associative behavior and masking its true capacity for independent reasoning; (2) The fixed-answer format struggles to evaluate complex, long-form association reasoning, limiting the ability to challenge models on intricate, multi-step relational inference. These limitations motivate our development of a new benchmark that prioritizes open-endedness to rigorously assess and ultimately enhance LVLMs’ association reasoning capabilities. Figure 1 : An overview of MM-OPERA. The RIA task challenges models to discover meaningful connections between unrelated elements, while the ICA task requires transferring relationship patterns from a context pair to a query item to generate an appropriate target. The reference answer represents just one possible valid response. The association reasoning paths are used to evaluate the coherence and depth of the step-by-step reasoning process. In cognitive science, association emerges from the interplay of convergent and divergent thinking : the former identifying meaningful connections and selecting optimal solutions; the latter generating multiple unique ideas [ 83 , 13 , 63 ] . The Remote Associates Test (RAT) [ 64 , 24 , 17 , 87 , 2 ] exemplifies this by requiring individuals to uncover links between distant concepts, a process vital for adaptive problem-solving. To mirror this in LVLMs and address the shortcomings of prior work, we propose MM-OPERA ( M ulti- M odal OP en- E nded R easoning-guided A ssociation), a benchmark designed to evaluate association reasoning without predefined constraints. It assesses how models identify and express meaningful links across distant concepts ( i.e. convergent thinking), expected to emerge through diverse reasoning paths ( i.e. divergent thinking). Table 1 highlights how MM-OPERA diverges from The Labyrinth of Links by adopting open-ended tasks, more challenging reasoning scenarios, and a broader scope of evaluation, enabling a deeper probe into LVLMs’ relational inference abilities. Table 1 : Comparison between The Labyrinth of Links and MM-OPERA. Dimension The Labyrinth of Links MM-OPERA (Ours) Task Format Multi-choice, closed-ended Free-form, Open-ended Association Tasks Basic Steps: Single / Synchronous / Asynchronous More Complex : Remote-Item Association / In-Context Association Association Scope Adjectives and Verb limited semantic concepts 3 relationship types, 13 ability dimensions ; broad cultural, linguistic and thematic contexts Evaluation Metrics Correctness-focused: Max / Mean Step, Success Ratio Multi-dimensional assessment : Score Rate, High Score Rate, \bigtriangleup HR, Reasoning Score, Reasonableness, Distinctiveness, Knowledgeability Evaluation Flexibility Option-based, limited generative capacity Fully generative , supports diverse reasoning paths and rationales MM-OPERA comprises 11,497 instances across two core tasks (Figure 1 ): Remote-Item Association (RIA), testing the ability to link distant concepts with structured reasoning, and In-Context Association (ICA), probing pattern recognition within in-context learning [ 29 ] . Spanning 13 associative dimensions and diverse cultural, linguistic, and thematic contexts, it offers a comprehensive evaluation framework. It prioritizes free-form responses, employing reference answers as heuristic quality benchmarks rather than rigid correctness criteria. To evaluate open-ended outputs, we design tailored LLM-as-a-Judge strategies with a cascading scoring rubric. Furthermore, by leveraging process-reward principles to trace reasoning steps, our evaluation captures cognitive flow and knowledge integration, surpassing traditional outcome-focused metrics. Our contributions are threefold: 1. MM-OPERA: We introduce a benchmark of 10,000 + instances for evaluating LVLMs’ association reasoning, centered on Remote-Item Association (RIA) and In-Context Association (ICA) tasks inspired by classic psychometric studies. It spans 13 analytical dimensions to enable comprehensive assessment. 2. LLM-as-a-Judge Strategies: To support open-ended evaluation, we design tailored LLM-as-a-Judge methods that assess both response quality and reasoning processes, enabling fine-grained and reliable scoring. 3. Profound Findings: Our analysis reveals key limitations of current LVLMs and highlights the critical role of association reasoning in advancing real-world, general-purpose AI. 2 Related Work Large Vision Language Models (LVLMs). Early studies [ 78 , 98 , 73 ] established the foundations of vision-language models. CoCa [ 94 ] , Flamingo [ 1 ] , and BLIP-2 [ 47 ] , advanced performance with enhanced architectures and large-scale multimodal pretraining. InstructBLIP [ 25 ] , MiniGPT-4 [ 103 ] , and LLaVA [ 56 ] , have refined multimodal instruction tuning and alignment strategies. Recent open-source LVLMs, e.g. , LLaVA-OneVision [ 45 ] , mPLUG-Owl3 [ 91 ]

Read the complete text