AMIGO: Agentic Multi-Image Grounding Oracle Benchmark
Min Wang, Ata Mahjoubfar · 2026
Read AMIGO: Agentic Multi-Image Grounding Oracle Benchmark by Min Wang, Ata Mahjoubfar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-28662.txt.
Introduction
Wang and Mahjoubfar introduce AMIGO, a benchmark for agentic vision-language models over many images. Most tests still check single-image, single-turn answers. AMIGO hides a target image in a gallery of similar images. The model must find it by asking attribute-focused yes or no questions. The oracle answers Yes, No, or Unsure, and invalid actions draw a Skip penalty. Metrics cover success, evidence checks, efficiency, protocol compliance, and robustness. Results show that final accuracy alone overstates grounded performance.
Catalog record
| CREATOR | Min Wang, Ata Mahjoubfar |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42406 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.28662
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AMIGO: Agentic Multi-Image Grounding Oracle Benchmark Min Wang, Ata Mahjoubfar Abstract Agentic vision-language models increasingly act through extended interactions, but most evaluations still focus on single-image, single-turn correctness. We introduce AMIGO ( A gentic M ulti- I mage G rounding O racle Benchmark), a long-horizon benchmark for hidden-target identification over galleries of visually similar images. In AMIGO, the oracle privately selects a target image, and the model must recover it by asking a sequence of attribute-focused Yes/No/Unsure questions under a strict protocol that penalizes invalid actions with Skip . This setting stresses (i) question selection under uncertainty, (ii) consistent constraint tracking across turns, and (iii) fine-grained discrimination as evidence accumulates. AMIGO also supports controlled oracle imperfections to probe robustness and verification behavior under inconsistent feedback. We instantiate AMIGO with Guess My Preferred Dress task and report metrics covering both outcomes and interaction quality, including identification success, evidence verification, efficiency, protocol compliance, noise tolerance, and trajectory-level diagnostics. † † footnotetext: This work is still in progress. We’ll update with more results and analysis soon. The dataset is provided as an ancillary file, anc/data.json . Work in Progress 1 Introduction Vision-language models (VLMs) [ 25 , 18 , 22 , 20 , 8 ] have advanced rapidly in instruction following and grounded reasoning, enabling strong performance on captioning, visual question answering, and multimodal dialogue. As VLMs become more agentic (planning and acting through extended exchanges), evaluation protocols increasingly need to measure not only correctness, but also interaction policies: how models gather information, maintain state, and adapt under uncertainty. Yet many benchmarks [ 5 , 16 , 28 , 14 , 30 ] remain static and short-horizon, typically scoring one-shot answers given one image (or a small fixed set), with limited visibility into strategy, memory, and belief revision over time. We study a complementary setting: interactive hidden-target identification over image collections. A user (or oracle) privately selects a target image from a gallery of visually similar candidates, and the model must discover the target by asking constrained questions about observable attributes. This formulation surfaces agentic challenges that are often invisible in single-turn evaluation: choosing questions that efficiently reduce ambiguity, preserving and applying constraints without drift, and performing cross-image comparisons that depend on the evolving candidate set rather than any single image in isolation. To operationalize this setting, we present AMIGO, an interactive benchmark centered on a simple but diagnostic loop. The model asks exactly one binary question per turn, receives Yes/No/Unsure feedback, and iteratively narrows the feasible candidate set. If the model violates protocol rules, the oracle returns Skip , revealing no information. This makes non-compliance measurable and separates wasted interaction from evidence-driven progress. AMIGO additionally allows occasional oracle inconsistencies to test whether models detect contradictions and seek verification instead of committing prematurely. We instantiate AMIGO with Guess My Preferred Dress task, where each gallery contains 6 to 40+ highly similar dress images. Dresses offer rich fine-grained variation (e.g., neckline construction, closures, seam placement, drape, and embellishments), making the task sensitive to careful observation and long-horizon constraint tracking. Contributions. (1) We introduce AMIGO, an interactive multi-image benchmark for hidden-target identification that foregrounds long-horizon planning and fine-grained cross-image grounding. (2) We define a constrained Yes/No/Unsure protocol with explicit invalid-action feedback ( Skip ) to diagnose sustained instruction following and common agentic failure modes. (3) We incorporate controlled oracle imperfections to test robustness and verification behaviors under inconsistent feedback. (4) We provide an evaluation suite that jointly measures outcomes and interaction quality, including verified identification, efficiency, protocol compliance, and trajectory-level diagnostics. Broader impact. AMIGO provides a controlled testbed for agentic VLM behaviors that matter in practice: planning under uncertainty, maintaining consistent state over extended interactions, and responding conservatively to ambiguity or contradictions. The resulting traces (candidate sets as state, questions as actions, and oracle responses, including Skip , as observations) can also support training and analysis of multimodal policies via offline learning. At the same time, hidden-target identification highlights calibration considerations: systems should avoid overconfident early commitments and remain protocol-compliant throughout long dialogues. 2 Related Work 2.1 Multi-image and multi-turn multimodal evaluation A growing set of benchmarks and datasets extends evaluation beyond single-image, single-turn QA [ 28 , 16 ] to multi-image and/or multi-turn settings. Multi-image benchmarks (e.g., MMIU [ 17 ] , MuirBench [ 23 ] , mPLUG-Owl3 [ 26 ] ) probe cross-image grounding and long visual context handling. Multi-image instruction-tuning datasets (e.g., Mantis [ 9 ] ) and multi-image multi-turn resources (e.g., MMDU [ 15 ] , MMCR [ 24 ] ) further study supervision over interleaved images and dialogue. Separately, multi-turn multimodal conversation benchmarks (e.g., MultiVerse [ 11 ] , ConvBench [ 13 ] , MMMT-IF [ 4 ] , MMCoQA [ 12 ] ) evaluate contextual coherence and instruction adherence over longer dialogues. These efforts primarily evaluate responding given a provided context, rather than interactive hidden-target identification where a model must actively uncover a user-privately selected target by asking constrained questions and tracking constraints over a long horizon. 2.2 Multimodal reasoning benchmarks General multimodal reasoning benchmarks assess whether models can integrate vision and language for multi-step inference beyond shallow cue matching. For example, EMMA [ 6 ] targets “organic” multimodal reasoning across domains, and MLLM-CompBench [ 10 ] emphasizes comparative reasoning under multimodal inputs. MMMU-Pro [ 29 ] further strengthens this line by making evaluation more resistant to text-only shortcuts and by requiring tighter visual grounding in challenging multimodal questions. These benchmarks are valuable for measuring reasoning capability on difficult perception-and-inference tasks, but they still primarily assess whether a model can solve a presented problem instance. In contrast, AMIGO evaluates whether an agentic VLM can acquire the missing information needed to solve the task: selecting informative questions over multiple turns, maintaining and revising a belief state over a candidate pool, and enforcing consistency under a constrained interaction protocol to identify a hidden target image. 2.3 Interactive clarification with hidden targets AMIGO is closely related to interactive clarification settings where an agent must uncover user intent through dialogue. InfoQuest [ 2 ] evaluates text-only agents that ask clarifying questions when critical context is hidden. ClariMM [ 19 ] extends this idea to multimodal clarification for underspecified user queries. More broadly, multimodal agent frameworks such as M3Searcher [ 27 ] and expert-guided benchmarks such as MIRAGE [ 3 ] follow a “seek information then decide” paradigm, often with retrieval or tool-mediated evidence acquisition. AMIGO differs in three ways. First, it studies hidden-target identification over a closed set of visually similar images, where the model must narrow the candidate pool via attribute queries answered with Yes/No/Unsure/ Skip feedback (which may be noisy). Second, it evaluates the interaction policy itself via trajectory-level signals such as compliance, redundancy, elimination dynamics, and contradiction detection/verification. Third, the constrained protocol yields reusable trajectories with a well-defined action space (attribute questions), observations (oracle responses), and automatically scored intermediate signals, making the data naturally suited for diagnosing and training long-horizon agentic multimodal policies. 3 Benchmark We introduce AMIGO, a benchmark for evaluating agentic VLMs on hidden-target, multi-image, multi-turn identification. AMIGO comprises a primary task, Guess My Preferred Dress , where the model must identify a user-privately selected target image from a gallery of highly similar candidates by asking discriminative attribute