DISSECT: Diagnosing Where Vision Ends and Language Priors Begin in Scientific VLMs CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Computer vision

Dikshant Kukreja, Kshitij Sah, Karan Goyal, Mukesh Mohania, Vikram Goyal · 2026

Read DISSECT: Diagnosing Where Vision Ends and Language Priors Begin in Scientific VLMs CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Computer vision by Dikshant Kukreja, Kshitij Sah, Karan Goyal, Mukesh Mohania, Vikram Goyal (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-06250.txt.

Introduction

Kukreja and four colleagues expose a perception-integration gap in vision-language models, where models describe diagrams correctly yet fail to reason. Their DISSECT benchmark poses 12,000 questions across chemistry and biology under five input modes, including a Model Oracle mode where the model verbalizes the image before reasoning. Gaps split performance into language-prior use, visual extraction, fidelity, and integration. Across 18 models, chemistry shows lower language-prior exploitability than biology, open-source models do better from their descriptions than images, and closed-source models show no gap.

Catalog record

CREATORDikshant Kukreja, Kshitij Sah, Karan Goyal, Mukesh Mohania, Vikram Goyal
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73745
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2604.06250

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DISSECT: Diagnosing Where Vision Ends and Language Priors Begin in Scientific VLMs CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Computer vision Dikshant Kukreja, Kshitij Sah, Karan Goyal, Mukesh Mohania, Vikram Goyal Abstract . When asked to describe a molecular diagram, a Vision-Language Model correctly identifies “a benzene ring with an -OH group.” When asked to reason about the same image, it answers incorrectly. The model can see but it cannot think about what it sees. We term this the perception-integration gap : a failure where visual information is successfully extracted but lost during downstream reasoning, invisible to single-configuration benchmarks that conflate perception with integration under one accuracy number. To systematically expose such failures, we introduce DISSECT , a 12,000-question diagnostic benchmark spanning Chemistry (7,000) and Biology (5,000). Every question is evaluated under five input modes— Vision+Text , Text-Only , Vision-Only , Human Oracle , and a novel Model Oracle in which the VLM first verbalizes the image and then reasons from its own description—yielding diagnostic gaps that decompose performance into language-prior exploitation, visual extraction, perception fidelity, and integration effectiveness. Evaluating 18 VLMs, we find that: (1) Chemistry exhibits substantially lower language-prior exploitability than Biology, confirming molecular visual content as a harder test of genuine visual reasoning; (2) Open-source models consistently score higher when reasoning from their own verbalized descriptions than from raw images, exposing a systematic integration bottleneck; and (3) Closed-source models show no such gap, indicating that bridging perception and integration is the frontier separating open-source from closed-source multimodal capability. The Model Oracle protocol is both model and benchmark agnostic, applicable post-hoc to any VLM evaluation to diagnose integration failures. 1. Introduction Consider a Chemistry question that shows a structural formula and asks: “Identify the functional group present in this compound.” A state-of-the-art Vision-Language Model (VLM) answers correctly. But remove the image and provide only the question text—the same model still answers correctly, exploiting memorized associations between question phrasing and likely answers ( 18 ; 5 ) . Now consider a question showing a skeletal structure of two constitutional isomers and asking: “Which isomer has the higher boiling point, and why?” Here, removing the image causes complete failure—the model cannot reason about intermolecular forces without seeing the structures. These two scenarios represent fundamentally different evaluation regimes, yet existing benchmarks conflate them under a single accuracy number. Current multimodal benchmarks for scientific reasoning suffer from two critical blind spots. First, they cannot distinguish genuine visual understanding from language-prior exploitation —the degree to which models answer “visual” questions without seeing anything. MathVerse ( 18 ) introduced multi-version evaluation but is restricted to mathematics, the most text-exploitable STEM subject. SciVerse ( 8 ) extended to science but varies knowledge level (how much domain expertise is provided), not input modality (which channel the model actually uses). Second, when a model fails, no existing benchmark can determine whether the failure is perceptual (the model cannot see the diagram), extractive (it cannot read text embedded in the image), or integrative (it sees the content but cannot reason over it). We introduce DISSECT ( DI agnostic S eparation of S eeing, E xtracting, and C ognitive T hinking), a diagnostic benchmark that addresses both limitations through three contributions. (1) A five-mode evaluation protocol with a novel Model Oracle. Every question is evaluated under five input configurations (c.f. Figure 1 ): Vision+Text (standard), Text-Only (image removed), Vision-Only (question embedded in image), Human Oracle (human-annotated symbolic labels), and Model Oracle (the VLM first verbalizes the image, then reasons from its own description). The Model Oracle isolates a failure mode invisible to prior work: the perception-integration gap , where a model can describe what it sees but fails to use that information during reasoning. (2) The DISSECT benchmark: 12,000 questions across two irreducibly visual sciences. DISSECT comprises 7,000 Chemistry and 5,000 Biology questions, each manually verified to require visual input for a human solver. We deliberately focus on these two subjects because molecular structures, reaction mechanisms, cell diagrams, and anatomical illustrations cannot be adequately described in text alone—unlike mathematical diagrams, which are symbolically reducible and already heavily represented in existing benchmarks ( 18 ; 13 ) . (3) Diagnostic findings across 18 VLMs. Our evaluation reveals a clear divide: open-source VLMs consistently outperform themselves when reasoning from their own verbalized descriptions rather than raw images, exposing a systematic integration bottleneck. Closed-source models show no such gap, indicating that the perception-integration bridge is the frontier separating open- from closed-source multimodal reasoning. Figure 1 . The DISSECT five-mode evaluation protocol. Each question is evaluated under Vision+Text, Text-Only, Vision-Only, Human Oracle, and Model Oracle configurations. Five boxes showing the same chemistry question rendered under five different input configurations. 2. Related Work Multi-Version Evaluation. MathVerse ( 18 ) transforms each math problem into six versions by progressively moving information from text to diagram, creating a spectrum from Text Dominant to Vision Only . Their key finding—that Qwen-VL-Max scores over 5% higher without diagrams—directly motivates our work. However, MathVerse is restricted to mathematics (geometry and functions), the most text-exploitable STEM subject. Its six versions vary the information distribution between modalities but do not include an oracle to separate perception from reasoning failures. SciVerse ( 8 ) , from the same group, extends to Physics, Chemistry, and Biology with five versions. Critically, SciVerse’s decomposition axis is knowledge content —it varies how much domain knowledge is embedded in the question (Knowledge-free/lite/rich) and how much is visual (Vision-rich/only). This answers “Does the model know enough science?” but cannot answer “Is the model actually looking at the image?” because SciVerse never provides the question text without an image. DISSECT’s decomposition is orthogonal: we fix the knowledge content and vary the input channel , enabling direct measurement of language-prior exploitation via the Text-Only mode and integration failure via the Model Oracle. Chemistry and Biology Benchmarks. ChemVTS-Bench ( 9 ) evaluates MLLMs under three modes: visual-only, visual-text, and SMILES-based symbolic input. However, its symbolic mode substitutes a machine-readable string (SMILES) for the image, testing whether models can reason from formal notation—a different question from whether they can reason from their own perception, which is what our Model Oracle tests. ChemVTS-Bench also lacks a text-only baseline and therefore cannot measure language-prior exploitation. ChemVLM ( 11 ) introduces MMCR-Bench (1,000 chemistry exam questions) but evaluates under a single V+T configuration. USNCO-V ( 5 ) evaluates 40 VLMs on Chemistry Olympiad exams and includes an image-removal ablation, finding that removing images sometimes improves accuracy. This corroborates our findings but is limited to Olympiad-level questions ( \sim 400 items) with only two modes (with/without image) and no mechanism to explain why image removal helps. For Biology, no dedicated VLM benchmark with visual dependency analysis exists. MMMU ( 17 ) and EXAMS-V ( 6 ) include biology questions among broader multi-discipline collections but use single-mode evaluation with no ablation across input configurations. Language Priors in VQA and VLMs. Goyal et al. ( 7 ) demonstrated that VQA models exploit statistical regularities between question types and answer distributions, achieving high accuracy without genuinely attending to images. Agrawal et al. ( 1 ) showed this persists under distribution shift. Recent work confirms that modern VLMs inherit this vulnerability: ViLP ( 14 ) found that GPT-4o achieves

Read the complete text