MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understanding

Qian Kou, Xiaofeng Shi, Yulin Li, Xiaosong Qiu, Xinyang Wang, Hua Zhou, Cao Dongxing · 2026

Read MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understanding by Qian Kou, Xiaofeng Shi, Yulin Li, Xiaosong Qiu, Xinyang Wang, Hua Zhou, Cao Dongxing (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30794.txt.

Introduction

Kou and colleagues introduce MechVQA, the first dataset for mechanical drawing understanding by multimodal large language models. Drawings stay brittle for these models, as dense annotations and weak domain knowledge make cues easy to miss. MechVQA holds 3.3 thousand pictures with 21 thousand question-answer pairs across ten tasks in Recognition, Reasoning, and Judging. Built through a semi-automated pipeline with quality control, it offers a testbed for real-world drawings. The authors build MechVL through multi-stage training, beating the closed-source baseline by over seven and a half points.

Catalog record

CREATORQian Kou, Xiaofeng Shi, Yulin Li, Xiaosong Qiu, Xinyang Wang, Hua Zhou, Cao Dongxing
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75753
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.30794

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understanding Qian Kou, Xiaofeng Shi, Yulin Li, Xiaosong Qiu, Xinyang Wang, Hua Zhou, Cao Dongxing Abstract Multimodal Large Language Models (MLLMs) have demonstrated significant achievements in general visual question answering (VQA) tasks. However, they remain brittle on mechanical engineering drawings, where high annotation density and weak domain knowledge, compounded by unreliable spatial relation reasoning under strict projection rules and geometric constraints, make decisive cues easy to miss and frequently lead to wrong answers. To bridge this gap, we introduce the first comprehensive mechanical drawing understanding dataset, MechVQA , created through a semi-automated construction and quality-control pipeline. MechVQA contains 3.3k high-density pictures with 21K question–answer pairs, spanning 10 different fine-grained tasks across three capability levels: Recognition , Reasoning , and Judging , providing a testbed to evaluate and improve MLLM understanding on real-world mechanical drawings. On top of MechVQA , we then develop the MechVL model through a multi-stage training paradigm, building a strong domain-specialized baseline. Extensive experimental results demonstrate that MechVL outperforms the strongest closed-source baseline by 7.57 percentage points on the MechVQA total score, significantly enhancing mechanical drawing understanding ability and providing a reusable foundation for deploying MLLMs in mechanical design and inspection scenarios. † † affiliationnotice: Equal contribution 1 Introduction Mechanical engineering drawings are the primary medium for communicating geometry, tolerances, and assembly intent in mechanical design and inspection. Unlike natural images, a drawing encodes semantics through a compact, standardized graphical language that combines orthographic multi-view projections, dense dimensioning, section views, symbolic notations, and structured textual content. As illustrated in Figure 1 (a), understanding such drawings requires more than generic visual perception: a model must (i) recognize high-density dimensions, callouts, and domain-specific symbols, (ii) reason about spatial relations under projection rules via cross-view feature correspondence, and (iii) apply drafting standards to interpret conventions such as geometric tolerances. Despite rapid progress in Multimodal Large Language Models (MLLMs), general-purpose models remain brittle on mechanical drawings ( 7 ; 47 ; 38 ) . The dominant failure mode is not merely optical recognition. High annotation density and symbol clutter make decisive cues easy to miss, and weak domain priors with unreliable spatial relation reasoning under strict projection rules and geometric constraints often yield structurally inconsistent interpretations and incorrect answers. Similar gaps between foundation models and expert-level performance have been observed across scientific and engineering domains, motivating domain-centric benchmarks that probe capabilities beyond generic VQA. ( 8 ; 22 ) Figure 1 : Overview of Mechanical Drawing Understanding and MechVQA. (a) Representative challenges , including high-density annotation recognition, projection-consistent spatial correspondence across views, and standards-aware interpretation of domain-specific symbols, specifications, and tables. (b) MechVQA task taxonomy organized into three capabilities ( Recognition, Reasoning, Judging ) with fine-grained subtasks, accompanied by representative question–answer examples grounded in mechanical drawings. There remains a lack of dedicated data that systematically covers comprehensive mechanical drawing understanding. Existing multimodal benchmarks in adjacent engineering settings are complementary but scoped to specific slices, such as rulebook-grounded requirements QA, blueprint symbol recognition, or AEC floor-plan literacy, and engineering drawing analysis is often only a small, non-open component within broader suites ( 10 ; 37 ; 21 ; 32 ) . Consequently, they do not provide a unified evaluation of mechanical part and complex assembly drawings that jointly stresses structured perception, multi-view consistency, and engineering-grade reasoning. In this work, we study mechanical drawing understanding with MLLMs under orthodox drafting conventions, focusing on real part and complex assembly drawings collected from publicly available textbooks, professional handbooks, and design platforms. We introduce MechVQA , a benchmark containing 3.3K high-quality drawings and 21K question-answer pairs. MechVQA is organized into three capability axes— Recognition , Reasoning , and Judging —and further decomposed into 10 fine-grained subtasks that jointly cover the core aspects of mechanical drawing understanding. Questions are also stratified into three difficulty levels, enabling systematic evaluation from direct perception to expert-level inference. Figure 1 (b) presents representative examples for all subtasks, and Table 1 compares MechVQA with general and CAD/mechanical VQA benchmarks. Building on MechVQA, we establish MechVL , a strong domain-specialized baseline trained through a multi-stage post-training pipeline. We first perform supervised instruction tuning (SFT) to obtain a reference policy for mechanical drawing understanding, and then further improve reliability and task performance through reinforcement learning with DAPO ( 46 ) . A key design is a taxonomy-aligned reward scheme that directly optimizes answer correctness, output format compliance, and explanation quality. Concretely, we combine three complementary rewards: (i) a format reward to enforce strict output structure and schema validity, (ii) an accuracy reward for factual correctness against ground-truth answers with numeric- and unit-sensitive handling, and (iii) a quality reward from an LLM-as-a-Judge that evaluates responses along the axes of Logic , Normativity , and Professionalism ( 5 ; 15 ) . This design directly targets common failure cases of SFT-only baselines on dense mechanical drawings, including missed decisive annotations, violated multi-view consistency, and numerically plausible but constraint-inconsistent results. Across extensive experiments, MechVL achieves consistent improvements over strong general-purpose MLLMs and surpasses closed-source baselines by 6% on the MechVQA total score, providing a reusable foundation for drawing-centric mechanical design and inspection workflows. The main contributions of this work are summarized as follows: • We introduce MechVQA , a benchmark for mechanical drawing understanding built from real part and assembly drawings, organized into three capability axes— Recognition , Reasoning , and Judging —with 10 fine-grained subtasks and three difficulty levels. • We establish MechVL , a domain-specialized baseline trained with multi-stage post-training, including supervised fine-tuning and DAPO-based self-play reinforcement learning with taxonomy-aligned rewards. • We provide extensive benchmarking and ablation results showing that domain-specialized post-training substantially improves performance on dense mechanical drawings, especially for cross-view reasoning, constraint-sensitive inference, and standards-aware judgment. Datasets Images Questions CAD/Mech 3D views Task Focus / Features VQA ( 4 ) 250,000 750,000 \times \times Gen. VL Grounding COCO-QA ( 34 ) 123,287 117,684 \times \times Basic Object/Color Recognition MMBench ( 24 ) 2,974 2,974 \times \times Comp. Reasoning Eval. VaseVQA ( 11 ) 31,773 93,544 \times \times 2D Archaeology Analysis VaseVQA-3D ( 48 ) 664 4,460 \times \checkmark 3D Archaeology Analysis AECV-Bench ( 21 ) 120 192 \checkmark \times AEC Drawing & Spatial OCR MechBench ( 35 ) - - \checkmark \times Physics & Mech. Components DesignQA ( 10 ) - 1,451 \checkmark \checkmark CAD Design Compliance TriView2CAD ( 25 ) 609,000 160,000 \checkmark \checkmark Ortho-Projection Reasoning FCM ( 32 ) 378 1,000+ \checkmark \checkmark Design to Manufacturing BlueprintSymVL ( 37 ) 200 200 \checkmark \times Blueprint Symbol Recognition MechVQA(Ours) 3,281 20,778 \checkmark \checkmark Mechanical Drawing Understanding Table 1: Comparison of VQA Datasets: General vs. CAD/Mechanical Domains. 2 Related Work MLLMs and Visual Question Answering. Pre-trained on large-scale multimodal corpora, Multimodal LLMs excel in general tasks like image–text retrieval and VQA. Visual Instruction Tuning further enhances their instruction-following capabilities, as demonstrated by LLaVA, MiniGPT-4, and Gemini 1.5 ( 23 ; 52 ; 14 ) . However, general-purpose models often lack domain-specific drafting knowledge (e.g., projection conventions and tolerances), causing misinterpretations in engineering analysis. Traditional VQA benchmarks focus on everyday scenes ( 4 ) , leaving a gap in engineering and CAD domains. Recent efforts have begun addressing this: CReFT-CAD ( 25 )

Read the complete text