ElectroVizQA: How well do Multi-modal LLMs perform in Electronics Visual Question Answering?

Pragati Shuddhodhan Meshram, Swetha Karthikeyan, Bhavya, Suma Bhat · 2024

Read ElectroVizQA: How well do Multi-modal LLMs perform in Electronics Visual Question Answering? by Pragati Shuddhodhan Meshram, Swetha Karthikeyan, Bhavya, Suma Bhat (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00102.txt.

Introduction

Pragati Meshram, Swetha Karthikeyan, Bhavya Bhavya, and Suma Bhat created ElectroVizQA in twenty twenty-four, a benchmark testing whether multimodal language models can actually see and reason about electronics diagrams. These models shine at general visual question answering but stumble on fundamental engineering problems, and specialized training data are scarce. Their data set targets digital electronics as taught in undergraduate courses.

It contains roughly six hundred twenty-six visual questions — the first benchmark of its kind for this domain, by the authors' account. They rigorously assessed how well current models understand and solve digital circuit questions from images. The paper maps capabilities and limitations in this specialized area, aiming to spur work on applying multimodal models to engineering education and close the gap with their general-domain fluency.

Catalog record

CREATORPragati Shuddhodhan Meshram, Swetha Karthikeyan, Bhavya, Suma Bhat
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-00102

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ElectroVizQA: How well do Multi-modal LLMs perform in Electronics Visual Question Answering? Pragati Shuddhodhan Meshram, Swetha Karthikeyan, Bhavya, Suma Bhat Abstract Multi-modal Large Language Models (MLLMs) are gaining significant attention for their ability to process multi-modal data, providing enhanced contextual understanding of complex problems. MLLMs have demonstrated exceptional capabilities in tasks such as Visual Question Answering (VQA); however, they often struggle with fundamental engineering problems, and there is a scarcity of specialized datasets for training on topics like digital electronics. To address this gap, we propose a benchmark dataset called ElectroVizQA specifically designed to evaluate MLLMs’ performance on digital electronic circuit problems commonly found in undergraduate curricula. This dataset, the first of its kind tailored for the VQA task in digital electronics, comprises approximately 626 visual questions, offering a comprehensive overview of digital electronics topics. This paper rigorously assesses the extent to which MLLMs can understand and solve digital electronic circuit questions, providing insights into their capabilities and limitations within this specialized domain. By introducing this benchmark dataset, we aim to motivate further research and development in the application of MLLMs to engineering education, ultimately bridging the performance gap and enhancing the efficacy of these models in technical fields. 1 1 1 Please contact the author for access to the dataset. Introduction The recent shift from single-modal models to multi-modal models aims to leverage diverse information sources by incorporating different modalities. This transition has brought about remarkable advancements in models like GPT-4o with demonstrated improvements compared to its predecessors on standard reasoning and related STEM benchmarks such as MATH ( Hendrycks et al. 2021a ) , GSM-8K ( Cobbe et al. 2021 ) , ScienceQA ( Lu et al. 2022b ) , MMLU ( Hendrycks et al. 2021b ) . As these models continue to evolve, their application to Visual Question Answering (VQA) has gained considerable attention. Extensive research has focused on VQA, where multimodal large language models (MLLM) such as InstructBlip ( Dai et al. 2023 ) , Llava ( Liu et al. 2023c ) , Sphinx ( Lin et al. 2023 ) , and GPT4-o ( OpenAI 2024 ) exhibit strong performance across various tasks in diverse domains. Recently released benchmarks like MathVista ( Lu et al. 2023 ) and systematic studies like MathVerse ( Zhang et al. 2024 ) have provided comprehensive evaluations of MLLMs over VQA tasks. Beyond these, domain-specific datasets, such as those in Llava-Med ( Liu et al. 2023b ) , JEE-Bench ( Arora, Singh et al. 2023 ) , and IconVQA ( Li et al. 2023 ) further illustrate the breadth of data utilized for fine-tuning these models to set challenging benchmarks. Building on these advancements in multi-modal learning, we focus on digital electronics, a foundational subject in engineering education and electronic design automation, where problem solving relies on a strong reasoning ability. For instance, the logical operations performed by digital gates AND, OR, NOT, XOR, XNOR have direct mathematical analogies, underscoring the fundamental nature of this subject. Beyond the reasoning aspect, answering these questions requires a systematic collation of information from tables, associated figures and text, thus posing definite challenges for multi-modal models. As students in all domains increasingly rely on MLLMs like ChatGPT, it is crucial for these models to provide reliable solutions in this fundamental field. This paper presents a manually created and curated dataset tailored to fill a benchmarking gap and evaluates MLLMs’ performance on answering these problems about digital electronic circuits. Figure 1: Example illustrating the challenges faced by LLMs in recognizing basic logic gates accurately The ElectroVizQA dataset is developed along three problem dimensions. (1) The Conceptual dimension captures fundamental concepts for solving digital electronics problems, such as Karnaugh Map (K-map), Truth Table; (2) The Visual context dimension relates to the visual elements in the dataset and spans the topics such as finite state machines (FSM), gates, encoders/decoders; and finally the Solving strategy dimension relates to the strategy required to solve the problem—factual, computational, or deep analysis requiring collating knowledge and deeper reasoning. Our dataset incorporates fine-grained question category labels derived from these problem dimensions, meticulously integrating textual and visual elements to assess multi-modal large language models in both visual and textual understanding within question-answering tasks. In addition to constructing the dataset, we benchmark LLMs and provide rigorous error analysis of their outputs. To develop this benchmark, we referenced two resources to curate the questions with figures. The first is a set of course notes 2 2 2 http://lumetta.web.engr.illinois.edu/120-S17/ece120-spring-2017-notes-for-students.pdf used in an introductory course on fundamental digital electronics for undergraduate students at a large U.S. public university and referenced with the author’s permission. The second textbook 3 3 3 https://textbookequity.org/Textbooks/TBQ˙Feher˙DigitalLogic.pdf(CreativeCommonsAttribution3.0License) covers additional topics in the domain and has a Creative Commons attribution license. Together, these textbooks provide a comprehensive coverage of the essential digital electronics topics, including foundational elements such as Karnaugh maps, truth tables, and combinational logic circuits spanning decoders, multiplexers, latches, flip-flops, counters, and finite state machines. Additionally, we utilize schemdraw 4 4 4 https://schemdraw.readthedocs.io/en/latest/index.html , an online library, to draw circuits when the visual elements were needed. We followed a systematic manual problem generation and review process to guarantee data quality. This resulted in 626 categorized questions from an initial pool of 800. This is followed by a quantitative and qualitative analysis of the outputs of state-of-the-art proprietary and open-source MLLMs on our dataset. Further, we designed prompts to identify error types and perform a critical analysis of LLMs’ performance on our dataset. Our findings reveal significant qualitative deficiencies, particularly in GPT4-o’s understanding of visual content, despite its reasonable performance with textual information. In sum, our main contributions include the following: • We propose the first benchmark for digital elecronics VQA, ElectroVizQA, which has 626 meticulously created questions with manual annotations for three primary problem dimensions. We expect this benchmark to serve as a strong and reliable test bed and to foster future research on problem-solving with MLLMs. • We conduct an extensive comparative analysis of MLLMs on our benchmark and investigate their visual and textual problem-solving capabilities. • To understand the challenges offered by our dataset, we conduct a careful manual error analysis of the MLLMs’ responses to our VQA dataset. Related Work Recently, several benchmark datasets have been developed to test the STEM reasoning and problem-solving capabilities of large language models ( Chang et al. 2024 ) mainly in the math and general science domains. GSM8k ( Cobbe et al. 2021 ) consists of grade school math word problems that require several steps of elementary arithmetic computations to solve. MATH dataset ( Hendrycks et al. 2021a ) consists of challenging competition mathematics problems. MMLU ( Hendrycks et al. 2021b ) covers 57 QA tasks including STEM domains like college mathematics, computer science, and physics. TheoremQA ( Chen et al. 2023 ) requires the application of mathematical theorems to solve questions. Further, a few benchmark datasets with visual questions for testing the multi-modal capabilities of LLMs in STEM problem-solving settings are also available. GeoQA ( Chen et al. 2021 ) contains geometric problems in Chinese middle school exams along with visual diagrams. MathVista ( Lu et al. 2023 ) includes mathematical reasoning on diagrams, logical reasoning on puzzles, statistical reasoning on functional plots,

Read the complete text