CircuitSense: A Hierarchical Circuit System Benchmark Bridging Visual Comprehension and Symbolic Reasoning in Engineering Design Process

Arman Akbari, Jian Gao, Yifei Zou, Mei Yang, Jinru Duan, Dmitrii Torbunov, Yanzhi Wang, Yihui Ren, Xuan Zhang, Northeastern University, MA, USA Brookhaven National Laboratory, NY, USA · 2025

Read CircuitSense: A Hierarchical Circuit System Benchmark Bridging Visual Comprehension and Symbolic Reasoning in Engineering Design Process by Arman Akbari, Jian Gao, Yifei Zou, Mei Yang, Jinru Duan, Dmitrii Torbunov, Yanzhi Wang, Yihui Ren, Xuan Zhang, Northeastern University, MA, USA Brookhaven National Laboratory, NY, USA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-22339.txt.

Introduction

Arman Akbari and eight co-authors present CircuitSense, a benchmark testing whether multimodal language models can read engineering diagrams as engineers do. Design work moves through levels of abstraction, from system plans down to single components, and each level needs visual understanding joined with math. Their 2025 benchmark holds more than eight thousand problems, from component schematics to system block diagrams, generated by a pipeline that also derives the symbolic equations automatically, giving exact labels.

Six leading models, closed and open, were evaluated across the full workflow of perception, analysis, and design. The closed models scored very high on perception tasks like recognizing components and tracing connections. On deriving symbolic equations and analytical reasoning, though, every model fell to low scores. That gap between seeing a circuit and reasoning about it mathematically is the paper's central finding.

Catalog record

CREATORArman Akbari, Jian Gao, Yifei Zou, Mei Yang, Jinru Duan, Dmitrii Torbunov, Yanzhi Wang, Yihui Ren, Xuan Zhang, Northeastern University, MA, USA Brookhaven National Laboratory, NY, USA
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-22339

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CircuitSense: A Hierarchical Circuit System Benchmark Bridging Visual Comprehension and Symbolic Reasoning in Engineering Design Process Arman Akbari, Jian Gao, Yifei Zou, Mei Yang, Jinru Duan, Dmitrii Torbunov, Yanzhi Wang, Yihui Ren, Xuan Zhang, Northeastern University, MA, USA Brookhaven National Laboratory, NY, USA Abstract Engineering design operates through hierarchical abstraction from system specifications to component implementations, requiring visual understanding coupled with mathematical reasoning at each level. While Multi-modal Large Language Models (MLLMs) excel at natural image tasks, their ability to extract mathematical models from technical diagrams remains unexplored. We present CircuitSense , a comprehensive benchmark evaluating circuit understanding across this hierarchy through 8,006+ problems spanning component-level schematics to system-level block diagrams. Our benchmark uniquely examines the complete engineering workflow: Perception, Analysis, and Design, with a particular emphasis on the critical but underexplored capability of deriving symbolic equations from visual inputs. We introduce a hierarchical synthetic generation pipeline consisting of a grid-based schematic generator and a block diagram generator with auto-derived symbolic equation labels. Comprehensive evaluation of six state-of-the-art MLLMs, including both closed-source and open-source models, reveals fundamental limitations in visual-to-mathematical reasoning. Closed-source models achieve over 85% accuracy on perception tasks involving component recognition and topology identification, yet their performance on symbolic derivation and analytical reasoning falls below 19%, exposing a critical gap between visual parsing and symbolic reasoning. Models with stronger symbolic reasoning capabilities consistently achieve higher design task accuracy, confirming the fundamental role of mathematical understanding in circuit synthesis and establishing symbolic reasoning as the key metric for engineering competence. Project page: https://circuitsense-benchmark.github.io 1 Introduction Mathematical modeling forms the foundation of all engineering disciplines. Mechanical engineers derive equations of motion to predict system dynamics ( Shabana 2005 ) ; optical engineers compute ray transfer matrices to design lens systems ( Saleh & Teich 2007 ) . It is a general practice for electronic engineers to translate circuit schematics into symbolic transfer functions to analytically examine the different aspects of the circuit performance (e.g., noise, stability, sensitivity, etc.). For example, a phase-locked loop (PLL) with insufficient phase margin will oscillate, destroying functionality of the entire integrated electronic system, yet this catastrophic failure can only be predicted through mathematical analysis of the feedback network’s poles and zeros ( Hanumolu et al. 2004 ) . Across these domains, the ability to translate visual representations such as circuit schematics, optical layouts, or system diagrams into precise mathematical formulations determines engineering success. This visual-to-mathematical reasoning represents a fundamental capability that no current AI system can replicate. However, unlike geometry or physics problems that operate in a single representational space, engineering uniquely requires this mathematical translation across multiple levels of hierarchy, from components to subsystems to complete architectures. While Multi-modal Large Language Models (MLLMs) excel at visual perception tasks, they exhibit a critical limitation: the inability to derive symbolic equation from visual representations ( Liu et al. 2025 ; Lu et al. 2021 ; Pan et al. 2025 ) . This failure is not merely technical but fundamental: equation derivation distinguishes true engineering comprehension from pattern matching. Existing visual circuit benchmarks ( Skelic et al. 2025 ; Shi et al. 2025 ) focus primarily on recognition-based tasks like identifying component types, answering basic multiple-choice questions, or performing shallow numerical calculations. The core capability that defines circuit understanding remains untested: the ability to extract mathematical relationships from visual circuit topology that is consistent across multiple system hierarchies. We focus on analog circuits as a particularly rich domain for evaluating visual-to-mathematical capabilities. The analog design process progresses through multiple stages: topology creation (determining device types and interconnections) ( Lai et al. 2025 ; Chang et al. 2024 ; Dong et al. 2023 ) , device sizing (optimizing physical dimensions for performance) ( Wang et al. 2020 ; Lyu et al. 2017 ; Cao et al. 2024 ) , and layout design ( Xu et al. 2019 ; Kunal et al. 2019 ; Crossley et al. 2013 ) (representing circuit as geometric shapes and physical layers for fabrication), with each stage building upon the previous. The design process in analog circuits suffers from long cycles where catastrophic failures like instability, oscillation, and excessive noise often remain hidden until final verification stages. The key to accelerating analog design lies in early mathematical analysis. This preventive approach relies on translating visual schematics into mathematical models. For example, for low frequency circuit design such as operational amplifier (Op-Amp), engineers derive equations to predict frequency response ( Kamath et al. 1974 ) , input and output referred noise ( Hillbrand & Russer 2003 ) , ensure stability margins, and optimize performance trade-off. For radio frequency circuits such as low noise amplifiers and power amplifiers, people derive the circuit’s input and output impedance to ensure impedance matching and optimal power transfer ( Wang et al. 2010 ; Nguyen et al. 2004 ) . Yet no existing benchmark evaluates whether AI systems possess this circuit understanding and symbolic reasoning capability. Without examining the symbolic derivation process, we cannot assess whether models truly understand circuits or merely memorize visual patterns, and consequently, whether they can genuinely assist human designers in accelerating design cycles and catching critical failures before costly fabrication. This gap prevents us from determining if MLLMs are ready to serve as engineering tools or remain sophisticated but superficial pattern matchers. Figure 1: Multi-level hierarchy of Phase Lock Loop Design. Moreover, circuit design inherently operates across multiple levels of abstraction, requiring engineers to seamlessly navigate between system architecture and component implementation. Engineers typically begin with high-level block diagrams to architect complex systems such as analog-to-digital converters (ADCs), phase-locked loops (PLLs), or multi-stage operational amplifiers. They then systematically decompose these architectural blocks into component-level schematics, implementing each functional block using transistors, op-amps, and passive elements. Figure 1 illustrates this hierarchical decomposition through a PLL example, showing how system-level blocks translate into transistor-level implementations. Appendix A.3 provides detailed analysis questions for both this PLL and a two-stage op-amp, demonstrating the multi-level reasoning required for comprehensive circuit understanding. Despite the critical importance of this hierarchical reasoning capability, no existing benchmark evaluates the ability to bridge between block diagrams and circuit schematics. Figure 2: Benchmark overview. CircuitSense evaluates circuit systems understanding across six hierarchy levels (resistor networks to system block diagrams), three task category (Perception, Analysis with equation derivation, and Design), using both curated problems and synthetically generated circuits systems with ground-truth symbolic equations. To fill this gap, we propose CircuitSense , the first benchmark that systematically evaluates circuit understanding through hierarchical mathematical reasoning. CircuitSense comprises 8,006 problems organized across six hierarchy levels from resistor networks to system-level block diagrams with open-ended and multiple-choice formats, testing three task categories that mirror the engineering workflow: Perception, Analysis, and Design. Our benchmark combines 2,986 carefully curated problems from authoritative textbooks and documents with 5,020 synthetically generated circuits, uniquely emphasizing symbolic derivation. We introduce a hierarchical synthetic generation pipeline consisting of a circuit schematic generator with guaranteed symbolic ground-truth equations, and a block diagram generator with symbolic transfer function ground-truth. This dual approach ensures both component-level depth and system-level breadth while preventing dataset contamination. As illustrated in Figure 3 , we evaluated CircuitSense over 6 state-of-the-art MLLMs and Gemini-2.5-Pro Google DeepMind 2025 showed the best performance among all tasks. Our main

Read the complete text