MultModLM: A multi-modal benchmark for Large-Language Model based hardware schematic generation

1 st Dhruv Kulkarni, 2 nd Sai Manoj Pudukotai Dinkarrao · 2026

Read MultModLM: A multi-modal benchmark for Large-Language Model based hardware schematic generation by 1 st Dhruv Kulkarni, 2 nd Sai Manoj Pudukotai Dinkarrao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27666.txt.

Introduction

Large language models are starting to enter hardware design work. Most studies there still use text-only tasks. Multimodal schematic generation has had no proper benchmark. Dhruv Kulkarni and Sai Manoj Pudukotai Dinkarrao introduce MultModLM in 2026 to fill that gap. The task is to draw hardware schematics from RTL descriptions. RTL is the register-transfer level used to describe circuits.

The dataset holds ninety-nine diverse RTL modules. They span arithmetic, control, and state-based designs. Schematics have no single correct form, so scoring is multi-stage. Rubric scoring, self-evaluation, cross-model checks, blind review, and human validation are combined. State-of-the-art models were tested. They produced schematics that look interpretable. Functional correctness remained limited, though. One finding stands out. Language-model evaluators showed near-zero agreement with human raters. Judging by language models is unreliable in structurally precise domains, the authors conclude.

Catalog record

CREATOR1 st Dhruv Kulkarni, 2 nd Sai Manoj Pudukotai Dinkarrao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.27666

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MultModLM: A multi-modal benchmark for Large-Language Model based hardware schematic generation 1 st Dhruv Kulkarni, 2 nd Sai Manoj Pudukotai Dinkarrao Abstract Recently, Large Language models (LLMs) find application in several fields. This extends to hardware definition and synthesis. However, most works at the intersection of LLMs and hardware generation focus on text-based tasks, creating a gap for multi-modal LLMs for RTL design. In this work, we introduce MultModLM , a benchmark for evaluating LLMs on the task of generating hardware schematics from RTL (Register Transfer Level) descriptions. The dataset consists of 99 diverse RTL modules spanning arithmetic, control, and state-based designs. To address the challenges of non-unique schematic representations, we propose a multi-stage evaluation framework combining rubric-based scoring, self-evaluation, cross-model assessment, blind evaluation, and human validation to enable exhaustive evaluation. Through experiments on state-of-the-art LLMs, we observe that while models can generate visually interpretable schematics, their functional correctness remains constrained. Furthermore, we find that LLM-based evaluators exhibit near-zero agreement with human raters, revealing, as a key finding, that LLM-as-a-judge paradigms are unreliable in structurally precise domains. These findings suggest that reliable evaluation of multi-modal hardware outputs remains an open challenge, motivating the need for more robust and domain-aware evaluation methodologies, as well as tools for structural evaluation, so as to enable formal equivalence checkers. I Introduction Large language models (LLMs) find increased utilization in varied fields, including hardware development and simulation [ 1 ] . In various stages and sub-stages of chip design, large language models are being utilized to enhance performance and security measures [ 1 ] . Despite this progress, existing works have primarily focused on text-based tasks, such as code generation ( [ 3 ] , [ 4 ] ), code debugging ( [ 6 ] ), and evaluation ( [ 2 ] ). This creates a crucial gap, since no benchmark exists for evaluating LLMs on multimodal hardware tasks, in particular, the generation of hardware schematics from RTL descriptions. Given the importance of hardware schematics in education as well as industry, it is crucial to attempt to ensure schematic generation is a lightweight and simple process, since it would be applicable in lowering the barrier to hardware design as well as utilizing this attempt to evaluate the capability of LLMs within the structural domain. Traditionally, generating schematics from RTL requires expensive proprietary EDA tools such as Synopsys or Cadence, which demand significant domain expertise and are inaccessible to many practitioners and researchers owing to costs. Attempting to extend existing tools like Vivado to full image or schematic generation will require extensive licensing, strong domain expertise, and at times proprietary information. The goal of this work is to attempt to create lightweight schematic generators using LLMs, as LLMs have already taken part in the Hardware Development Life cycle. To address and study the capability or inefficiency of LLMs in hardware schematic generation requires the details of the instances where they fail as well as the extent. For visual based RTL outputs, to our knowledge, no such benchmark exists. This absence makes it impossible to systematically measure LLM performance, compare models, or identify failure modes in this domain. Therefore, this is a core bottleneck preventing further progress. To address this bottleneck, we introduce MultModLM, a novel benchmark dataset for RTL to schematic generation. The generation of schematic from RTLs task is fundamentally different from code generation, as it requires models to capture structural, functional, and visual aspects of digital circuits simultaneously. The importance of text-to-visual LLM pipelines and their cross-domain validity has been demonstrated in works such as [ 7 ] , further motivating exploration in the hardware domain. Evaluating LLM-generated schematics is non-trivial and presents unique challenges. First, schematic representations are not unique: multiple structurally different schematics can be functionally equivalent, meaning there is no single ground truth. Second, unlike code, visual outputs cannot be directly simulated or formally verified. We define hardware correctness along three axes: (1) functional consistency with the RTL description, (2) structural fidelity capturing datapath and control organization, and (3) schematic validity and legibility, including timing semantics and signal integrity. The MultModLM benchmark dataset consists of 99 RTL modules spanning arithmetic units, finite state machines, and control logic. We propose a multi-stage evaluation pipeline combining rubric-based scoring, self-evaluation, cross-model assessment, blind evaluation, and human validation. The 99-module dataset, despite being moderate in size, covers the primary circuit categories encountered in RTL design. The benchmark is not intended for deep model training but for evaluation, in particular for probing LLM capability on a structured and diverse set of circuits. Future work would delve into extending the dataset for fine-tuning purposes as well as a deep reset purpose. Our central finding is that LLM-based evaluators exhibit near-zero agreement with human raters (Cohen’s \kappa\approx 0 ), revealing a fundamental limitation of LLM-as-a-judge in this domain and motivating the need for more robust, domain-aware evaluation methodologies. The main contributions of this work are as follows: • We introduce a novel multimodal benchmark for evaluating RTL-to-schematic generation using LLMs. • We propose a structured evaluation framework combining rubric-based scoring, cross-model assessment, and human evaluation. • We provide empirical evidence that LLM-based evaluators exhibit near-zero agreement with human judgment in hardware schematic evaluation, highlighting a fundamental limitation of LLM-as-a-judge in structurally precise domains. II Benchmark Dataset II-A Dataset Construction In a typical EDA flow, schematics serve as the bridge between high-level RTL descriptions and physical layout. They provide a human-interpretable visual representation of the circuit’s logical structure, enabling design verification, debugging, and communication between engineers before committing to synthesis. Traditionally, schematics are produced by expensive proprietary tools such as Synopsys or Cadence, requiring significant domain expertise. Automating this step via LLMs is therefore a practically relevant and an underexplored problem — yet no benchmark exists to evaluate whether LLMs can do this reliably. To fill this gap, we construct MultModLM , a dataset of 99 RTL modules that serve as prompts for LLM-based schematic generation. To be explicit about the division of labor: we provide the RTL descriptions; the LLM is tasked with generating the corresponding hardware schematics. The RTL modules were sourced from open-source repositories such as OpenRISC [ 5 ] and human generation. No ground-truth schematics are provided, as schematic representations are not unique, since different synthesis strategies can produce structurally different yet functionally equivalent circuits. Correctness is instead assessed through the multi-stage evaluation framework described in Section III. The RTL designs span a range of algorithmic and structural patterns commonly encountered in digital logic design. The majority of modules implement numeric property checking circuits, including tasks such as Armstrong number detection, palindrome detection, and other number-based predicates. These designs involve arithmetic operations, conditional logic, and iterative computations over digit representations, making them particularly challenging for LLMs to construct schematics for, thus stress-testing the benchmark effectively on moderate complexity. In addition, the dataset includes finite state machine (FSM)-based detectors, arithmetic datapath components, simple counters, and control logic modules. This diversity ensures coverage of both combinational datapath logic and sequential state-based designs, which are common in RTL implementations. Each design is provided as a standalone Verilog module, with varying levels of complexity, ranging from small combinational circuits to multi-stage sequential designs. II-B Dataset Characteristics The category-wise breakdown

Read the complete text