Multimodal Chip Physical Design Engineer Assistant

Yun-Da Tsai, Chang-Yu Chao, Liang-Yeh Shen, Tsung-Han Lin, Haoyu Yang, Mark Ho, Yi-Chen Lu, Wen-Hao Liu, Shou-De Lin, Mark Ren · 2025

Read Multimodal Chip Physical Design Engineer Assistant by Yun-Da Tsai, Chang-Yu Chao, Liang-Yeh Shen, Tsung-Han Lin, Haoyu Yang, Mark Ho, Yi-Chen Lu, Wen-Hao Liu, Shou-De Lin, Mark Ren (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-15872.txt.

Introduction

Yun-Da Tsai leads nine co-authors in an assistant that helps chip designers fight routing congestion and understand why it forms. Physical design tools predict congestion well enough, but their feedback is rarely readable or directly usable. The assistant uses a multimodal large language model that reads layout images, tables, and text together.

Feature creation is automated through genetic prompting guided by the model, and an interpretable preference learner weighs the trade-offs that drive congestion across the three input forms. Findings are gathered into a Design Suggestion Deck that names the strongest layout features and proposes targeted fixes. On the CircuitNet benchmark the method beats existing models on both accuracy and explainability. A case study and qualitative checks confirm the learned preferences match real design principles and can be acted on.

Catalog record

CREATORYun-Da Tsai, Chang-Yu Chao, Liang-Yeh Shen, Tsung-Han Lin, Haoyu Yang, Mark Ho, Yi-Chen Lu, Wen-Hao Liu, Shou-De Lin, Mark Ren
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-15872

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multimodal Chip Physical Design Engineer Assistant Yun-Da Tsai, Chang-Yu Chao, Liang-Yeh Shen, Tsung-Han Lin, Haoyu Yang, Mark Ho, Yi-Chen Lu, Wen-Hao Liu, Shou-De Lin, Mark Ren Abstract Modern chip physical design relies heavily on Electronic Design Automation (EDA) tools, which often struggle to provide interpretable feedback or actionable guidance for improving routing congestion. In this work, we introduce a Multimodal Large Language Model Assistant (MLLMA) that bridges this gap by not only predicting congestion but also delivering human-interpretable design suggestions. Our method combines automated feature generation through MLLM-guided genetic prompting with an interpretable preference learning framework that models congestion-relevant tradeoffs across visual, tabular, and textual inputs. We compile these insights into a "Design Suggestion Deck" that surfaces the most influential layout features and proposes targeted optimizations. Experiments on the CircuitNet benchmark demonstrate that our approach outperforms existing models on both accuracy and explainability. Additionally, our design suggestion guidance case study and qualitative analyses confirm that the learned preferences align with real-world design principles and are actionable for engineers. This work highlights the potential of MLLMs as interactive assistants for interpretable and context-aware physical design optimization. 1 Introduction The field of chip physical design encompasses critical prediction tasks such as congestion, timing, Design Rule Check (DRC), and IR Drop. These tasks, traditionally tackled using Electronic Design Automation (EDA) tools, often suffer from slow processing speeds and limited ability to provide actionable guidance for design improvement. Existing approaches using image-to-image translation models focus primarily on prediction accuracy but lack interpretability and the ability to suggest corrective measures. This paper explores the potential of Multimodal Large Language Models (MLLMs) in providing interpretable predictions along with actionable suggestions to assist engineers in optimizing their designs. Recent advances in foundation models and instruction-tuned Large Language Models (LLMs) have led to impressive progress in tasks that require reasoning, code generation, and interactive assistance. However, their application to chip physical design remains largely underexplored. In particular, current EDA solutions lack the capability to provide real-time, explainable, and multimodal assistance that integrates layout visuals, tabular metrics, and design constraints into a unified decision-making process. We argue that a Multimodal Language Model Assistant (MLLMA) tailored to chip physical design can fill this gap. The motivation for this research is to address the gap between prediction accuracy and interpretability in physical design tasks. While current models are adept at predicting various metrics such as congestion and DRC violations, they fail to explain the underlying causes or offer corrective measures. By leveraging multimodal LLMs, we aim to move beyond mere prediction and offer a framework that provides both predictions and accompanying explanations. These explanations can be compiled into a "Design Suggestion Deck" which assists engineers by presenting comprehensible, human-readable reasoning to guide their design decisions. The primary research challenge is developing an MLLM-based agent framework capable of processing multimodal data inputs, including geometric images, tabular features, and circuit graphs, and providing interpretable predictions with actionable design suggestions. Furthermore, we aim to compare this approach against baseline models to demonstrate its effectiveness in improving design quality and interpretability. Our method consists of two stages: (1) Automate Feature Engineering and Generation, and (2) Interpretable Preferences Learning to Design Suggestion Deck. In the first stage, a Multimodal Large Language Model (MLLM) agent generates and extracts numeric features from Macro Region, RUDY, and RUDY pin images and text-based features from configuration and logs. Inspired by the Genetic Instruct ( Majumdar et al. 2024 ) , a genetic algorithm incorporating mutation and crossover is applied to expand the multimodal feature pool, with a deduplication process ensuring uniqueness. Feature importance and cross validation scores are evaluated using a Random Forest model to guide feature selection and refinement. The second stage uses these features and their importance scores to create a Design Suggestion Deck. The MLLM agent translates important features into interpretable rules, providing actionable guidance for chip design optimization. Our contributions are summarized as follows: • Development of a multimodal LLM framework capable of handling geometric image features, tabular data, and circuit graph inputs for prediction tasks in physical design. • Design of an interpretable preference learning approach that generates interpretable predictions and actionable suggestions compiled into a "Design Suggestion Deck". • Evaluation of the proposed framework against the baseline models that demonstrate strong improvements in design quality and efficiency. 2 Related Work 2.1 Multimodal Language Model Assistant Recent works such as Mipha ( Zhu et al. 2024 ) , MM-ReAct ( Yang et al. 2023 ) , and MultiModal-GPT ( Gong et al. 2023 ) demonstrate that aligning visual inputs with language reasoning modules allows models to solve complex real-world tasks. Mipha, for example, focuses on efficiency by enabling multi-turn multimodal dialogue with domain grounding, while MM-ReAct leverages reasoning traces and visual memory for tool-augmented perception. In the context of chip design, prior efforts such as Chip-Chat ( Blocklove et al. 2023 ) investigate interactive co-design with conversational LLMs for hardware description generation, while ChipNeMo ( Liu et al. 2023 ) introduces domain-adapted LLMs for industrial chip tasks including EDA script generation and bug analysis. Unlike these works, which primarily focus on language-to-code translation or domain adaptation, our approach leverages MLLMs for spatially grounded reasoning over fine-grained layout features such as routing demand and macro placement. Additionally, we explicitly model design tradeoffs through learned interpretable preferences, extending the role of LLMs from passive code generators to interactive design assistants capable of justification and feedback. 2.2 Congestion Map Prediction ClusterNet ( Min et al. 2023 ) focuses on predicting routing congestion caused by netlist topology using netlist clustering and Graph Neural Networks (GNNs). By employing the Leiden algorithm to generate cohesive clusters and developing a GNN-based model to generate cluster embeddings, ClusterNet achieves improved prediction performance and congestion optimization without requiring placement and routing (P&R) tools. CircuitNet ( Chai et al. 2023 , Jiang et al. 2023 ) provides an open-source dataset for machine learning applications in VLSI CAD, facilitating the development and benchmarking of ML models for tasks like congestion prediction and design rule check (DRC) violation prediction. The dataset includes diverse samples collected from various chip designs and supports comprehensive data, including routability, timing, and power metrics. CircuitFormer ( Zou et al. 2023 ) introduces a novel approach by treating circuit components as point clouds and utilizing Transformer-based point cloud perception methods for feature extraction. This method enables direct feature extraction from raw data without any preprocessing, allows for end-to-end training, and achieves state-of-the-art performance in congestion prediction tasks on both the CircuitNet and ISPD2015 datasets, as well as in DRC violation prediction tasks on the CircuitNet dataset. MPGD ( Yang et al. 2024 ) proposes Mini-Pixel Batch Gradient Descent, a plug-and-play optimization algorithm that focuses on the most informative entries in the congestion map. By integrating this method into predictive AI models for physical design flows, MPGD enhances congestion prediction accuracy and efficiency, demonstrating its effectiveness in optimizing chip design processes. 3 Methodology Our proposed MLLM assistant consists of two core components: (1) automated feature generation and engineering, and (2) interpretable preference modeling for generating actionable design suggestions. 3.1 Automated Feature Generation and Engineering Figure 1 : Overview of the Genetic Instruct framework. (a) Spatial metrics are extracted from routability images using domain-specific rules.

Read the complete text