SLDB: An End-To-End Heterogeneous System-on-Chip Benchmark Suite for LLM-Aided Design
Elisavet Lydia Alvanaki, Kevin Lee, Luca P. Carloni · 2025
Read SLDB: An End-To-End Heterogeneous System-on-Chip Benchmark Suite for LLM-Aided Design by Elisavet Lydia Alvanaki, Kevin Lee, Luca P. Carloni (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06376.txt.
Introduction
Elisavet Lydia Alvanaki, Kevin Lee, and Luca Carloni introduce SLDB, a benchmark suite for language-model-aided chip design, in a 2025 paper. Language models now write correct hardware description code, hinting at AI help for real design work. Yet current benchmarks stop at single components, while real chips are systems of many parts.
SLDB targets that system level. It offers ten baseline system-on-chip designs whose parts can combine into a vast number of tile-based chips through a synthetic library. Full configurations ship with the set: accelerator hookup code, communication settings, and test programs, all built for the ESP platform. The suite lets researchers judge whether models can handle integration and setup, not just small coding puzzles.
Catalog record
| CREATOR | Elisavet Lydia Alvanaki, Kevin Lee, Luca P. Carloni |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-06376
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SLDB: An End-To-End Heterogeneous System-on-Chip Benchmark Suite for LLM-Aided Design Elisavet Lydia Alvanaki, Kevin Lee, Luca P. Carloni Abstract Over the last few years, Large Language Models (LLMs) have emerged as a valuable tool for Electronic Design Automation (EDA). State-of-the-art research in LLM-aided design has demonstrated the ability of LLMs to generate syntactically correct RTL code, showcasing encouraging prospects for integrating AI into the hardware design process. A key enabler of these advancements is the availability of high-quality benchmarks to evaluate new approaches. However, existing datasets and benchmarks fall short of system-level design, as they focus primarily on component-level information and low-complexity designs. To address this gap, we introduce the System-Level Design Benchmark (SLDB), a dataset tailored for evaluating LLMs in system-level integration and configuration tasks. SLDB includes a curated benchmark suite of 10 baseline SoC designs, whose components can be combined into an exponential number of distinct tile-based SoCs through a synthetic library. The dataset provides full SoC configurations, accelerator integration code, communication parameters, and accelerator-aware system configurations, along with testing-application code, compatible with the ESP platform [ 1 ] . I Introduction Modern hardware systems are becoming increasingly complex, requiring more efficient design methodologies that balance the trade-offs between performance, energy, and design effort. This complexity has increased with the rise of System-on-Chip (SoC) architectures, which integrate multiple, often heterogeneous components into a single chip to enhance performance across many application workloads. While heterogeneous SoCs yield significant gains in performance and efficiency, their design and validation require substantial engineering effort. Optimized hardware components, designed at the register-transfer level (RTL), demand resource-intensive engineering efforts, characterized by long iteration cycles and extended time-to-market [ 2 ] [ 3 ] . These challenges are exacerbated for complex SoCs, underlining the need for more efficient design methodologies. In response to this scaling trend, agile design methodologies have emerged, incorporating modularity and design-space exploration techniques to accelerate the design process [ 4 , 5 , 6 ] . However, despite these advances, hardware design remains constrained by manual, human-driven RTL development, limiting automation in the design flow. Large Language Models (LLMs) have become a promising approach to addressing the growing complexity of hardware design. By leveraging prior designs, LLMs have shown notable results in the generation of RTL designs, making them an increasingly popular research domain in the context of agile hardware design. State-of-the-art works [ 7 , 8 , 9 , 10 ] have demonstrated the ability of LLMs to generate syntactically and often functionally correct code at the component level. As a result, two main challenges have emerged in LLM-aided design : 1) Establishing fair and comprehensive evaluation methodologies that assess LLM-generated hardware across diverse design aspects, including correctness, performance, and scalability. 2) Expanding LLM-aided design methodologies beyond individual components to complex system-level architectures. To pave the way for addressing these challenges, we introduce the System-Level Design Benchmark (SLDB). SLDB is a curated dataset that enables the evaluation of LLM performance in system-level design of heterogeneous SoCs. In the short term, SLDB provides designers with a valuable resource to assess the performance of LLMs in various system-level design tasks, such as system integration and workload-aware end-to-end system configurations. Designers may use SLDB to evaluate their LLM-aided methodologies post-synthesis ( coarse-grain evaluation ) or to evaluate the ability of these methodologies to generate modular system-level abstractions ( fine-grain evaluation ), such as integration code and system-level design parameters. Designers may also use the SLDB synthetic library to validate more complex applications, dataflows, and component interactions. In the longer term, as LLM-aided design progresses, we envision SLDB serving as a solution to benchmark LLM-generated SoC designs against SoCs included in the dataset. SLDB features: • Ten synthesized, heterogeneous SoC designs, integrating diverse workloads, accelerator-aware system configurations, and full-stack communication parameters. • A synthetic library extension that seamlessly combines integrated accelerators to generate an exponential number of distinct SoC designs. Our paper includes a comprehensive case study on LLM performance in modular SoC design using example design tasks from the open-source ESP platform [ 1 ] . This study demonstrates how using SLDB provides insights into LLM capabilities for hierarchical and scalable SoC generation. II Related work Various works have introduced benchmarks and methodologies for evaluating the generation of RTL code. Koios is an advanced benchmark suite for Deep Learning EDA research [ 11 ] . Benchmarks and datasets specifically tailored for LLM-aided hardware design have been developed. In particular, Thakur et al. propose a fine-tuning methodology alongside a testbench-based evaluation framework to assess the functional and syntax correctness of LLM-generated Verilog code of homework-based design questions [ 12 ] . VerilogEval is an open-source benchmark using Verilog instructional resources as a data source [ 13 ] . Nazzal et al. propose a dataset for systolic array DNN accelerators [ 14 ] that is based on templated generation using Gemmini [ 15 ] . Zhang et al. propose criteria for data selection along with a Verilog dataset with corresponding natural language descriptions [ 16 ] . Moving towards larger-scale designs, RTLLM is an open-source benchmark for more complex RTL components, along with a prompt engineering methodology to enhance LLM-aided RTL generation [ 17 ] . RTL-Repo is a benchmark using repository-level hierarchy for multi-file RTL projects [ 18 ] . Met-Rex is a synthesizable benchmark to evaluate reasoning LLMs based on post-synthesis performance metrics [ 19 ] . To date, efforts in benchmark development have focused primarily on component-level tasks and lack the architectural complexity necessary to evaluate system-level design methodologies. While suggested pass rates 1 1 1 Pass rates (often referred to as pass@k ) are defined as the percentage of tasks the LLM performs successfully in any of k independent runs (as defined in [ 20 ] ). In LLM-aided design, a successful task is defined as a design that is functionally correct. for LLM-generated hardware may appear promising, they fail to evaluate designs at the system level, where design parameters, interface specification, timing and communication play a critical role. Current evaluations are limited to self-contained hardware components, whereas realistic, scalable hardware systems functionally depend on the interplay between components, where specification adherence is crucial. Without a structured system-level benchmark, it remains challenging to assess whether LLM-aided design can effectively scale beyond individual functional blocks to full heterogeneous SoCs. III Background III-A System-on-Chip Architectures Modern SoC architectures feature a growing set of heterogeneous components, including specialized hardware accelerators that offer major performance and energy-efficiency gains [ 21 ] . These components impose diverse requirements in terms of area, communication, and delay, and yield distinct design-space tradeoffs. Designers have managed these trade-offs in different manners, aiming to achieve a balance between performance and design effort. In the context of accelerator integration, these tradeoffs have resulted in two dominating paradigms: loosely coupled and tightly coupled accelerators [ 22 ] . Fig. 1 : The ESP tile-based architecture [ 1 ] . Tightly coupled accelerators integrate directly with processor cores and memory hierarchies, significantly reducing latency. However, this tight integration limits flexibility, complicates scaling across workloads, and often requires custom instruction-set architecture (ISA) extensions and compiler modifications, thus increasing engineering effort. In contrast, loosely coupled accelerators (LCAs) operate as independent processing units, typically communicating with