TPU-Gen: LLM-Driven Custom Tensor Processing Unit Generator
Deepak Vungarala, Mohammed E. Elbtity, Sumiya Syed, Sakila Alam, Kartik Pandit, Arnob Ghosh, Ramtin Zand, Shaahin Angizi · 2025
Read TPU-Gen: LLM-Driven Custom Tensor Processing Unit Generator by Deepak Vungarala, Mohammed E. Elbtity, Sumiya Syed, Sakila Alam, Kartik Pandit, Arnob Ghosh, Ramtin Zand, Shaahin Angizi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05951.txt.
Introduction
Deepak Vungarala and co-authors use language models to design custom AI chips, in work from 2025. Tensor chips speed up neural networks, but designing a good one takes rare skill and long effort. Few quality design sets exist to train an automated designer.
Their framework is the first to generate tensor units, both exact and approximate, from model guidance. A curated open data set covers many array layouts and math unit variants. Search-augmented generation grounds the model in real designs and cuts made-up output. The focus is the systolic array, the beating heart of most tensor chips. Reuse and tailoring for each workload are built into the flow.
Catalog record
| CREATOR | Deepak Vungarala, Mohammed E. Elbtity, Sumiya Syed, Sakila Alam, Kartik Pandit, Arnob Ghosh, Ramtin Zand, Shaahin Angizi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-05951
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TPU-Gen: LLM-Driven Custom Tensor Processing Unit Generator Deepak Vungarala, Mohammed E. Elbtity, Sumiya Syed, Sakila Alam, Kartik Pandit, Arnob Ghosh, Ramtin Zand, Shaahin Angizi Abstract The increasing complexity and scale of Deep Neural Networks (DNNs) necessitate specialized tensor accelerators, such as Tensor Processing Units (TPUs), to meet various computational and energy efficiency requirements. Nevertheless, designing optimal TPU remains challenging due to the high domain expertise level, considerable manual design time, and lack of high-quality, domain-specific datasets. This paper introduces TPU-Gen, the first Large Language Model (LLM) based framework designed to automate the exact and approximate TPU generation process, focusing on systolic array architectures. TPU-Gen is supported with a meticulously curated, comprehensive, and open-source dataset that covers a wide range of spatial array designs and approximate multiply-and-accumulate units, enabling design reuse, adaptation, and customization for different DNN workloads. The proposed framework leverages Retrieval-Augmented Generation (RAG) as an effective solution for a data-scare hardware domain in building LLMs, addressing the most intriguing issue, hallucinations. TPU-Gen transforms high-level architectural specifications into optimized low-level implementations through an effective hardware generation pipeline. Our extensive experimental evaluations demonstrate superior performance, power, and area efficiency, with an average reduction in area and power of 92% and 96% from the manual optimization reference values. These results set new standards for driving advancements in next-generation design automation tools powered by LLMs. I Introduction The rising computational demands of Deep Neural Networks (DNNs) have driven the adoption of specialized tensor processing accelerators, such as Tensor Processing Units (TPUs). These accelerators, characterized by low global data transfer, high clock frequencies, and deeply pipelined Processing Elements (PEs), excel in accelerating training and inference tasks by optimizing matrix multiplication [ 1 ] . Despite their effectiveness, the complexity and expertise required for their design remain significant barriers. Static accelerator design tools, such as Gemmini [ 2 ] and DNNWeaver [ 3 ] , address some of these challenges by providing templates for systolic arrays, data flows, and software ecosystems [ 4 , 5 ] . However, these tools still face limitations, including complex programming interfaces, high memory usage, and inefficiencies in handling diverse computational patterns [ 6 , 7 ] . These constraints underscore the need for innovative solutions to streamline hardware design processes. Large Language Models (LLMs) have emerged as a promising solution, offering the ability to generate hardware descriptions from high-level design intents. LLMs can potentially reduce the expertise and time required for DNN hardware development by encapsulating vast domain-specific knowledge. However, realizing this potential requires overcoming three critical challenges. First, existing datasets are often limited in size and detail, hindering the generation of reliable designs [ 8 , 9 ] . Second, while fine-tuning is essential to minimize the human intervention, fine-tuning LLMs often results in hallucinations producing non-sensical or factually incorrect responses, compromising their applicability [ 10 , 11 ] . Finally, an effective pipeline is needed to mitigate these hallucinations and ensure the generation of consistent, contextually accurate code [ 11 ] . Therefore, the core questions we seek to answer are the following– Can there be an effective way to rely on LLM to act as a critical mind and adapt implementations like Retrieval-Augmented Generation (RAG) to minimize hallucinations? Can we leverage domain-specific LLMs with RAG through an effective pipeline to automate the design process of TPU to meet various computational and energy efficiency requirements? TABLE I: Comparison of the Selected LLM-based HDL/HLS generators. Property Ours [ 10 ] [ 9 ] [ 8 ] [ 12 ] [ 13 ] [ 14 ] [ 15 ] [ 16 ] [ 17 ] [ 18 ] Function TPU Gen. Verilog Gen. AI Accel. Gen. Verilog Gen. Verilog Gen. Verilog Gen. Hardware Verf. Hardware Verf. Verilog Gen. \dagger AI Accel. Gen. Chatbot ∗ ✓ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✗ ✗ ✗ Dataset ✓ ✓(Verilog) ✗ NA NA NA ✗ ✗ ✓ ✓ ✓ Output format Verilog Verilog HLS Verilog Verilog Verilog Verilog HDL Verilog Verilog Chisel Auto. Verif. ✓ ✗ ✗ ✗ ✗ ✓ ✓ ✗ ✓ ✗ ✓ Human in Loop Low Medium Medium Medium High Low Low Low Low Low Low Fine tuning ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✗ RAG ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ∗ A user interface featuring Prompt template generation for the input of LLM. † Not applicable. To answer this question, we develop the first-of-its-kind TPU-Gen as an automated exact and approximate TPU design generation framework with a comprehensive dataset specifically tailored for ever-growing DNN topologies. Our contributions in this paper are threefold: (1) Due to the limited availability of annotated data necessary for efficient fine-tuning of an open-source LLM, we introduce a meticulously curated dataset that encompasses various levels of detail and corresponding hardware descriptions, designed to enhance LLMs’ learning and generative capabilities in the context of TPU design; (2) We develop TPU-Gen as a potential solution to reduce hallucinations leveraging RAG and fine-tuning, to align best for the LLMs to streamline the approximate TPU design generation process considering budgetary constraints (e.g., power, latency, area), ensuring a seamless transition from high-level specifications to low-level implementations; and (3) We design extensive experiments to evaluate our approach’s performance and reliability, demonstrating its superiority over existing methods. We anticipate that TPU-Gen will provide a framework that will influence the future trajectory of DNN hardware acceleration research for generations to come 1 1 1 The dataset and fine-tuned models are open-sourced. The link is omitted to maintain anonymity since the GitHub anonymous link should be under 2GB which is exceeded in this study. . II background LLM for Hardware Design. LLMs show promise in generating Hardware Description Language (HDL) and High-Level Synthesis (HLS) code. Table I compares notable methods in this field. VeriGen [ 10 ] and ChatEDA [ 19 ] refine hardware design workflows, automating the RTL to GDSII process with fine-tuned LLMs. ChipGPT [ 8 ] and Autochip [ 13 ] integrate LLMs to generate and optimize hardware designs, with Autochip producing precise Verilog code through simulation feedback. Chip-Chat [ 12 ] demonstrates interactive LLMs like ChatGPT-4 in accelerating design space exploration. MEV-LLM [ 20 ] proposes multi-expert LLM architecture for Verilog code generation. RTLLM [ 21 ] and GPT4AIGChip [ 9 ] enhance design efficiency, showcasing LLMs’ ability to manage complex design tasks and broaden access to AI accelerator design. To the best of our knowledge, GPT4AIGChip [ 9 ] and SA-DS [ 18 ] are a few initial works focus on an extensive framework specifically aimed at the generation of domain-specific AI accelerator designs where SA-DS focus on creating a dataset in HLS and employ fine-tuning free methods such as single-shot and multi-shot inputs to LLM. Other works for hardware also include creation of SPICE circuits [ 22 , 23 ] . However, the absence of prompt optimization, tailored datasets, model fine-tuning, and LLM hallucination pose a barrier to fully harnessing the potential of LLMs in such frameworks [ 19 , 18 ] . This limitation confines their application to standard LLMs without fine-tuning or In-Context Learning (ICL) [ 19 ] , which are among the most promising methods for optimizing LLMs [ 24 ] . Retrieval-Augmented