QuArch: A Question-Answering Dataset for AI Agents in Computer Architecture
Shvetank Prakash Andrew Cheng Jason Yik Arya Tschand Radhika Ghosal, [.2em] Ikechukwu Uchendu Jessica Quaye Jeffrey Ma Shreyas Grampurohit Sofia Giannuzzi, [.2em] Arnav Balyan Fin Amin Aadya Pipersenia Yash Choudhary Ankita Nayak, [.2em] Amir Yazdanbakhsh Vijay Janapa Reddi, [.4em] Harvard Universit · 2025
Read QuArch: A Question-Answering Dataset for AI Agents in Computer Architecture by Shvetank Prakash Andrew Cheng Jason Yik Arya Tschand Radhika Ghosal, [.2em] Ikechukwu Uchendu Jessica Quaye Jeffrey Ma Shreyas Grampurohit Sofia Giannuzzi, [.2em] Arnav Balyan Fin Amin Aadya Pipersenia Yash Choudhary Ankita Nayak, [.2em] Amir Yazdanbakhsh Vijay Janapa Reddi, [.4em] Harvard Universit (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-01892.txt.
Description
Prakash and co-authors quiz language models on computer architecture in a 2025 paper. Their dataset, QuArch, holds 1,500 human-checked question-and-answer pairs spanning chip design, memory systems, and squeezing out speed. It aims to grade, and then teach, AI agents that might someday help design hardware.
The best closed model scored eighty-four percent; the best small open model managed seventy-two. Memory systems, chip-to-chip links, and benchmarking tripped models most. Fine-tuning on the dataset lifted small models by up to eight points. The authors publish the set and a leaderboard, framing it as a foundation for AI that reasons about the machines it runs on.
Catalog record
| CREATOR | Shvetank Prakash Andrew Cheng Jason Yik Arya Tschand Radhika Ghosal, [.2em] Ikechukwu Uchendu Jessica Quaye Jeffrey Ma Shreyas Grampurohit Sofia Giannuzzi, [.2em] Arnav Balyan Fin Amin Aadya Pipersenia Yash Choudhary Ankita Nayak, [.2em] Amir Yazdanbakhsh Vijay Janapa Reddi, [.4em] Harvard Universit |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-01892