QuArch: A Question-Answering Dataset for AI Agents in Computer Architecture

Shvetank Prakash Andrew Cheng Jason Yik Arya Tschand Radhika Ghosal, [.2em] Ikechukwu Uchendu Jessica Quaye Jeffrey Ma Shreyas Grampurohit Sofia Giannuzzi, [.2em] Arnav Balyan Fin Amin Aadya Pipersenia Yash Choudhary Ankita Nayak, [.2em] Amir Yazdanbakhsh Vijay Janapa Reddi, [.4em] Harvard Universit · 2025

Read QuArch: A Question-Answering Dataset for AI Agents in Computer Architecture by Shvetank Prakash Andrew Cheng Jason Yik Arya Tschand Radhika Ghosal, [.2em] Ikechukwu Uchendu Jessica Quaye Jeffrey Ma Shreyas Grampurohit Sofia Giannuzzi, [.2em] Arnav Balyan Fin Amin Aadya Pipersenia Yash Choudhary Ankita Nayak, [.2em] Amir Yazdanbakhsh Vijay Janapa Reddi, [.4em] Harvard Universit (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-01892.txt.

Description

Prakash and co-authors quiz language models on computer architecture in a 2025 paper. Their dataset, QuArch, holds 1,500 human-checked question-and-answer pairs spanning chip design, memory systems, and squeezing out speed. It aims to grade, and then teach, AI agents that might someday help design hardware.

The best closed model scored eighty-four percent; the best small open model managed seventy-two. Memory systems, chip-to-chip links, and benchmarking tripped models most. Fine-tuning on the dataset lifted small models by up to eight points. The authors publish the set and a leaderboard, framing it as a foundation for AI that reasons about the machines it runs on.

Catalog record

CREATORShvetank Prakash Andrew Cheng Jason Yik Arya Tschand Radhika Ghosal, [.2em] Ikechukwu Uchendu Jessica Quaye Jeffrey Ma Shreyas Grampurohit Sofia Giannuzzi, [.2em] Arnav Balyan Fin Amin Aadya Pipersenia Yash Choudhary Ankita Nayak, [.2em] Amir Yazdanbakhsh Vijay Janapa Reddi, [.4em] Harvard Universit
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-01892

Browse the full library · About the Tower of Babel