MHRC-Bench: A Multilingual Hardware Repository-Level Code Completion Benchmark

Qingyun Zou, Jiahao Cui, Nuo Chen, Bingsheng He, Weng-Fai Wong · 2026

Read MHRC-Bench: A Multilingual Hardware Repository-Level Code Completion Benchmark by Qingyun Zou, Jiahao Cui, Nuo Chen, Bingsheng He, Weng-Fai Wong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03708.txt.

Description

Qingyun Zou and four co-authors present MHRC-Bench, the first benchmark for repository-level code completion in hardware description languages. Language models complete software code well, but chip design code was largely ignored by existing benchmarks, even though hardware projects also span many interdependent files. The benchmark includes separate training and evaluation parts and covers three major hardware coding styles.

Each completion target carries labels about code structure and hardware meaning, derived from syntax tree analysis. The team ran a broad evaluation of current models on the evaluation set and analyzed the results. A shared hardware benchmark matters because chip teams cannot judge coding assistants using software tests alone; the languages, idioms, and failure costs of hardware design are quite different.

Catalog record

CREATORQingyun Zou, Jiahao Cui, Nuo Chen, Bingsheng He, Weng-Fai Wong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-03708

Browse the full library · About the Tower of Babel