QHackBench: Benchmarking Large Language Models for Quantum Code Generation Using PennyLane Hackathon Challenges
Abdul Basit 1 , Minghao Shao 1 , Haider Asif 1,2 , Nouhaila Innan 1,2 , Muhammad Kashif 1,2 , Alberto Marchisio 1,2 , Muhammad Shafique 1,2 · 2025
Read QHackBench: Benchmarking Large Language Models for Quantum Code Generation Using PennyLane Hackathon Challenges by Abdul Basit 1 , Minghao Shao 1 , Haider Asif 1,2 , Nouhaila Innan 1,2 , Muhammad Kashif 1,2 , Alberto Marchisio 1,2 , Muhammad Shafique 1,2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-20008.txt.
Description
Basit and co-authors present QHackBench, a 2025 benchmark testing how well large language models write quantum code. The test problems come from real PennyLane hackathon challenges, graded for working code, valid syntax, and successful runs across difficulty levels. Models were tried with plain prompting and with retrieval-augmented generation, where the model can look up library material first.
Retrieval helped roughly as much as a specially enriched PennyLane data set, above all on hard algorithmic tasks. A multi-agent pipeline that critiques and repairs wrong answers lifted success rates further. The authors promise to release the benchmark, framework, and results openly. The work targets AI help for quantum programming, a field where expert coders are scarce.
Catalog record
| CREATOR | Abdul Basit 1 , Minghao Shao 1 , Haider Asif 1,2 , Nouhaila Innan 1,2 , Muhammad Kashif 1,2 , Alberto Marchisio 1,2 , Muhammad Shafique 1,2 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-20008