QHackBench: Benchmarking Large Language Models for Quantum Code Generation Using PennyLane Hackathon Challenges

Abdul Basit 1 , Minghao Shao 1 , Haider Asif 1,2 , Nouhaila Innan 1,2 , Muhammad Kashif 1,2 , Alberto Marchisio 1,2 , Muhammad Shafique 1,2 · 2025

Read QHackBench: Benchmarking Large Language Models for Quantum Code Generation Using PennyLane Hackathon Challenges by Abdul Basit 1 , Minghao Shao 1 , Haider Asif 1,2 , Nouhaila Innan 1,2 , Muhammad Kashif 1,2 , Alberto Marchisio 1,2 , Muhammad Shafique 1,2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-20008.txt.

Description

Basit and co-authors present QHackBench, a 2025 benchmark testing how well large language models write quantum code. The test problems come from real PennyLane hackathon challenges, graded for working code, valid syntax, and successful runs across difficulty levels. Models were tried with plain prompting and with retrieval-augmented generation, where the model can look up library material first.

Retrieval helped roughly as much as a specially enriched PennyLane data set, above all on hard algorithmic tasks. A multi-agent pipeline that critiques and repairs wrong answers lifted success rates further. The authors promise to release the benchmark, framework, and results openly. The work targets AI help for quantum programming, a field where expert coders are scarce.

Catalog record

CREATORAbdul Basit 1 , Minghao Shao 1 , Haider Asif 1,2 , Nouhaila Innan 1,2 , Muhammad Kashif 1,2 , Alberto Marchisio 1,2 , Muhammad Shafique 1,2
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-20008

Browse the full library · About the Tower of Babel