QASM-Eval: A Dataset to Train and Evaluate LLMs on OpenQASM-3 Beyond Quantum Circuits
Zhenxiao Fu, Lei Jiang, Fan Chen · 2026
Read QASM-Eval: A Dataset to Train and Evaluate LLMs on OpenQASM-3 Beyond Quantum Circuits by Zhenxiao Fu, Lei Jiang, Fan Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30358.txt.
Introduction
Zhenxiao Fu, Lei Jiang and Fan Chen introduce QASM-Eval, a full data set to train and test LLMs on OpenQASM 3, aimed at hardware code. This needs hardware skills past gate lists: mid-circuit checks and feedback, timing, and pulse wave control. QASM-Eval holds a test set of 1,200 tasks and a train set of over 4,000 tasks, covering logic, timing, pulse control, and more. Tuned Llama-3-8B beats zero-shot GPT-5.6-Terra, while tuned Llama-3-70B hits about 61 percent pass@1 and nears few-shot GPT-5.6-Terra.
Catalog record
| CREATOR | Zhenxiao Fu, Lei Jiang, Fan Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58087 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30358
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
QASM-Eval: A Dataset to Train and Evaluate LLMs on OpenQASM-3 Beyond Quantum Circuits Zhenxiao Fu, Lei Jiang, Fan Chen Abstract Quantum computing remains in the Noisy Intermediate-Scale Quantum (NISQ) era, where the performance is highly constrained to noise. Addressing the limitation often requires hardware-facing capabilities beyond gate-sequence circuit specification, including mid-circuit measurement and classical feedback for quantum error correction (QEC), precise timing control for dynamical decoupling (DD), and pulse-level waveform access for calibration. OpenQASM 3 was introduced to expose exactly these capabilities, providing a hardware-level programming interface. However, despite the rapid progress of large language models in code generation, there is still no dataset specifically designed to train and evaluate LLMs on OpenQASM 3 programs that involve its advanced hardware-oriented features. To address this gap, we introduce QASM-Eval, the first comprehensive dataset designed to train and evaluate LLMs on OpenQASM 3. Rather than focusing on quantum algorithm design or reasoning, QASM-Eval explicitly targets the language’s hardware-facing features. QASM-Eval comprises an expert-verified test set of 100 tasks and a training set of 4,000 tasks, systematically covering classical logic, timing scheduling, pulse control, and complex real-world workflows. To automatically validate generated programs, we check syntax, quantum states and program timeline using an extended verifier. Our evaluation reveals that while state-of-the-art LLMs struggle heavily in OpenQASM 3 coding tasks, targeted fine-tuning on QASM-Eval yields significant gains. Specifically, fine-tuning Llama-3-8B approaches zero-shot GPT-5.2 performance, while Llama-3-70B achieves an 85% overall pass@1, outperforming few-shot-augmented GPT-5.2. QASM-Eval provides a crucial benchmark and training foundation to accelerate the development of reliable LLM assistants for hardware-facing quantum programming in NISQ era. Data and code: https://github.com/fuzhenxiao/QASM-Eval 1 Introduction Quantum computing has shown advantage in domains such as chemical simulation [ 12 , 32 ] , optimization [ 1 ] , and quantum machine learning [ 13 , 7 ] . Yet practical quantum hardware remains in the Noisy Intermediate-Scale Quantum (NISQ) regime [ 15 , 38 ] ,where quantum processors still suffer from quantum noise, a stochastic disturbance corrupting quantum states and causing computational deviations. Although numerous approaches exist to mitigate the effects of noise, such as quantum error correction (QEC) [ 19 ] , dynamical decoupling (DD) [ 28 ] , and routine calibration [ 29 ] , each of them requires specific, low-level hardware controls. First, QEC relies on mid-circuit measurement and runtime classical computation/feedback [ 42 ] . Second, intervention methods such as DD are highly dependent on precise control over gate timing [ 10 ] . Third, because qubits naturally drift over time and calibrated gate fidelities keep degrading [ 9 , 39 ] , sustaining performance requires not only routine calibration but also pulse-level access to actively tune control waveforms. However, existing high-level quantum programming tools such as Qiskit [ 22 ] , Cirq [ 17 ] , and PennyLane [ 5 ] , lack comprehensive support for these fine-grained hardware controls. OpenQASM 3 [ 16 ] addresses these limitations by serving as a hardware-aware intermediate representation that links algorithms to physics. Unlike high-level tools, OpenQASM 3 exposes hardware instructions that directly fulfill the diverse operations required for noise mitigation. First, it supports hardware-embedded classical logic and control flow, enabling runtime mid-circuit operations. Second, it introduces explicit gate timing and scheduling constructs that allow dynamic operation duration, alignment, and delay insertion. Finally, OpenQASM 3 further extends to pulse-level control, allowing developers to describe or tune physical control waveforms directly, thus providing users with a means to actively manage calibration details. Taken together, these capabilities make OpenQASM 3 a key enabler for improving quantum-computing performance in the NISQ era. Given the increasing complexity and features of OpenQASM 3, as well as the rapid progress of Large Language Models (LLMs) for code generation [ 24 , 44 , 36 , 25 ] , using LLMs to assist OpenQASM programming is a natural next step. However, this direction is currently limited by the lack of appropriate datasets. Existing OpenQASM-related resources fall into two unsatisfactory categories. Some datasets, such as Veri-Q [ 14 ] and QASMBench [ 33 ] , were designed to benchmark quantum algorithms or hardware platforms rather than to support LLM training or evaluation. Others, such as QCircuitBench [ 47 ] and Agent-Q [ 23 ] , remain confined to gate-sequence circuit generation and do not cover the critical features of OpenQASM 3, including classical logic, timing scheduling, and pulse control. As summarized in Table 1 , no existing resource simultaneously targets LLM-based code generation and captures the core features that define OpenQASM 3’s central role in improving NISQ-era quantum-computing performance. OpenQASM datasets LLM-targeted Classical Logic Timing Scheduling Pulse Control Our work (QASM-Eval) \checkmark \checkmark \checkmark \checkmark QCircuitBench [ 47 ] \checkmark \times \times \times Agent-Q [ 23 ] \checkmark \times \times \times Veri-Q [ 14 ] \times \times \times \times QASMBench [ 33 ] \times \times \times \times Table 1: Comparison of representative OpenQASM-related resources by whether they target LLM code generation and whether they cover major OpenQASM 3 feature dimensions. To address these gaps, we propose QASM-Eval , a dataset of OpenQASM 3 coding tasks where critical logic segments are replaced by natural language prompts for LLMs to complete, paired with canonical solutions. Our main contributions are: • We construct an OpenQASM 3 task suite for both LLM training and evaluation. We release a test set with 100 tasks, a training set with 4000 tasks, and two targeted fine-tuned models. The dataset covers key OpenQASM 3 capabilities, including classical logic/control flow, timing constraints, and pulse-level control. To validate these features, we extend existing toolchains with new OpenQASM 3 support so that generated code can be automatically verified for syntactic, semantic, and scheduling constraints. • Using the QASM-Eval test set, we analyze the limitations of current models on OpenQASM 3 tasks and evaluate the effectiveness of our training data for fine-tuning: on Llama-8B and Llama-70B, our fine-tuning improves pass@1 by 28–58%; the fine-tuned 8B model approaches the zero-shot performance of GPT-5.2, while the fine-tuned 70B model exceeds few-shot-augmented GPT. 2 Related Work Quantum Computing in the NISQ Era Practical quantum computing remains constrained by the NISQ regime, where device performance is limited by quantum noises originated from various sources, such as state-preparation and measurement (SPAM) errors [ 43 ] , imperfect gate implementations [ 41 ] , crosstalk [ 2 ] and decoherence [ 30 ] , which accumulate as circuits deepen. A broad line of work therefore focuses on noise mitigation and suppression. Quantum error correction (QEC) [ 19 , 42 ] provides a principled route to fault tolerance by encoding logical qubits into multiple physical qubits, checking the status of qubit via mid-circuit measurement, and locating potential errors via classical computation [ 3 , 6 ] . Dynamical decoupling (DD) [ 28 , 10 ] reduces noise by inserting carefully timed pulse sequences to average out low-frequency noise, which makes its highly sensitive to precise control over operation timing and spacing, especially in dynamic circuits. Calibration is also critical. Although providers routinely recalibrate devices to maintain fidelity [ 29 ] , qubit frequencies, gate parameters, and readout characteristics still keep drifting between calibrations [ 9 , 39 ] . Therefore, advanced users may additionally require direct access to customized control waveforms [