Qiskit QuantumKatas: Adapting Microsoft’s Quantum Computing Exercises for LLM Evaluation

Juan Cruz-Benito, Ismael Faro · 2026

Read Qiskit QuantumKatas: Adapting Microsoft’s Quantum Computing Exercises for LLM Evaluation by Juan Cruz-Benito, Ismael Faro (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27210.txt.

Introduction

Cruz-Benito and Faro adapt a well-known quantum computing curriculum for testing language models. They move Microsoft QuantumKatas from Q sharp to Qiskit and add a test frame with prompts, reference solutions, and deterministic checks by classical simulation. The set holds 350 tasks in 26 groups, from basic gates to Grover search, error correction, key distribution, and quantum games. Across 39200 runs with 16 models, scores vary widely, models handle known algorithms far better than problem encoding, and chain-of-thought prompting helps only some models.

Catalog record

CREATORJuan Cruz-Benito, Ismael Faro
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75719
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.27210

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Qiskit QuantumKatas: Adapting Microsoft’s Quantum Computing Exercises for LLM Evaluation Juan Cruz-Benito, Ismael Faro Abstract We adapt Microsoft’s QuantumKatas—a well-established quantum computing curriculum—from Q# to Qiskit, the most widely-adopted quantum computing framework, and package it with an evaluation framework for systematic LLM assessment. The resulting benchmark comprises 350 tasks across 26 categories, spanning fundamental gates through advanced algorithms (Grover’s, Simon’s, Deutsch-Jozsa), error correction, key distribution, and quantum games. Each task includes a natural language prompt, canonical solution, and deterministic test verification via classical circuit simulation. By building on the QuantumKatas’ proven pedagogical design rather than creating tasks from scratch, we inherit a principled difficulty progression and comprehensive concept coverage while contributing the framework adaptation, evaluation infrastructure, and empirical analysis. We evaluate 16 LLMs across 7 prompting configurations—a total of 39,200 model runs—to demonstrate the benchmark’s utility. Three key findings emerge: (1) the benchmark effectively differentiates model capabilities, with best-configuration pass rates ranging from 32.3% to 83.1% and a 26.1 pp average gap between frontier and open-source models; (2) models perform well at implementing known algorithms (SimonsAlgorithm 82.1%, BasicGates 81.6%) but struggle with problem encoding (SolveSATWithGrover 34.4%, DistinguishUnitaries 40.0%); and (3) chain-of-thought prompting shows a modestly bimodal effect—it is the best strategy for three models (two of them explicitly reasoning-tuned per vendor documentation) but degrades performance for the rest, leaving it mid-pack in aggregate (56.3% mean) behind few-shot-5 (57.8%). We release the benchmark, evaluation framework, and baseline results to support research on LLM capabilities in quantum computing. Keywords Large Language Models \cdot Quantum Computing \cdot Qiskit \cdot Benchmark \cdot Code Generation 1 Introduction Large language models (LLMs) have demonstrated strong code generation capabilities across many programming languages and domains ( 13 ; 6 ) , yet their performance on specialized scientific computing—particularly quantum computing—remains underexplored. Quantum computing poses a unique challenge: its fundamentally different computational paradigm requires understanding of superposition, entanglement, and measurement, concepts with no direct classical analogue. Existing code generation benchmarks target general programming (HumanEval ( 13 ) , MBPP ( 6 ) ), data science (DS-1000 ( 31 ) ), or software engineering (SWE-bench ( 29 ) ). Quantum-specific benchmarks have begun to emerge—Qiskit HumanEval ( 49 ) provides 150+ hand-curated tasks and QuantumBench ( 35 ) offers multiple-choice questions—but there remains a need for larger-scale, pedagogically-structured benchmarks that enable fine-grained analysis of quantum programming capabilities. We introduce the Qiskit QuantumKatas benchmark—a translation of Microsoft’s QuantumKatas ( 33 ) from Q# to Qiskit ( 28 ; 3 ) , packaged with an evaluation framework for systematic LLM assessment. The original QuantumKatas are a well-established educational resource for learning quantum computing through hands-on programming, and we build directly on their design: From the QuantumKatas we inherit 350 tasks across 26 categories (basic gates through Grover’s search and quantum error correction) with a pedagogical progression that supports fine-grained capability assessment. Our contributions on top of this foundation are: (i) a complete Qiskit translation ( 48 ) with an evaluation pipeline featuring deterministic verification via classical circuit simulation (statevector comparison), multi-provider LLM support, and configurable prompting strategies; (ii) a large-scale empirical study of 16 models across 7 prompting configurations (39,200 runs), including prompting-strategy analysis and fine-grained profiling across 26 topics; and (iii) analytical contributions: solution-diversity analysis via AST similarity, category-independence assessment, normalized difficulty metrics, and evidence that chain-of-thought prompting helps reasoning-tuned models but hurts others in this domain. To validate the benchmark, we evaluate 16 LLMs across 7 prompting configurations. Best-configuration pass rates span 32.3% to 83.1%, and category-level analysis reveals that models implement known algorithms well (SimonsAlgorithm 82.1%, BasicGates 81.6%) but struggle with problem encoding (SolveSATWithGrover 34.4%, DistinguishUnitaries 40.0%). We also find that chain-of-thought prompting is modestly bimodal: it is the best strategy for three models (two of them explicitly reasoning-tuned, GPT-5.3-Codex and Gemini 3.1 Pro, plus Gemma 4 26B-A4B) but degrades performance for the majority, leaving it mid-pack in aggregate (56.3% mean) behind few-shot-5 (57.8%). 2 Related Work 2.1 Code Generation Benchmarks Several benchmarks have been developed to evaluate LLM code generation capabilities. HumanEval ( 13 ) introduced 164 Python programming problems with unit tests. MBPP ( 6 ) expanded this with 974 crowd-sourced Python tasks. More recent benchmarks like DS-1000 ( 31 ) focus on data science libraries, while SWE-bench ( 29 ) evaluates real-world software engineering tasks from GitHub issues. 2.2 Scientific Computing and Reasoning Benchmarks Domain-specific benchmarks have also emerged: SciCode ( 47 ) evaluates scientific programming across physics, chemistry, and biology, while GPQA ( 42 ) and MATH ( 25 ) test scientific and mathematical reasoning without code generation. CURIE ( 15 ) extends this multitask direction to long-context scientific reasoning across ten tasks spanning materials science, physics, quantum computing, and biology, with the best evaluated model reaching only 32% accuracy. In the adjacent quantum-chemistry domain, QuantumChem-200K ( 54 ) provides a 200K-molecule dataset for fine-tuning LLMs on DFT-level property prediction, illustrating the quantum-flavored benchmarking effort outside quantum computing proper. Quantum computing—which requires both scientific reasoning and specialized programming—has received comparatively limited attention despite its growing importance. 2.3 Quantum Computing and LLMs IBM’s Qiskit Code Assistant ( 26 ; 17 ) represents an effort to create domain-specific LLMs for quantum computing. Several benchmarks have emerged for evaluating LLMs on quantum tasks, which we organize by what they measure. Code generation benchmarks evaluate the ability to produce working quantum programs. Qiskit HumanEval ( 49 ) introduced 150+ hand-curated Qiskit tasks with difficulty ratings, and Qiskit HumanEval-Hard ( 18 ) raised the bar by stripping import statements and boilerplate. QCoder ( 34 ) draws from quantum computing programming contest problems, emphasizing domain-specific metrics like circuit depth, while QuanBench ( 24 ) evaluates 44 Qiskit tasks using both functional correctness (Pass@K) and quantum semantic equivalence via process fidelity—finding that even the best models achieve below 40% Pass@1. Multi-framework extensions such as QuanBench+ ( 46 ) broaden this evaluation to aligned tasks across Qiskit, PennyLane, and Cirq, and PennyLang ( 8 ) releases a PennyLane-centric corpus of over 3,000 code samples paired with retrieval-augmented prompting. Circuit and algorithm design benchmarks focus on quantum circuit synthesis. QCircuitBench ( 52 ) provides 120K+ data points across 25 algorithms for AI-driven circuit generation, and QHackBench ( 9 ) uses PennyLane hackathon challenges for an alternative framework perspective. StabilizerBench ( 40 ) narrows the focus to AI-assisted quantum error correction, providing tasks for stabilizer-circuit synthesis verifiable via the Gottesman–Knill formalism. Conceptual understanding benchmarks assess knowledge without requiring code. QuantumBench ( 35 ) provides approximately 800 multiple-choice questions spanning nine areas of quantum science, and QC-Bench ( 1 ) extends this knowledge-evaluation direction to over 6,000 expert-level questions covering quantum algorithms, error correction, and security protocols, evaluated across 31 LLMs; the related Quantum-Audit preprint ( 2 ) additionally probes false-premise detection and topic-specific reasoning gaps across 26 models. Adjacent quantum-LLM directions target related modalities or skills. QCalEval ( 12 ) benchmarks vision-language models on the interpretation of quantum-hardware calibration plots, and QuantumQA ( 41 ) pairs a physics-consistent quantum-mechanics dataset with verification-aware reinforcement learning to improve LLM scientific reasoning. These efforts are complementary to code-generation benchmarks like ours, exercising different parts of the quantum-LLM stack. Beyond benchmarking, recent work explores alternative paradigms. QUASAR ( 53 ) applies agentic reinforcement learning with tool-augmented LLMs to

Read the complete text