Revisiting Quantum Code Generation: Where Should Domain Knowledge Live?

Oscar Novo, Oscar Bastidas-Jossa, Alberto Calvo, Antonio Peris, Carlos Kuchkovsky · 2026

Read Revisiting Quantum Code Generation: Where Should Domain Knowledge Live? by Oscar Novo, Oscar Bastidas-Jossa, Alberto Calvo, Antonio Peris, Carlos Kuchkovsky (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22184.txt.

Introduction

Novo and co-authors ask where domain knowledge should live in quantum code generation tools. They compare a fine-tuned Qiskit model with general language models on the Qiskit-HumanEval benchmark. The general models get help from retrieval or from agents with execution feedback. The fine-tuned baseline scores about 47% pass at one try. General models reach 60 to 65% with no help or with retrieval. The best setup, an agent with execution feedback, reaches 85%. The authors conclude that inference-time help beats domain fine-tuning here.

Catalog record

CREATOROscar Novo, Oscar Bastidas-Jossa, Alberto Calvo, Antonio Peris, Carlos Kuchkovsky
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52047
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.22184

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Revisiting Quantum Code Generation: Where Should Domain Knowledge Live? Oscar Novo, Oscar Bastidas-Jossa, Alberto Calvo, Antonio Peris, Carlos Kuchkovsky Abstract Recent advances in large language models (LLMs) have enabled the automation of an increasing number of programming tasks, including code generation for scientific and engineering domains. In rapidly evolving software ecosystems such as quantum software development, where frameworks expose complex abstractions, a central question is how best to incorporate domain knowledge into LLM-based assistants while preserving maintainability as libraries evolve. In this work, we study specialization strategies for Qiskit code generation using the Qiskit-HumanEval benchmark. We compare a parameter-specialized fine-tuned baseline introduced in prior work against a range of recent general-purpose LLMs enhanced with retrieval-augmented generation (RAG) and agent-based inference with execution feedback. Our results show that modern general-purpose LLMs consistently outperform the parameter-specialized baseline. While the fine-tuned model achieves approximately 47% pass@1 on Qiskit-HumanEval, recent general-purpose models reach 60–65% under zero-shot and retrieval-augmented settings, and up to 85% for the strongest evaluated model when combined with iterative execution-feedback agents—representing an improvement of more than 20% over zero-shot general-purpose performance and more than 35% over the parameter-specialized baseline. Agentic execution feedback yields the most consistent improvements, albeit at increased runtime cost, while RAG provides modest and model-dependent gains. These findings indicate that performance gains can be achieved without domain-specific fine-tuning, instead relying on inference-time augmentation, thereby enabling a more flexible and maintainable approach to LLM-assisted quantum software development. 1 Introduction Large language models (LLMs) [ 23 ] have recently emerged as powerful tools for automating a wide range of programming tasks, including code completion, program synthesis, and software maintenance. Their impact has been particularly notable in scientific and engineering domains, where LLMs are increasingly used to assist with complex workflows involving specialized libraries, numerical methods, and domain-specific abstractions. As these models continue to improve in scale, reasoning capability, and context length [ 5 ] , an important open question is how best to incorporate domain knowledge into LLM-based systems for specialized scientific software ecosystems. In quantum computing, the development of reliable and efficient software remains a key challenge, given the complexity of quantum programming abstractions and the rapid evolution of quantum software development kits (SDKs), such as Qiskit [ 9 ] , Cirq [ 6 ] , PennyLane [ 4 ] , and Braket SDK [ 1 ] . These SDKs expose diverse programming abstractions, execution backends, and rapidly evolving APIs, making correctness and maintainability central concerns for automated code generation. Broadly, domain adaptation for LLM-based code generation can be approached through two complementary paradigms. The first is training-time parameter-level specialization , in which a base model is fine-tuned on curated domain corpora to encode framework-specific knowledge in its weights. The second, which we refer to as inference-time system-level specialization , adapts a general-purpose model at inference time by embedding it within a structured pipeline that incorporates retrieval mechanisms and agent-based execution-feedback loops. This approach enables context-aware code generation by incorporating external retrieval and execution feedback, supporting iterative self-correction without modifying the model’s parameters. Prior work demonstrated that training-time specialization can yield substantial improvements for Qiskit code generation by fine-tuning a Granite-based model on curated Qiskit corpora and evaluating on the Qiskit-HumanEval benchmark [ 8 , 13 ] . However, the landscape of general-purpose LLM capabilities has evolved rapidly, with notable improvements in long-context reasoning, code synthesis, and tool use. At the same time, training-time specialization entails non-trivial costs, including data curation, training infrastructure, maintenance overhead, and limited adaptability to evolving software libraries. These considerations motivate a reassessment of the trade-offs between parameter-level specialization and inference-time system-level approaches. In this work, we revisit Qiskit code generation and systematically re-evaluate performance on Qiskit-HumanEval [ 13 ] using a range of recent general-purpose LLMs, alongside a parameter-specialized fine-tuned baseline. Beyond direct model comparisons, we study inference-time augmentation strategies, including retrieval-augmented generation (RAG) [ 10 ] and agent-based workflows [ 24 ] that incorporate execution feedback. These techniques enable models to dynamically access external knowledge and iteratively refine generated code without modifying model parameters, offering a flexible alternative to static specialization. Our study aims to answer the following research questions: (i) Do modern general-purpose LLMs match or exceed the performance of a parameter-specialized baseline on Qiskit code generation benchmarks? (ii) To what extent can retrieval and agentic inference strategies improve performance without additional training? and (iii) What are the implications of these findings for the design, cost, and long-term maintainability of LLM-based systems for quantum software development? The contributions of this paper are threefold. First, we provide an updated empirical evaluation of Qiskit code generation performance in the context of recent general-purpose LLMs using Qiskit-HumanEval. Second, we analyze the effectiveness of retrieval-augmented and agent-based inference strategies for Qiskit code generation. Third, we discuss broader implications with respect to reproducibility, benchmark leakage risk, cost-efficiency, and the sustainability of training-time specialization approaches. 2 Related Work The most closely related prior work to this study is the Qiskit Code Assistant study introduced by Dupuis et al. [ 8 ] . In that work, the authors propose a domain-specific approach to quantum code generation by fine-tuning a Granite-based large language model on curated Qiskit-related corpora. To evaluate the effectiveness of this strategy, they introduce the Qiskit-HumanEval benchmark [ 13 ] , a quantum programming benchmark derived from HumanEval [ 5 ] , and report substantial improvements over general-purpose code models on Qiskit-centric tasks. Table 1 reproduces results reported by Dupuis et al. [ 8 ] , comparing general-purpose code models, instruction-tuned variants, and domain-specific fine-tuned models. In particular, the Qiskit-fine-tuned Granite-20B model shows a large performance gain on Qiskit-HumanEval relative to the base Granite model and several general-purpose LLMs, motivating domain-specific fine-tuning as an effective strategy for quantum code generation at the time of publication. Table 1: HumanEval (HE) and Qiskit-HumanEval (QHE) pass@1 results reported by Dupuis et al. [ 8 ] . Pass@1 denotes the fraction of tasks for which a single generated solution passes all unit tests. Model names are simplified for clarity. Model HE QHE CodeLLaMA-34B (Python-Specialized) 52.43% 26.73% DeepSeek-Coder-33B (Base) 49.39% 39.60% DeepSeek-Coder-33B (Instruction-Tuned) 68.9% 35.64% StarCoder2-15B 45.12% 37.62% Granite-20B (Base Code Model) 38.41% 20.79% Granite-20B (Qiskit-Fine-Tuned) 36.58% 46.53% While these findings provide strong evidence for the effectiveness of parameter-level specialization, the evaluation predates recent advances in general-purpose LLM capabilities and does not consider inference-time augmentation strategies such as retrieval-augmented generation (RAG) [ 10 ] or agent-based execution feedback [ 24 ] . As a result, it remains unclear whether similar gains can be achieved using modern general-purpose models combined with enhanced inference techniques, without resorting to domain-specific fine-tuning. 2.1 Inference-Time Augmentation for Code Generation Beyond parameter-level specialization, recent work has explored inference-time augmentation techniques that enhance the capabilities of large language models without additional training. One prominent approach is retrieval-augmented generation (RAG), which couples an LLM with an external retrieval component to dynamically incorporate relevant information—such as documentation, API references, or source code—into the model’s input context during generation [ 10 ] . By grounding generation in retrieved context, RAG reduces reliance on memorized knowledge and has been shown to improve performance on knowledge-intensive and API-heavy coding tasks [ 10 ] . Complementary to retrieval-based methods, agent-based inference frameworks extend LLMs with

Read the complete text