Quantum Knowledge Distillation for Large Language Models
Lingxiao Li, Yihao Wang, Jiacheng Fan, Jing Li, Sujuan Qin, Qiaoyan Wen, Fei Gao · 2025
Read Quantum Knowledge Distillation for Large Language Models by Lingxiao Li, Yihao Wang, Jiacheng Fan, Jing Li, Sujuan Qin, Qiaoyan Wen, Fei Gao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13205.txt.
Introduction
Lingxiao Li and co-authors wrote a paper from 2025 on shrinking large language models with the help of quantum circuits. Big models are costly to run, most of all on small devices. A known fix trains a small student model to copy a large teacher. The team lets a quantum circuit play the student, since quantum states can hold rich patterns in few parts.
Their student used just eleven quantum bits. In classic simulation, it beat several standard student methods on text sorting tasks, in both accuracy and speed. The team then ran the trained circuits on a real quantum processor in China. The model held its performance there despite noise and limited sampling on the hardware. They frame the work as a first step toward quantum-native ways to compress ever larger models. Their code is shared in public.
Catalog record
| CREATOR | Lingxiao Li, Yihao Wang, Jiacheng Fan, Jing Li, Sujuan Qin, Qiaoyan Wen, Fei Gao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-13205
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Quantum Knowledge Distillation for Large Language Models Lingxiao Li, Yihao Wang, Jiacheng Fan, Jing Li, Sujuan Qin, Qiaoyan Wen, Fei Gao Abstract As foundational tools in natural language processing, Large Language Models (LLMs) have immense parameter scales, which makes deployment and inference increasingly prohibitive, especially in resource-constrained devices. Therefore, knowledge distillation for LLMs, i.e., compressing the LLM to a smaller model, is meaningful. With strong parameter representation capacity, quantum computing is regarded as a promising solution. Here, we propose a Quantum knowledge Distillation model for LLMs (QD-LLM) that leverages variational quantum circuits to learn from LLMs. In classical simulation, QD-LLM outperforms several mainstream distillation methods on multiple text classification tasks in terms of both accuracy and efficiency using only 11 qubits. The results reveal an interesting phenomenon that the simulation of quantum student models may be regarded as a new class of quantum-inspired classical algorithms. Remarkably, we deploy the obtained circuits on the Baihua superconducting quantum processor via the Quafu platform to assess practical feasibility. The model maintains stable inference performance despite hardware constraints such as decoherence and finite sampling. In summary, QD-LLM marks a foundational step in connecting quantum computing with LLMs, demonstrating the feasibility of quantum-native approaches that aim to compress and deploy models of increasingly larger scales. The code of this article has been open-sourced at https://github.com/Lilingxiao-bupt/QD-LLM. Introduction Recent advances in artificial intelligence have led to the emergence of numerous Large Language Models (LLMs) 1 , 2 , 3 , 4 , which have become the backbone of modern natural language processing (NLP) systems, consistently delivering state-of-the-art performance across a wide range of tasks 5 , 6 , 7 . However, deploying LLMs 8 , 9 , 10 , 11 for various tasks typically requires loading the models for inference, which results in substantial computational and memory overhead. To address this problem, Knowledge Distillation (KD) for LLMs 12 , 13 has been widely adopted as a means of transferring task-specific knowledge from large-scale models (called teacher models) to smaller models (called student models) 14 . Distilled models offer lower computational costs compared to using LLMs directly while preserving comparable inference capability. As LLMs continue to grow in parameter count, there remains room for further optimization in model compression, especially in scenarios with extremely low resources or where faster inference is required. Fortunately, extensive research has demonstrated that quantum computing can solve problems efficiently using fewer quantum parameters 15 , 16 , 17 , 18 , 19 due to its powerful parameter representation capability. In particular, quantum neural networks (QNNs) 20 , 21 , 22 , 23 have been explored as student models in knowledge distillation, with most existing studies focusing on small-scale image classification tasks using shallow classical 24 , 25 or quantum teacher models 26 . However, current QNN-based student models have not yet demonstrated the ability to distill LLMs for natural language classification tasks. In other words, whether these models can capture the complexity, semantic richness, and real-world variability inherent in natural language understanding of LLMs remains an open question. To bridge this challenge, we propose a Quantum knowledge Distillation algorithm for LLMs (QD-LLM). Unlike previous quantum distillation studies limited to small models or small tasks, QD-LLM fills the gap of LLMs distillation by treating LLMs as fixed teacher models and training a quantum student model to approximate their outputs. The student model is implemented as a variational quantum circuit specifically designed for this distillation setting. During the training phase, the circuit parameters are optimized to minimize the difference between the outputs of the student model and the teacher model. Once trained, the student performs inference independently, relying solely on the learned circuit. This makes it well-suited for deployment in scenarios with limited computational resources. We evaluate QD-LLM through a comprehensive series of simulation experiments and real-device deployments, where the model is implemented as a variational quantum circuit with only 11 qubits and assessed on three real-world natural language processing tasks: Emotion Analysis 27 , Hiding Detection 28 , and Thematic Analysis 29 . To ensure a fair comparison, all models, including quantum models simulated on classical hardware and classical baselines, are trained under identical conditions with aligned input representations. Compared to mainstream classical baselines, QD-LLM significantly reduces memory consumption and achieves efficient training and inference on classification tasks due to lower parameter counts. In addition, we deploy the quantum model obtained from QD-LLM in a real quantum device, the Baihua processor provided by the Quafu platform 30 , to examine the practical feasibility of deploying QD-LLM. Our student model achieves stable and reliable inference results in real devices, despite hardware constraints such as decoherence and shot noise. Interestingly, our simulation results reveal a compelling phenomenon: the quantum student model obtained from our approach outperforms existing mainstream classical models, including the ones distilled from LLM and the ones from task-specific training, in both memory consumption and classification accuracy on practically meaningful tasks. This unexpected observation reveals that the simulation of our quantum student model can be regarded as a new class of quantum-inspired classical algorithm, which is a direct simulation of the quantum algorithm. Furthermore, the strong simulation results imply that, as the quality of qubits in real hardware continues to improve, our quantum student model may emerge as a promising candidate for achieving quantum supremacy on practically meaningful tasks, even though the inference accuracy on current hardware is limited. Note that the above observations benefit from the fact that only a small-scale quantum circuit (here, 11 qubits and a shallow depth) is enough for the above tasks in our student method. Results QD-LLM Overall Text, as one of the main forms of communication, is widely used on social networks. Influenced by factors like users’ intentions and personalities, texts on social media exhibit diverse styles and themes. Sentiment and topic analysis of such content can support various applications and decisions, making it a prominent subject in computational linguistics. In this work, we use text as the research object to introduce the design of the QD-LLM method. In the proposed method, LLMs act as teacher models to guide the training of QNN, which is designated as student models. During the inference phase, QD-LLM only uses the trained quantum circuits, without requiring the LLMs, allowing for efficient inference. The overall architecture of QD-LLM is shown in Figure 1 . Quantum distillation for LLMs LLM teacher model In LLMs, given a sequence of text (token sequence) T:{w_{1}},{w_{2}},\cdots,{w_{t-1}} , the model can generate the probability of each token in the vocabulary V=\{{v_{1}},{v_{2}},\cdots,{v_{n}}\} as the next token, represented by a conditional probability distribution P({w_{t}}|{w_{1}},\cdots,{w_{t-1}}) , where n is the size of the vocabulary. This is computed as follows: P({w_{t}}|{w_{1}},{w_{2}},\cdots,{w_{t-1}})={\text{Softmax}}(LM({w_{1}},{w_{2}},\cdots,{w_{t-1}})),{\text{ }}{w_{t}}{\text{ in }}V. (1) The function LM(\cdot) abstracts formula (1) and represents the complete language model calculation process, where {\text{Softmax}}(\cdot) represents the probability conversion process. Constructing the vocabulary V is central to the operation of LLMs, and the order and size of vocabularies differ between models. To distill a generative LLM effectively, the student model must align with the teacher model’s vocabulary. However, current vocabularies have reached the scale of tens of thousands, which cannot be effectively learned by the limited number of qubits in quantum models. Therefore, in this paper,