Systematic benchmarking of quantum computers: status and recommendations
Jeanette Miriam Lorenz 1 2, Thomas Monz 3 4, Jens Eisert 5 6, Daniel Reitzner 7, Félicien Schopfer 8 · 2025
Read Systematic benchmarking of quantum computers: status and recommendations by Jeanette Miriam Lorenz 1 2, Thomas Monz 3 4, Jens Eisert 5 6, Daniel Reitzner 7, Félicien Schopfer 8 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04905.txt.
Introduction
Jeanette Miriam Lorenz and a large team of co-authors published a report in 2025 on how to benchmark quantum computers systematically. Quantum hardware can only scale if buyers and builders can measure it honestly, yet tests remain scattered. The document surveys the state of practice and recommends a common structure.
Benchmarks are sorted into five levels. The first levels cover single parts such as qubits and gates, then the whole processor. Higher levels cover the software stack, integration with classical supercomputers and cloud access, and finally real applications. Standardization matters most, the authors argue, because results must be reproducible and comparable across labs and vendors. Their recommendations push for shared evaluation routines woven into the wider quantum technology effort.
Catalog record
| CREATOR | Jeanette Miriam Lorenz 1 2, Thomas Monz 3 4, Jens Eisert 5 6, Daniel Reitzner 7, Félicien Schopfer 8 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 85 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-04905
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Systematic benchmarking of quantum computers: status and recommendations Jeanette Miriam Lorenz 1 2, Thomas Monz 3 4, Jens Eisert 5 6, Daniel Reitzner 7, Félicien Schopfer 8 Abstract Architectures for quantum computing can only be scaled up when they are accompanied by suitable benchmarking techniques. The document provides a comprehensive overview of the state and recommendations for systematic benchmarking of quantum computers. Benchmarking is crucial for assessing the performance of quantum computers, including the hardware, software, as well as algorithms and applications. The document highlights key aspects such as component-level, system-level, software-level, HPC-level, and application-level benchmarks. Component-level benchmarks focus on the performance of individual qubits and gates, while system-level benchmarks evaluate the entire quantum processor. Software-level benchmarks consider the compiler’s efficiency and error mitigation techniques. HPC-level and cloud benchmarks address integration with classical systems and cloud platforms, respectively. Application-level benchmarks measure performance in real-world use cases. The document also discusses the importance of standardization to ensure reproducibility and comparability of benchmarks, and highlights ongoing efforts in the quantum computing community towards establishing these benchmarks. Recommendations for future steps emphasize the need for developing standardized evaluation routines and integrating benchmarks with broader quantum technology activities. List of abbreviations ACES Averaged circuit eigenvalue sampling is a component-level benchmarking method that is capable of detecting and characterising correlations. FTQC Fault-tolerant quantum computing relies on careful computational instructions such that errors do not spread during QC in an uncontrollable fashion. These carefully designed instructions are accompanied by QEC routines that remove undesired errors during the computation and allow, in principle, for arbitrarily long reliable computations. KPI Key performance indicators are measured according to some suitable metrics and are derived from several benchmarking methods. MCDA Multi-criteria decision aiding methods are techniques designed to evaluate multiple, often conflicting criteria to aid decision-making and ranking. NISQ Noisy intermediate-scale quantum [ 1 ] devices describe quantum devices available at present and in the near future that do not allow FTQC. They contain essentially all quantum and hybrid computational schemes that are not contained within the group of FTQC. QC Quantum computing . QEC Quantum error correction embodies schemes for error reduction in QC that rely on quantum redundant encoding and measurements which need not be necessarily fault tolerant. RB Randomized benchmarking refers to a family of benchmarking processes in which a set of gate operations is randomly generated such that the total sequence ideally corresponds to the identity operation. The decay of the success rate as a function of the length of the sequence allows one to infer the average error rate in quantum gate sets. SDK Software Development Kit . I Introduction While quantum computers were already proposed in the 1980s, their practical usefulness remains an open question despite substantial recent breakthroughs in realizing fault-tolerant quantum devices in the last decade [ 2 , 3 ] . Potential applications of quantum computing include all academic or industrial use cases which rely on simulating quantum mechanical systems (such as, e.g., in drug development or materials science), on solving linear systems of equations, on addressing combinatorial optimization challenges and on enhancing machine learning by suitable quantum subroutines. The computational paradigm of quantum computing was first proposed by D. Deutsch in 1985, who thought about a stronger version of the Church-Turing thesis. Richard Feynman pointed out in his 1982 article [ 4 ] that classical computers may have difficulties in simulating quantum systems and that quantum devices may be of use here, and added substantial insights into notions of mutual simulatability and computational efforts. Only in the mid 1990s, a key quantum algorithms was proposed by Peter Shor [ 5 ] , providing a practically relevant algorithm to factor big numbers into their prime factors, featuring a super-polynomial advantage over the best known classical algorithms. Lov K. Grover around the same time proposed an algorithm for an efficient search through unsorted databases [ 6 ] , featuring a polynomial quantum advantage. First experimental realizations of small noisy quantum computers were established in laboratory settings in the 1990s, but they could not yet be used to run any extended computations. This state of affairs started to change only more than 20 years later: In 2019, Google demonstrated an experiment on the Google Sycamore chip featuring 53 superconducting qubits. In this experiment, Google claimed a quantum advantage by solving a specific mathematical problem significantly faster than what was possible on any available classical supercomputer [ 7 ] . Although this claim was challenged later by classical simulations [ 8 , 9 ] – but also subsequently strengthened by more sophisticated experiments [ 10 , 2 , 11 ] – it did trigger an increased interest in quantum computing as a technology and in its disruptive potential for a significant number of industrial challenges. Similarly, in 2020, the group of J.-W. Pan claimed a quantum advantage on Gaussian boson sampling [ 12 ] , a specific quantum random sampling scheme based on photonic Gaussian input states. In 2023, a team of IBM suggested having reached the regime of quantum utility [ 13 ] , which constitutes a regime where quantum computers are comparable to classical computers. The state of the art chip by Google Quantum AI is the Willow chip, a 72-qubit and a 105-qubit quantum processor designed for quantum error correction experiments [ 2 ] . By now, various quantum hardware vendors have started to offer access to their quantum computers for academic and industrial use, partly via cloud access. This has triggered work towards practical testing of quantum computer capabilities. However, despite various claims of quantum supremacy, advantage or utility, usefulness of quantum computers for concrete tasks that tackle immediate practical industrially relevant applications have not be demonstrated yet. Significant advances on the quantum hardware, software and quantum algorithm side are required before this could be demonstrated. For instance, different works [ 14 , 15 ] have shown that currently proposed algorithms such as, e.g., Grover’s algorithm, would result in longer execution times than those of classical algorithms even if fault-tolerant quantum computers would exist. This is because typical quantum gate operation times on quantum computers (in the ns to \mu s regimes) are typically longer than gate operation times on classical computers. Therefore, quantum algorithms may need to provide an exponential or super-polynomial computational speed-up in comparison to classical computers [ 14 ] to practically accelerate calculations time-wise. Additionally, the significant progress over the last years in the area of quantum hardware and software development have led to first attempts at quantum software stacks [ 16 ] . The goal of such stacks is to implement abstraction layers for using quantum computers similar to what is employed in classical computers. These software development activities are intended to make quantum computers more broadly available for academics, industry, and the general public. One goal of these software developments aims at extending from purely quantum algorithms to hybrid quantum-classical algorithms, which will be composed of both quantum and classical computing infrastructure. As a consequence, to assess whether the use of a quantum computer is beneficial to solve a given use case, one will need to assess both the performance of the quantum part of a computation, as well as the performance of all