Claim against Measurement: Statistical Artefacts in Quantum Error Mitigation Benchmarks

Dominik Koester, Wolfgang Mauerer · 2026

Read Claim against Measurement: Statistical Artefacts in Quantum Error Mitigation Benchmarks by Dominik Koester, Wolfgang Mauerer (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29872.txt.

Introduction

Dominik Köster and Wolfgang Mauerer audit statistics behind quantum error mitigation benchmarks. It bridges noisy devices and tolerant machines, yet judging studies face little checking of claims. Reviewing eighty-one papers against eight criteria covering rigour, reruns, and reporting, a quarter use inferential methods, while over two fifths report uncertainty without tests. A zero-noise case exposes two flaws: parameter shifts flipping conclusions between gain and loss, and drift inflating effects. The team proposes minimum standards: parameter notes, drift checks, inferential tests, effect sizes.

Catalog record

CREATORDominik Koester, Wolfgang Mauerer
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52082
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.29872

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Claim against Measurement: Statistical Artefacts in Quantum Error Mitigation Benchmarks Dominik Koester, Wolfgang Mauerer Abstract Quantum Error Mitigation (QEM)is widely regarded as a plausible bridge from Noisy Intermediate Scale Quantum (NISQ)devices to Fault Tolerant Quantum Computers (FTQC). Yet the empirical studies used to assess the effectiveness of QEMtechniques on concrete problems have received comparatively little scrutiny with respect to the validity of their conclusions. We systematically review 81 recent QEMpapers using an eight-criterion framework covering statistical rigour, reproducibility, and reporting quality. Among the 59 papers for which statistical evidence is applicable, only 15 (25%) use inferential methods, while 25 (42%) report uncertainty only descriptively, without testing whether the claimed effects are statistically supported. To demonstrate the consequences of these omissions, we use Zero-Noise Extrapolation (ZNE)as a representative and widely used case study and identify two compounding sources of artefacts in current QEMbenchmarks. First, we observe parameter sensitivity : in a 132-configuration sweep, implicitly assumed choices such as scale factors, extrapolation method, and hardware calibration are not merely incidental but active , with variations changing conclusions from statistically significant improvement to statistically significant degradation. Second, we identify a drift-induced effectiveness illusion : in a 72-hour longitudinal study on real hardware, temporal drift alone can make the same ZNEconfiguration exhibit an effect size more than three times as large, depending solely on when it is executed, and also drastically reduces the effective number of independent observations. These findings do not imply that QEMmethods are intrinsically unsound; rather, they show that current evaluation practice can make mitigation performance appear more robust than the evidence warrants. We therefore propose minimum reporting standards for QEMevaluations, including explicit parameter documentation, robustness checks, longitudinal drift assessment, and inferential statistical testing with effect-size reporting. I Introduction We remain in the NISQera of quantum computing [ 1 , 2 , 3 , 4 ] , still some distance from FTQCarchitectures with sufficiently many qubits, low error rates, and operational Quantum Error Correction (QEC) [ 5 , 6 ] . Although QECis clearly the long-term route to reliable quantum computation, present implementations still incur substantial qubit overheads [ 7 , 8 ] . In this intermediate regime, QEMhas emerged (among hardware-efficient problem formulations [ 9 , 10 , 11 ] or target design automation techniques [ 12 , 13 ] ) as a pragmatic approach for reducing the impact of noise [ 14 ] on current NISQdevices without implementing full error correction [ 15 , 16 , 17 , 18 ] . Techniques such as ZNE [ 15 , 16 ] , Probabilistic Error Cancellation (PEC) [ 15 , 19 ] , and Clifford data regression [ 20 ] have shown promise in improving the accuracy of quantum computations. Beyond dedicated benchmarking studies, QEMtechniques are increasingly adopted as standard components in broader quantum computing research – from variational quantum eigensolvers and quantum structure simulations [ 21 , 22 ] via optimisation [ 23 , 24 , 25 , 26 , 27 ] and machine learning [ 28 , 29 , 30 , 31 ] to quantum simulation of many-body systems [ 32 ] and demonstrations of evidence for quantum utility [ 33 ] – where framework-default parameters are silently inherited and their influence is not examined. Artefacts arising from such implicitly assumed parameter choices therefore propagate beyond QEMevaluation into any higher-level result that relies on mitigation as a building block. However, the empirical evaluation of QEMtechniques themselves frequently rests on experiments whose statistical foundations are not always commensurate with the strength of the conclusions drawn from them. Previous work has already identified important statistical challenges in the evaluation of QEMtechniques, including the need for explicit hypothesis testing [ 34 ] . In this work, however, our systematic review of 81 recent QEMpapers (Section IV ) shows that such concerns remain largely unaddressed in practice: most papers rely on descriptive reporting rather than inferential statistical evidence. This is not merely a matter of presentation, but one with direct experimental consequences. Reported QEMimprovements are often small, since current NISQhardware constrains experiments to shallow circuits and shot budgets limited by cost, placing many studies in regimes where the true effect is modest at best [ 35 , 36 ] . In such settings, shot noise, hardware drift, and implicitly assumed parameter choices can each be sufficient to alter the experimental conclusion. Without the use of careful inferential testing, a comprehensive awareness of influence factors and boundary conditions, as well as the adequate availability of all ingredients required to perform faithful reproductions or replications, neither the original study nor any subsequent replication can reliably distinguish genuine mitigation effects from statistical or experimental artefacts. In this paper, we identify two compounding sources of artefacts in QEMevaluation, each independently capable of producing misleading results. First, a systematic replication study shows that documented parameters usually represent only a small portion of the full space of possible parameters for an experiment: the choices experiments do not explicitly specify – such as scale factors, folding strategy, or calibration snapshot – are active , meaning their variation can shift the interpretation of the outcome of an experiment from significant improvement to significant worsening compared to an unmitigated baseline. Second, a longitudinal hardware study reveals that temporal drift alone can produce large variation in apparent ZNEeffectiveness on the same device at different access times; given the dominant use of cloud services with indeterministic batch access for many quantum experiments, this presents a substantial challenge for both reproducibility and interpretation of empirical results. We focus on ZNEwith Richardson extrapolation as a representative case, as it is the most widely reported QEMtechnique in our corpus, and propose a minimum reporting checklist for QEMevaluations. Concretely, our contributions are as follows: • We provide a systematic eight-criterion review of 81 QEMpapers, revealing that the majority of papers lack inferential statistics and drift control. • We introduce a replication pipeline and case study demonstrating that implicitly assumed ZNEparameters are active : their variation shifts experimental outcomes across a large fraction of tested configurations. • A longitudinal drift study shows that temporal hardware drift produces a drift-induced effectiveness illusion , where apparent ZNEeffectiveness varies substantially across identical experiments at different times. • We derive minimum reporting standards for QEMbenchmarks to avoid incorrect claims and misinterpretations of QEMexperiments. II Related Work Cai et al. [ 18 ] provide a comprehensive review of QEMtechniques, including their theoretical foundations, practical implementations, and performance analysis. Takagi et al. [ 37 ] derive fundamental sampling-overhead bounds under global depolarising noise, generalised by Quek et al. [ 38 ] for local noise. Krebsbacher et al. [ 39 ] derive variance bounds for Richardson extrapolation, guiding scale factor selection to minimise amplification. On the statistical side, Saki et al. [ 34 ] propose a hypothesis-testing framework for QEMevaluation, while Li et al. [ 40 ] survey the use of statistical methods in quantum software testing, finding that formal statistical methods are similarly rare in that domain. Moguel et al. [ 41 ] review quantum benchmarking methods, proposing a quantum experiment guideline extending established best practices from classical software benchmarking. On reproducibility, Senapati et al. [ 42 , 43 ] highlight the reproducibility challenges in quantum machine learning, being device variability and temporal drift. Hirasaki

Read the complete text