Paired Seed Evaluation: Statistical Reliability for Learning-Based Simulators

Udit Sharma ID · 2025

Read Paired Seed Evaluation: Statistical Reliability for Learning-Based Simulators by Udit Sharma ID (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-24145.txt.

Introduction

Udit Sharma studies a subtle statistical point in how machine learning systems are compared. These systems look random, but their randomness comes from seeded generators, so the same seed replays the exact same run. Standard evaluations ignore this and treat runs of rival systems as independent, throwing away a free source of precision.

Sharma analyzes what happens when competing systems are instead evaluated on identical seeds, so their runs are matched pair by pair. Whenever outcomes under the same seed move together, this pairing strictly reduces the variance of the comparison. Using an extended learning-based multi-agent economic simulator, the paper shows paired evaluation exposing clear differences in both overall and distributional outcomes that stayed statistically inconclusive under independent runs at the same budget. The lesson is practical: seed pairing sharpens comparisons without extra computing.

Catalog record

CREATORUdit Sharma ID
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-24145

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Paired Seed Evaluation: Statistical Reliability for Learning-Based Simulators Udit Sharma ID Abstract Machine learning systems appear stochastic but are deterministically random, as seeded pseudorandom number generators produce identical realisations across repeated executions. Learning-based simulators are increasingly used to compare algorithms, design choices, and interventions under such stochastic dynamics, yet evaluation outcomes often exhibit high variance due to random initial conditions, environment noise, and learning stochasticity. We analyse the statistical structure of comparative evaluation in such settings and show that commonly used independent evaluation designs fail to exploit shared sources of randomness across alternatives. We formalise a paired evaluation design in which competing systems are evaluated under identical random seeds, inducing matched realisations of stochastic components and strict variance reduction whenever outcomes are positively correlated at the seed level, a condition that commonly arises in learning based environments with shared early trajectories and training dynamics. We characterise how this variance reduction propagates through the evaluation pipeline, leading to tighter confidence intervals, higher statistical power, improved directional stability and substantial effective sample size gains at fixed computational budgets. Empirical results show that seed level correlations are typically large and positive, producing order of magnitude efficiency gains relative to independent evaluation. Paired seed evaluation is weakly dominant in practice, improving statistical reliability when correlation is present and reducing to standard independent evaluation without loss of validity when it is not. These results suggest a simple but underused principle for simulation based research: exploiting shared randomness is not merely a variance reduction trick, but a foundational design choice for statistically efficient and reproducible evaluation in modern machine learning systems and benchmarks. firstpage,color=gray!90,angle=0,scale=0.28, xpos=0in,ypos=-5in, ]*correspondence: [email protected] 1 Introduction Learning based simulators and benchmarks have become a central empirical tool in modern machine learning. They are increasingly being used to evaluate algorithms, compare design choices, and assess interventions in settings where analytic guarantees are unavailable. As benchmark suites and simulation environments grow more complex and computationally expensive, empirical conclusions are increasingly drawn from a limited number of stochastic runs, placing greater weight on the statistical reliability of evaluation procedures, particularly in benchmark settings where multiple algorithms are compared within a shared simulator. A core difficulty is that outcomes in learning based simulators often exhibit high variance across random seeds. This variance is widely documented and arises from interacting sources such as random initial conditions, stochastic environment transitions, exploration noise, minibatch sampling, and optimiser nondeterminism [ 4 , 6 , 11 ] . The consequence is that naive evaluation can produce unstable rankings and inflated uncertainty, especially in the few-run regime that has become common as benchmarks and training pipelines have grown more expensive [ 1 ] . This broader reproducibility discourse in machine learning has repeatedly emphasised that reporting practices, experimental controls, and uncertainty quantification must be treated as first class scientific objects rather than presentation details [ 11 ] . The prevailing practice in many works is to evaluate alternatives using independent random seeds for each condition, then summarise performance using simple aggregates such as mean and standard deviation, sometimes with limited or no interval estimates. Several works have argued that such reporting can be misleading when the number of runs is small, since point estimates can be dominated by statistical uncertainty and may not support the implied strength of comparative claims. Motivated by this, multiple recent contributions propose more rigorous evaluation toolkits and protocols, including interval estimation, robust aggregate metrics, and explicit attention to variability across runs [ 4 , 11 , 1 , 2 ] . This paper focuses on a complementary and practically lightweight source of statistical efficiency that is often available in learning based simulations but not routinely exploited. Although learning systems are commonly treated as stochastic, their randomness is typically deterministic, governed by seeded pseudorandom number generators that reproduce identical stochastic realisations across repeated executions. As a result, outcomes under alternative algorithm configurations or interventions can be positively correlated at the level of random seeds when they share the same underlying stochastic realisation. This is natural in simulators where the same seed induces matched initial states, matched exogenous randomness, and matched algorithmic randomness, leading to aligned trajectories, even when learned behaviour differs. We formalise a paired evaluation design that holds the seed fixed across alternatives and estimates effects via within-seed differences. The design does not change the estimand, and it requires no additional modelling assumptions. Instead, it changes how randomness is coupled across conditions. When seed level correlation is positive, pairing yields strict variance reduction relative to independent evaluation, while if correlation is weak or absent, the paired runs can be treated as independent without loss of validity, making the approach weakly dominant for comparative evaluation. This work positions paired evaluation as a design principle for simulation based empirical research rather than a new estimator or modelling assumption. The approach is broadly applicable across learning based simulators and benchmark environments where alternative systems are evaluated under shared sources of stochasticity. In such settings, pairing converts common randomness from a source of noise into a source of statistical efficiency, improving inference under fixed computational budgets. 2 Evaluation Practice in Learning-Based Simulators Across learning-based simulators and benchmark environments, evaluation typically runs each algorithm or intervention under independently sampled random seeds and reports aggregate statistics such as the mean and standard deviation [ 4 , 6 , 11 , 1 ] .This convention remains standard in recent benchmark studies and simulator-based evaluations, including modern deep reinforcement learning benchmarks [ 3 , 7 ] , contemporary multi-agent learning frameworks [ 15 , 10 ] , and learning-based simulators for complex economic and social systems [ 16 , 8 ] . Such evaluations are typically sufficient for demonstrating qualitative behaviour, convergence properties, and broad performance trends. In learning-based simulators, where the objective is comparative inference across alternatives under fixed computational budgets, the prevailing evaluation convention places strong implicit assumptions on how stochasticity is treated. In much of the existing literature, random seeds are treated primarily as a device for reproducibility, rather than as a source of exploitable statistical structure. Treating all runs as independent discards information about shared sources of randomness induced by the simulator and the learning process. While this loss is often negligible at large evaluation scales, it can materially affect uncertainty and stability in the few-run regime that is now common in expensive simulation settings. These observations motivate a more careful treatment of randomness at the evaluation design level. Exploiting shared randomness does not require introducing new sources of coupling or modifying the simulator. Any exogenous source of stochastic variation that is held fixed across alternatives can, in principle, support paired comparison. In learning-based simulators, random seeds provide a universally available mechanism for inducing such common randomness, as they jointly determine initial conditions, environment transitions, and algorithmic stochasticity. Conditioning on the seed therefore preserves exogeneity while enabling paired evaluation without altering existing experimental protocols. While paired seeds have occasionally been used as an implementation detail in specific algorithmic and simulation contexts, such as risk-controlled self-modification [ 13 ] and climate emulation [ 5 ] , their role as a general evaluation design principle has not been formalised. Paired evaluation has also been explored in other settings, including biomedical model comparison [ 9

Read the complete text