Statistically Valid Hyperparameter Selection From Tuning to Guarantees

Amirmohammad Farzaneh, Osvaldo Simeone · 2026

Read Statistically Valid Hyperparameter Selection From Tuning to Guarantees by Amirmohammad Farzaneh, Osvaldo Simeone (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25601.txt.

Description

Every deployed AI system carries knobs set by hand: thresholds, prompts, inference settings. Grid search finds good settings but certifies nothing about their reliability. Amirmohammad Farzaneh and Osvaldo Simeone devote a 2026 monograph to selection with guarantees. Their base is the learn then test paradigm. Candidate settings become hypotheses in a multiple testing problem. The needed tools, from p values to concentration bounds, are built from first principles in an appendix. Chosen settings then provably meet stated risk limits, with controlled error odds.

Four extensions widen the core method. One covers risks beyond averages, such as quantiles. Another enforces several reliability demands at once through Pareto testing. A third cuts the labelling budget with sequential, anytime valid inference. A fourth handles scarce ground truth by mixing betting based tests with prediction powered inference. Image classification, wireless packet scheduling, and prompt engineering supply the comparisons against heuristic tuning.

Catalog record

CREATORAmirmohammad Farzaneh, Osvaldo Simeone
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE252 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.25601

Browse the full library · About the Tower of Babel