Statistically Valid Hyperparameter Selection From Tuning to Guarantees
Amirmohammad Farzaneh, Osvaldo Simeone · 2026
Read Statistically Valid Hyperparameter Selection From Tuning to Guarantees by Amirmohammad Farzaneh, Osvaldo Simeone (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25601.txt.
Description
Every deployed AI system carries knobs set by hand: thresholds, prompts, inference settings. Grid search finds good settings but certifies nothing about their reliability. Amirmohammad Farzaneh and Osvaldo Simeone devote a 2026 monograph to selection with guarantees. Their base is the learn then test paradigm. Candidate settings become hypotheses in a multiple testing problem. The needed tools, from p values to concentration bounds, are built from first principles in an appendix. Chosen settings then provably meet stated risk limits, with controlled error odds.
Four extensions widen the core method. One covers risks beyond averages, such as quantiles. Another enforces several reliability demands at once through Pareto testing. A third cuts the labelling budget with sequential, anytime valid inference. A fourth handles scarce ground truth by mixing betting based tests with prediction powered inference. Image classification, wireless packet scheduling, and prompt engineering supply the comparisons against heuristic tuning.
Catalog record
| CREATOR | Amirmohammad Farzaneh, Osvaldo Simeone |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 252 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.25601