POCAII : Parameter Optimization with Conscious Allocation using Iterative Intelligence

Joshua Inman, Tanmay Khandait, Lalitha Sankar, Giulia Pedrielli · 2025

Read POCAII : Parameter Optimization with Conscious Allocation using Iterative Intelligence by Joshua Inman, Tanmay Khandait, Lalitha Sankar, Giulia Pedrielli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11745.txt.

Introduction

Joshua Inman and three colleagues present a 2025 paper on a smarter way to tune machine learning models on a tight budget. Hyperparameter search usually mixes two jobs: proposing candidate settings and spending compute to test them. Their algorithm, POCAII, splits the jobs deliberately, favoring wide searching early and deep evaluation late.

Against three leading tuners, POCAII won most clearly when budgets were small, exactly the regime of practitioners training expensive models. Its results were also steadier, with lower variance across runs. The authors argue the explicit split, guided by principled search rules in both phases, is what buys the efficiency. They add that cheap, reliable tuning matters more in practice than peak performance at huge budgets.

Catalog record

CREATORJoshua Inman, Tanmay Khandait, Lalitha Sankar, Giulia Pedrielli
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-11745

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

POCAII : Parameter Optimization with Conscious Allocation using Iterative Intelligence Joshua Inman, Tanmay Khandait, Lalitha Sankar, Giulia Pedrielli Abstract In this paper we propose for the first time the hyperparameter optimization (HPO) algorithm POCAII . POCAII differs from the Hyperband and Successive Halving literature by explicitly separating the search and evaluation phases and utilizing principled approaches to exploration and exploitation principles during both phases. Such distinction results in a highly flexible scheme for managing a hyperparameter optimization budget by focusing on search (i.e., generating competing configurations) towards the start of the HPO process while increasing the evaluation effort as the HPO comes to an end. POCAII was compared to state of the art approaches SMAC, BOHB and DEHB. Our algorithm shows superior performance in low-budget hyperparameter optimization regimes. Since many practitioners do not have exhaustive resources to assign to HPO, it has wide applications to real-world problems. Moreover, the empirical evidence showed how POCAII demonstrates higher robustness and lower variance in the results. This is again very important when considering realistic scenarios with extremely expensive models to train. K eywords Hyperparameter optimization \cdot Optimal budget allocation \cdot Bayesian optimization 1 Introduction Large data-intensive models such as Convolutional Neural Networks (CNNs), Autoencoders, and Deep Belief Networks (DBNs) have become widespread across different domains, including computer vision (CV), natural language processing (NLP), robotics, in a plethora of applications spanning medical care to driving and portfolio management ( Zöller and Huber, 2021 ) . These models are highly configurable and require careful selection of hyperparameter values (e.g., learning rate, batch size, dropout rate, number of epochs) to achieve optimal performance, measured by metrics such as validation accuracy, F1-score, area under the ROC curve (AUC-ROC), or minimization of validation loss ( Feurer and Hutter, 2019 ; Bischl et al., 2023 ) . This problem is challenging because the validation loss is not known in closed form and therefore it requires training the model to be evaluated given a specific parametrization, which is a knowingly a time demanding process. As a statement to such complexity, a rich literature has been developed for the algorithmic, sequential generation of candidate hyperparameters, referred to as Hyperparameter optimization (HPO) ( Feurer and Hutter, 2019 ; Bischl et al., 2023 ; Raiaan et al., 2024 ) . Hyperparameter optimization (HPO) methods aim to automate the search for optimal configurations, with approaches broadly differing in how they balance exploration-exploitation trade-offs and manage computational costs. While the literature encompasses diverse strategies (e.g., evolutionary algorithms, gradient-based methods, meta-learning), two classes of approaches have particularly gained success due to their scalability and empirical success: (1) Bayesian Optimization (BO), and (2) multi-fidelity resource allocation ( Feurer and Hutter, 2019 ) . These approaches address core challenges in HPO, but introduce critical limitations that motivate our work. Bayesian Optimization (BO) frameworks treat HPO as a surrogate-guided search problem, and is a promising approach since it can treat the loss function as a black box with little to no assumptions on the function properties ( Snoek et al., 2012 ) . By modeling the loss function using surrogate models, BO iteratively selects hyperparameters. Algorithms such as Snoek et al. (2012) , Hutter et al. (2011) , Snoek et al. (2015) demonstrate the flexibility of BO, differing primarily in their choice of surrogate model and acquisition function design. However, BO’s reliance on fast feedback—inexpensive evaluations of the objective function—limits its scalability to computationally intensive tasks (e.g., training deep networks with many epochs), where even a single evaluation is prohibitively costly. Multi-fidelity resource allocation strategies, such as Successive Halving (SH) ( Jamieson and Talwalkar, 2016a ) and Hyperband (HB) ( Li et al., 2018 ) , address this limitation by dynamically allocating resources across configurations. These methods exploit low-fidelity approximations (e.g., training with fewer epochs, subsets of data, or smaller models) to cheaply eliminate poor candidates, reserving full budgets for top performers. However, their reliance on heuristic elimination schedules and assumptions about fidelity-performance correlations can lead to suboptimal allocations, particularly when early performance metrics poorly predict final model quality ( Raiaan et al., 2024 ) . Bayesian optimization with hyperband (BOHB) ( Falkner et al., 2018 ) bridges the gap between BO and multi-fidelity methods. By integrating Bayesian optimization with Hyperband’s resource allocation, BOHB uses surrogate models to guide the search for promising configurations while dynamically adapting fidelity levels (e.g., epoch budgets). This synergy allows BOHB to outperform standalone BO in low-budget scenarios and match its performance in high-budget settings. Despite its advancements, BOHB inherits unresolved challenges: (1) fidelity schedules (e.g., epoch increments or data subset sizes) are predefined rather than learned, limiting adaptability to problem-specific dynamics; (2) surrogate models may fail to capture complex relationships between fidelity levels and performance, leading to misguided resource allocation; and (3) the interaction between configuration sampling and fidelity selection remains unexplored, leading to redundant evaluations or premature convergence when fidelity correlations are weak. To address the aforementioned challenges, we introduced the Parameter Optimization with Conscious Allocation (POCA) in our preliminary work in Inman et al. (2023) . POCA improves existing methods by designing a novel optimization budget allocation scheme by shortening certain Hyperbands by reducing the maximum budget a configuration in that Hyperband can receive. Specifically, POCA allocates fewer resources to configurations selected early in the optimization process, i.e., when the number of configurations is small and only noisy loss evaluations are available. This allows more budget to be directed toward promising configurations discovered later in the search, while exploring a large number of cheap, configurations early on. Additionally, POCA emphasizes exploration during the early stages, ensuring broader coverage of the hyperparameter space. Our initial experiments demonstrated that POCA outperforms BOHB during small and medium-budget optimizations. ( Inman et al., 2023 ) . Example. Consider a case where the total budget available is B=600 , and the practioner wants no individual configuration to receive a budget larger than 8 with available budgets of 1,2,4,\text{and }8 . Hyperband or BOHB would run 5 hyperbands and would sample a total of 100 configurations. On the other hand, POCA would run 6 hyperbands, saving resources by reducing the budget assigned to early Hyperbands, allowing POCA to test 138 configurations in total. Our analysis identified three critical limitations in the search dynamics of the original POCA framework, which hinder its ability to efficiently identify optimal hyperparameter configurations. First, the algorithm systematically underallocates training budgets to high-performing configurations discovered early in the hyperparameter optimization (HPO) process. Since early-stage discoveries are deprioritized despite their potential, this leads to early discarding of promising candidates, resulting in suboptimal final selections. Second, POCA exhibits excessive exploitation by allocating substantial resources to configurations whose validation performance has plateaued, thereby wasting computational effort on uninformative candidates. Third, the ranking mechanism used to construct the Tree-structured Parzen Estimator (TPE) introduces bias by directly comparing validation metrics of configurations trained to different budgets. Specifically, configurations trained longer are inherently favored over those trained with smaller budgets, even if the latter exhibit superior potential. This bias creates a feedback loop where optimal configurations trained with limited budgets are deprioritized and excluded from further resource allocation. Collectively, these flaws negatively affect the efficiency and fairness of the HPO process, as they imbalance exploration-exploitation trade-offs, misallocate computational resources,

Read the complete text