An information-matching approach to optimal experimental design and active learning
Yonatan Kurniawan, Tracianne B. Neilsen, Benjamin L. Francis, Alex M. Stankovic, Mingjian Wen, Ilia Nikiforov, Ellad B. Tadmor, Vasily V. Bulatov, Vincenzo Lordi, Mark K. Transtrum · 2024
Read An information-matching approach to optimal experimental design and active learning by Yonatan Kurniawan, Tracianne B. Neilsen, Benjamin L. Francis, Alex M. Stankovic, Mingjian Wen, Ilia Nikiforov, Ellad B. Tadmor, Vasily V. Bulatov, Vincenzo Lordi, Mark K. Transtrum (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-02740.txt.
Description
Abstract The efficacy of mathematical models heavily depends on the quality of the training data, yet collecting sufficient data is often expensive and challenging. Many modeling applications require inferring parameters only as a means to predict other quantities of interest (QoI). Because models often contain many unidentifiable (sloppy) parameters, QoIs often depend on a relatively small number of parameter combinations.
Catalog record
| CREATOR | Yonatan Kurniawan, Tracianne B. Neilsen, Benjamin L. Francis, Alex M. Stankovic, Mingjian Wen, Ilia Nikiforov, Ellad B. Tadmor, Vasily V. Bulatov, Vincenzo Lordi, Mark K. Transtrum |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-02740
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An information-matching approach to optimal experimental design and active learning Yonatan Kurniawan, Tracianne B. Neilsen, Benjamin L. Francis, Alex M. Stankovic, Mingjian Wen, Ilia Nikiforov, Ellad B. Tadmor, Vasily V. Bulatov, Vincenzo Lordi, Mark K. Transtrum Abstract The efficacy of mathematical models heavily depends on the quality of the training data, yet collecting sufficient data is often expensive and challenging. Many modeling applications require inferring parameters only as a means to predict other quantities of interest (QoI). Because models often contain many unidentifiable (sloppy) parameters, QoIs often depend on a relatively small number of parameter combinations. Therefore, we introduce an information-matching criterion based on the Fisher Information Matrix to select the most informative training data from a candidate pool. This method ensures that the selected data contain sufficient information to learn only those parameters that are needed to constrain downstream QoIs. It is formulated as a convex optimization problem, making it scalable to large models and datasets. We demonstrate the effectiveness of this approach across various modeling problems in diverse scientific fields, including power systems and underwater acoustics. Finally, we use information-matching as a query function within an Active Learning loop for material science applications. In all these applications, we find that a relatively small set of optimal training data can provide the necessary information for achieving precise predictions. These results are encouraging for diverse future applications, particularly active learning in large machine learning models. A model’s predictive performance depends strongly on the quality and quantity of data available for training. Curating comprehensive datasets, however, often confronts practical constraints, including instrumentation, available resources, and cost. Optimal experimental design (OED) leardi_experimental_2009 and active learning (AL) alizadeh_survey_2021 emerge as practical data collection strategies. Intentionally designing maximally informative experiments guarantees that data are most informative relative to the underlying phenomena of interest, minimize costs, and meet operational requirements. These methodologies have broad applications across scientific domains, including sensor placement problems in power systems yuill_optimal_2011 ; peng_optimal_2006 ; aminifar_contingency-constrained_2010 ; milosevic_nondominated_2003 ; chakrabarti_optimal_2008 ; soudi_optimal_1999 and underwater acoustics wood_optimisation_2003 ; dosso_optimal_1999 ; barlee_array_2002 ; barlee_array_2002 ; dosso_array_2006 ; tidwell_designing_2019 , the development of accurate interatomic potentials in materials science csanyi_learn_2004_mod ; artrith_high-dimensional_2012_mod ; podryabinkin_active_2017 ; gubaev_accelerating_2019 , and many other scientific fields casey2007optimal ; Jeong_Zhuang_Transtrum_Zhou_Qiu_2018 ; wang_active-learning_2021 ; kim_uncertainty_2022 . Many OED criteria utilize the Fisher information matrix (FIM), whose inverse establishes a lower bound on parameter covariance, known as the Cramér-Rao bound cramer1999mathematical ; streiner_precision_2006 ; van2007parameter . Common approaches optimize parameter precision through the FIM, such as by minimizing its trace (A-optimality) jacroux_-optimality_1989 ; butler_approximate_2008 ; jones_-optimal_2021 , or maximizing its determinant (D-optimality) andere-rendon_design_1997 ; balsa-canto_computing_2008 ; podryabinkin_active_2017 ; jones_-optimal_2021 or smallest eigenvalues (E-optimality) chow_e-optimality_1997 ; balsa-canto_computing_2008 ; morgan_e-optimality_2011 . However, many applications of predictive models do not require precise parameter estimates per se , but accurate predictions for key quantities of interest (QoIs) casey2007optimal ; transtrum2012optimal ; wen_force-matching_2017_mod ; shmueli_predictive_2011 ; tavazza_uncertainty_2021 . This distinction is well illustrated by sloppy models , where many parameter combinations are practically unidentifiable , yet still yield precise predictions transtrum_geometry_2011 ; transtrum2015perspective ; brouwer_underlying_2018 . Sloppy models exhibit a characteristic information spectrum, where FIM eigenvalues are nearly uniformly spaced on a log scale over many orders of magnitude. Many eigenvalues are smaller than the model’s evaluation precision, rendering the above OED criteria sensitive to numerical noise. Motivated by these considerations, we develop an information-matching method for OED that prioritizes the precision of predictions for the target QoIs. This approach uses a model parameterized by \boldsymbol{\theta} in two key scenarios: training, \bm{f} , and prediction, \bm{g} (see Fig. 1 ). Given a dataset of M independent inputs \{\mathbf{x}_{m}\}_{m=1}^{M} and their corresponding ground truth labels \{\mathbf{p}_{m}\}_{m=1}^{M} , we first use \bm{f}(\boldsymbol{\theta};\mathbf{x}_{m}) to train the parameters against the data \{\mathbf{x}_{m},\mathbf{p}_{m}\} . Then, we use the trained parameters to predict the target QoIs \mathbf{q} corresponding to the input \mathbf{y} through the mapping \bm{g}(\boldsymbol{\theta};\mathbf{y}) . In this scenario, \mathbf{y} may act as an input, similar to \mathbf{x}_{m} , as a control parameter for the QoIs, or as a discrete index to distinguish between different QoI values. The information-matching method leverages both scenarios to identify the minimal subset of training data that contains the information necessary to precisely constrain the parameters relevant to target QoIs. Our strategy is, given a target precision for target QoIs, align the FIM for the training data with that of the QoIs, ensuring the training data carry the information needed to constrain the predictions precisely. Thus, only the parameter combinations that need to be identified are trained, bypassing numerical stability issues in cases where the FIMs are ill-conditioned. Figure 1: The information-matching workflow. The panels represent (middle) a model parameterized by \boldsymbol{\theta} , (left) the target QoIs to predict, and (right) the dataset for training. First, one selects the target precision for the QoIs (blue envelope in the left panel). This QoI precision induces a minimal confidence region in parameter space: blue ellipse in the center panel. The information matching criterion selects training data and target precision (right panel) such that the resulting parameter uncertainty (yellow ellipse in the center panel) is more restrictive than that induced by the QoIs. Propagating the parameter uncertainty to the QoIs gives predictions that are at least as precise as the original target. This relationship holds even if the target uncertainties become infinite for certain inputs (dashed blue curves in the left panel), resulting in the target parameter confidence diverging for some parameter combinations (dashed blue ellipse in the middle panel, extending in some directions). The FIM is defined as the expectation value of the Hessian of the log-likelihood over the probability of the labels. For weighted least-squares, by far the most common regression scenario, the negative log-likelihood is (up to an additive constant) \ell(\boldsymbol{\theta})=\frac{1}{2}\sum_{m=1}^{M}w_{m}\lVert\mathbf{p}_{m}-\bm{f}(\boldsymbol{\theta};\mathbf{x}_{m})\rVert_{2}^{2}, (1) where the weight w_{m} is the inverse variance of the label \mathbf{p}_{m} , i.e., w_{m}=1/\sigma_{m}^{2} . As shown in the supplement, the FIM for the training scenario is given by \mathcal{I}(\boldsymbol{\theta})=\sum_{m=1}^{M}w_{m}\mathcal{I}_{m}(\boldsymbol{\theta}),\quad\mathcal{I}_{m}(\boldsymbol{\theta})=J_{\bm{f}}^{T}(\boldsymbol{\theta};\mathbf{x}_{m})J_{\bm{f}}(\boldsymbol{\theta};\mathbf{x}_{m}), (2) where \mathcal{I}_{m} and J_{\bm{f}}(\boldsymbol{\theta};\mathbf{x}_{m}) are the FIM and the Jacobian matrix of \bm{f}(\boldsymbol{\theta};\mathbf{x}_{m}) with respect to the parameters \boldsymbol{\theta} corresponding to the m -th datum, respectively transtrum_geometry_2011 . This equation highlights a generic, fundamental property of the FIM that the expected information in the entire training data is the sum of information from each independent datum. Notably, the FIM denotes the expected information over the probability of the labels and does not depend on the observed label value \mathbf{p}_{m} . This property means the FIM can be evaluated for any model predictions, including downstream applications for which ground truth labels are not available, and accounts for its broad appeal in OED. The FIM then quantifies how much information about the model parameters is required to achieve a target precision. We denote the target precision of the QoIs \mathbf{q} by the covariance matrix \mathbf{\Sigma} ; the FIM for the QoIs is given as \mathcal{J}(\boldsymbol{\theta})=J_{\bm{g}}^{T}(\boldsymbol{\theta})\mathbf{\Sigma}^{-1}J_{\bm{g}}(\boldsymbol{\theta}), (3) where J_{\bm{g}}(\boldsymbol{\theta}) is the Jacobian matrix of the proxy \bm{g}(\boldsymbol{\theta};\mathbf{y}) with respect to the parameters \boldsymbol{\theta} . The information-matching method leverages the FIMs of both the training data and target QoIs (Eqs. ( 2 ) and ( 3 ), respectively) to guarantee that the information in the training data is sufficient to achieve the target precision in the