Optimal Robust Bounded Bias and Bounded Variance Designs Journal: U. Alberta preprint series
Douglas P. Wiens · 2025
Read Optimal Robust Bounded Bias and Bounded Variance Designs Journal: U. Alberta preprint series by Douglas P. Wiens (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-21806.txt.
Introduction
Douglas P. Wiens works on experimental design when the model being fitted may be wrong. In robust design, a strategy is minimax: choose the design minimizing the worst prediction error over a class of models. That worst-case error splits into a variance part and a bias part caused by the model errors.
Wiens studies two constrained versions of the problem. One minimizes the predictors' variance subject to a cap on the worst-case bias. The other minimizes the worst-case bias subject to a cap on the variance. He shows the solutions to both are the minimax designs, with their tuning constants chosen appropriately. Conversely, every minimax design solves each constrained problem for a suitable cap. The result ties three design philosophies: tuning a minimax design is the same act as choosing how much bias or variance to tolerate. One family of designs serves all three purposes.
Catalog record
| CREATOR | Douglas P. Wiens |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-21806
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimal Robust Bounded Bias and Bounded Variance Designs Journal: U. Alberta preprint series Douglas P. Wiens Abstract Designs which are minimax in the presence of model misspecifications have been constructed so as to minimize the maximum, over classes of alternate response models, of the integrated mean squared error of the predicted values. This mean squared error decomposes into a term arising solely from variation, and a bias term arising from the model errors. Here we consider two associated problems: (i) design so as to minimize the variance, subject to a bound on the bias, and (ii) design so as to minimize the bias, subject to a bound on the variance. We show that solutions to both problems are given by the minimax designs, with appropriately chosen values of their tuning constant. Conversely, any minimax design solves both problems for appropriate choices of the bounds on the bias or variance. 2010 MSC Primary 62F35 , Secondary 62K05 1 Introduction and summary The theory of robustness of design was largely initiated by 1 , who investigated the robustness of some classical experimental designs in the presence of certain model inadequacies, e.g. designs optimal for a low order polynomial response when the true response was a polynomial of higher order. 2 derived designs for straight line regression, robust in the presence of alternate response functions. 5 ; 6 extended these results to multiple regression responses and in a variety of other directions – see 7 for a summary of these and other approaches to robustness of design. Designs which are minimax in the presence of model misspecifications aim to minimize the maximum, over classes of alternate response models, of the integrated mean squared error ( imse ) of the predicted values. In this note we first briefly review the theory of such designs, and discuss a decomposition of the imse into a convex combination of two terms – one arising solely from variation, and the other arising from the bias due to the model errors. Then in § 3 we propose two associated problems: (i) design so as to minimize the variance, subject to a bound on the bias, and (ii) design so as to minimize the bias, subject to a bound on the variance. We show that solutions to both problems are given by the minimax designs, with appropriately chosen values of the mixing parameter. Conversely, any minimax design solves both problems for appropriate choices of the bounds on the bias or variance. Examples and methods of implementation are studied in § 4 . The matlab code used to prepare these examples is available on the author’s personal website. 2 Minimax robustness of design The general minimax design problem is phrased in terms of an approximate regression response E\left[Y\left(\boldsymbol{x}\right)\right]\approx\boldsymbol{f}^{\prime}\left(\boldsymbol{x}\right)\boldsymbol{\theta}, (1) for p regressors \boldsymbol{f} , each functions of q independent variables \boldsymbol{x} , and a parameter \boldsymbol{\theta} . Since ( 1 ) is an approximation the interpretation of \boldsymbol{\theta} is unclear; we define this target parameter by \boldsymbol{\theta}=\arg\min_{\boldsymbol{\eta}}\int_{\mathcal{X}}\left(E\left[Y\left(\boldsymbol{x}\right)\right]-\boldsymbol{f}^{\prime}\left(\boldsymbol{x}\right)\boldsymbol{\eta}\right)^{2}\mu\left(d\boldsymbol{x}\right), (2) where \mu\left(d\boldsymbol{x}\right) represents either Lebesgue measure or counting measure, depending upon the nature of the design space \mathcal{X} with \int_{\mathcal{X}}\mu\left(d\boldsymbol{x}\right)<\infty . We then define \psi\left(\boldsymbol{x}\right)=E\left[Y\left(\boldsymbol{x}\right)\right]-\boldsymbol{f}^{\prime}\left(\boldsymbol{x}\right)\boldsymbol{\theta} . This results in the class of responses E\left[Y\left(\boldsymbol{x}\right)\right]=\boldsymbol{f}^{\prime}\left(\boldsymbol{x}\right)\boldsymbol{\theta}+\psi\left(\boldsymbol{x}\right) , with – by virtue of ( 2 ) – \psi satisfying the orthogonality requirement \int_{\mathcal{X}}\boldsymbol{f}\left(\boldsymbol{x}\right)\psi\left(\boldsymbol{x}\right)\mu\left(d\boldsymbol{x}\right)=\boldsymbol{0}.\ (3) Assuming that \mathcal{X} is rich enough that the matrix \boldsymbol{A}=\int_{\mathcal{X}}\boldsymbol{f}\left(\boldsymbol{x}\right)\boldsymbol{f}^{\prime}\left(\boldsymbol{x}\right)\mu\left(d\boldsymbol{x}\right) is invertible, the parameter defined by ( 2 ) and ( 3 ) is unique. We identify a design with its design measure – a probability measure \xi\left(d\boldsymbol{x}\right) on \mathcal{X} . Define \boldsymbol{M}_{\xi}=\int_{\mathcal{X}}\boldsymbol{f}\left(\boldsymbol{x}\right)\boldsymbol{f}^{\prime}\left(\boldsymbol{x}\right)\xi\left(d\boldsymbol{x}\right),\text{ \ }\boldsymbol{b}_{\psi,\xi}=\int_{\mathcal{X}}\boldsymbol{f}\left(\boldsymbol{x}\right)\psi\left(\boldsymbol{x}\right)\xi\left(d\boldsymbol{x}\right), and assume \xi is such that \boldsymbol{M}_{\xi} is invertible. For an anticipated design of n , not necessarily distinct, points the covariance matrix of the least squares estimator \boldsymbol{\hat{\theta}} , assuming homoscedastic errors with variance \sigma_{\varepsilon}^{2} , is \left(\sigma_{\varepsilon}^{2}/n\right)\boldsymbol{M}_{\xi}^{-1} , and the bias is E\left[\boldsymbol{\hat{\theta}-\theta}\right]=\boldsymbol{M}_{\xi}^{-1}\boldsymbol{b}_{\psi,\xi} ; together these yield the mean squared error ( mse ) matrix \text{{mse}}\left[\boldsymbol{\hat{\theta}}\right]=\frac{\sigma_{\varepsilon}^{2}}{n}\boldsymbol{M}_{\xi}^{-1}+\boldsymbol{M}_{\xi}^{-1}\boldsymbol{b}_{\psi,\xi}\boldsymbol{b}_{\psi,\xi}^{\prime}\boldsymbol{M}_{\xi}^{-1} of the parameter estimates, whence the mse of the fitted values \hat{Y}\left(\boldsymbol{x}\right)=\boldsymbol{f}^{\prime}\left(\boldsymbol{x}\right)\boldsymbol{\hat{\theta}} is \text{{mse}}\left[\hat{Y}\left(\boldsymbol{x}\right)\right]=\frac{\sigma_{\varepsilon}^{2}}{n}\boldsymbol{f}^{\prime}\left(\boldsymbol{x}\right)\boldsymbol{M}_{\xi}^{-1}\boldsymbol{f}\left(\boldsymbol{x}\right)+\left(\boldsymbol{f}^{\prime}\left(\boldsymbol{x}\right)\boldsymbol{M}_{\xi}^{-1}\boldsymbol{b}_{\psi,\xi}\right)^{2}+\psi^{2}\left(\boldsymbol{x}\right). A loss function that is commonly employed is the integrated mse of the predictions : \displaystyle\text{{imse}}\left(\xi|\psi\right) \displaystyle= \displaystyle\int_{\mathcal{X}}\text{{mse}}\left[\hat{Y}\left(\boldsymbol{x}\right)\right]\mu\left(d\boldsymbol{x}\right) (4) \displaystyle= \displaystyle\frac{\sigma_{\varepsilon}^{2}}{n}tr\left(\boldsymbol{AM}_{\xi}^{-1}\right)+\boldsymbol{b}_{\psi,\xi}^{\boldsymbol{\prime}}\boldsymbol{M}_{\xi}^{-1}\boldsymbol{AM}_{\xi}^{-1}\boldsymbol{b}_{\psi,\xi}+\int_{\mathcal{X}}\psi^{2}\left(\boldsymbol{x}\right)\mu\left(d\boldsymbol{x}\right). The dependence on \psi is eliminated by adopting a minimax approach, according to which one first maximizes ( 4 ) over a neighbourhood of the assumed response. This neighbourhood is constrained by ( 3 ) and by a bound \int_{\mathcal{X}}\psi^{2}\left(\boldsymbol{x}\right)\mu\left(d\boldsymbol{x}\right)\leq\tau^{2}/n , required so that errors due to bias and to variation remain of the same order, asymptotically. Define \psi_{0}\left(\boldsymbol{x}\right)=\sqrt{n}\psi\left(\boldsymbol{x}\right)/\tau and \nu=\tau^{2}/\left(\sigma_{\varepsilon}^{2}+\tau^{2}\right) . Then \max_{\psi} imse \left(\xi|\psi\right) is \left(\sigma_{\varepsilon}^{2}+\tau^{2}\right)/n times I_{\nu}\left(\xi\right)=\left(1-\nu\right)\text{{var}}\left(\xi\right)+\nu\text{{bias}}\left(\xi\right), where var \left(\xi\right)=tr\boldsymbol{AM}_{\xi}^{-1} is the integrated variance of the predictors and bias \left(\xi\right)=\max_{\psi_{0}}\text{{bias}}\left(\xi|\psi_{0}\right) , where \text{{bias}}\left(\xi|\psi_{0}\right)=\boldsymbol{b}_{\psi_{0},\xi}^{\boldsymbol{\prime}}\boldsymbol{M}_{\xi}^{-1}\boldsymbol{AM}_{\xi}^{-1}\boldsymbol{b}_{\psi_{0},\xi}+1 (5) is the integrated (squared) bias, with \psi_{0} constrained by ( 3 ) and \int_{\mathcal{X}}\psi_{0}^{2}\left(\boldsymbol{x}\right)\mu\left(d\boldsymbol{x}\right)=1 . 3 Optimal Bounded Bias and Bounded Variance designs Let \Xi be a class of designs on \chi , for instance all probability measures on \left[-1,1\right] – requiring appropriate approximations to make them implementable – or all exact designs, i.e. with integer allocations, on a finite design space. For \min_{\xi\in\Xi}\text{{bias}}\left(\xi\right)\leq b^{2}\leq\max_{\xi\in\Xi}\text{{bias}}\left(\xi\right), (6) consider the problem (B): Minimize var \left(\xi\right) in the class \Xi_{b}\subset \Xi of designs for which bias \left(\xi\right)\leq b^{2} . We call a solution to (B) an Optimal Bounded Bias design with bias bound b^{2} , denoted OBB( b^{2} ). For \min_{\xi\in\Xi}\text{{var}}\left(\xi\right)\leq s^{2}\leq\max_{\xi\in\Xi}\text{{var}}\left(\xi\right), (7) consider the problem (S): Minimize bias \left(\xi\right) in the class \Xi_{s}\subset \Xi of designs for which var \left(\xi\right)\leq s^{2} . We call a solution to (S) an Optimal Bounded Variance design with variance bound s^{2} , denoted OBV( s^{2} ). For \nu\in\left[0,1\right] define \xi_{\nu}=\arg\min_{\xi}I_{\nu}\left(\xi\right) . Set b^{2}\left(\nu\right)= bias \left(\xi_{\nu}\right) and s^{2}\left(\nu\right)= var \left(\xi_{\nu}\right) . Note that then \xi_{\nu}\in\Xi_{b\left(\nu\right)}\cap\Xi_{s\left(\nu\right)}. In § 3.1 we prove and discuss Theorem 1 . This asserts that solutions to (B) and (S) are given by \xi_{\nu} , for some \nu\in\left[0,1\right] , for any b^{2} and any s^{2} . Conversely, any \xi_{\nu} is a solution to (B) and (S) for appropriate values of b^{2} and s^{2} . Theorem 1 . (a) Optimal Bounded Bias designs with bias bound b^{2} satisfying ( 6 ) are given by \xi=\left\{\begin{array}[]{cc}\xi_{\nu},&\text{if }b^{2}=b^{2}\left(\nu\right)\text{, for }0\leq\nu\leq 1\text{;}\\ \xi_{0},&\text{if }b^{2}\geq b^{2}\left(0\right)\text{.}\end{array}\right. (b) Optimal Bounded Variance designs with variance bound s^{2} satisfying ( 7 ) are given by \xi=\left\{\begin{array}[]{cc}\xi_{\nu},&\text{if }s^{2}=s^{2}\left(\nu\right)\text{, for }0\leq\nu\leq 1\text{;}\\ \xi_{1},&\text{if }s^{2}\geq s^{2}\left(1\right)\text{.}\end{array}\right. 3.1 Proof of Theorem 1 By the I-optimal design 4 we mean the minimizer of I_{0}\left(\xi\right) , i.e. of the Integrated Variance of the Predicted Values. By the uniform design we mean the design \xi\left(d\boldsymbol{x}\right)\propto\mu\left(d\boldsymbol{x}\right) . These designs play special roles – they turn out to be \xi_{0} and \xi_{1} , respectively. Lemma 1 . The design \xi_{0} , minimizing I_{0}\left(\xi\right)= var \left(\xi\right) in \Xi , is I-optimal and the design \xi_{1} , minimizing I_{1}\left(\xi\right)= bias \left(\xi\right) in \Xi , is uniform. Proof : That \xi_{0} is the I-optimal design follows from the definition: I_{0}\left(\xi\right)= var \left(\xi\right) . By ( 5 ), bias \left(\xi\right)\geq 1 . This lower bound is attained by the uniform design, since then \boldsymbol{b}_{\psi,\xi}=\int_{\mathcal{X}}\boldsymbol{f}\left(\boldsymbol{x}\right)\psi\left(\boldsymbol{x}\right)\xi\left(d\boldsymbol{x}\right)\propto\int_{\mathcal{X}}\boldsymbol{f}\left(\boldsymbol{x}\right)\psi\left(\boldsymbol{x}\right)\mu\left(d\boldsymbol{x}\right)=\boldsymbol{0}, by ( 3 ). \square By Lemma 1 , the lower bounds of the ranges ( 6 ) and ( 7 ) are attained by \xi_{1} and \xi_{0} respectively. Lemma 2 . (a) \xi_{\nu} is a solution to