Risk reversal for least squares estimators under nested convex constraints

Omar Al-Ghattas · 2026

Read Risk reversal for least squares estimators under nested convex constraints by Omar Al-Ghattas (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16041.txt.

Introduction

Omar Al-Ghattas studies a sharp twist in least squares fits under tight rules. His 2026 preprint sits in the field of chance based search for best fits. Most would think a tighter set of rules can not raise risk. Risk here means the mean loss of the fit. The fit is made by casting data onto the allowed set.

Al-Ghattas shows that this plain hope can fail. It fails even in plain, well known cases. A stricter rule set can in fact leave a worse fit. That flip warns teams who add rules to cut risk. They must test the effect, not just trust the rule. The proof gives a firm base for that care.

Catalog record

CREATOROmar Al-Ghattas
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE90 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-16041

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Risk reversal for least squares estimators under nested convex constraints Omar Al-Ghattas Abstract In constrained stochastic optimization, one naturally expects that imposing a stricter feasible set does not increase the statistical risk of an estimator defined by projection onto that set. In this paper, we show that this intuition can fail even in canonical settings. We study the Gaussian sequence model, a canonical test bed that offers a deliberately austere setting for the phenomenon considered here, where for a compact, convex set Θ ⊂ ℝ d \Theta\subset\mathbb{R}^{d} one observes Y = θ ⋆ + σ ​ Z , Z ∼ N ⁡ ( 0 , I d ) , Y=\theta^{\star}+\sigma Z,\qquad Z\sim N(0,I_{d}), and seeks to estimate an unknown parameter θ ⋆ ∈ Θ \theta^{\star}\in\Theta . The natural estimator in this setting is the least squares estimator (LSE), which coincides with the Euclidean projection of Y Y onto Θ \Theta . We construct an explicit example exhibiting risk reversal : for sufficiently large noise, there exist nested compact convex sets Θ S ⊂ Θ L \Theta_{S}\subset\Theta_{L} and a parameter θ ⋆ ∈ Θ S \theta^{\star}\in\Theta_{S} such that the LSE constrained to Θ S \Theta_{S} has strictly larger risk than the LSE constrained to Θ L \Theta_{L} . Moreover, we demonstrate that risk reversal can persist at the level of worst-case risk, in the sense that the supremum of the risk over the smaller constraint set exceeds that over the larger one. We clarify this phenomenon by contrasting the behavior of the LSE across noise regimes. In the vanishing-noise limit, the risk admits a first-order expansion governed by the statistical dimension of the tangent cone at θ ⋆ \theta^{\star} , and risk reversal cannot occur: tighter constraints uniformly reduce risk. In contrast, in the diverging-noise regime, the risk is determined by global geometric interactions between the constraint set and random noise directions. Here, the embedding of Θ S \Theta_{S} within Θ L \Theta_{L} , rather than the local geometry at θ ⋆ \theta^{\star} , plays a decisive role in the behavior of the LSE and can reverse the risk ordering. These results reveal a previously unrecognized failure mode of the LSE, and of projection-based estimators more broadly. They demonstrate that in sufficiently noisy settings, tightening a constraint can paradoxically degrade statistical performance. tablesection algorithmsection 1 Introduction In statistical estimation, it is standard to analyze estimators under the assumption that the unknown parameter belongs to a prescribed feasible set. Such constraints encode prior structural information, reduce the effective complexity of the parameter space, and are therefore widely believed to improve estimation accuracy. This perspective naturally fosters the intuition that, when a constraint is correctly specified, further restricting the feasible set in a way that preserves the truth should not worsen performance. In applied work, constraints rarely appear fully formed. More often, a statistician begins with a simple modeling assumption, such as boundedness or smoothness, and then gradually refines the model as additional insight becomes available. New constraints may be introduced to reflect domain knowledge, improve interpretability, or enforce desired qualitative behavior. Each refinement is typically viewed as a step toward a more faithful representation of the underlying problem, and thus toward better statistical performance. From this perspective, it is far from obvious that adding valid structure could ever worsen statistical performance. We study this question in the Gaussian sequence model, one of the simplest and most benign settings for constrained estimation, under well-behaved (namely, compact and convex) constraints. As we will argue, this model also provides a deliberately austere setting in which to exhibit the phenomenon we identify. Here, the natural estimator is the least squares estimator (LSE), which coincides with the maximum likelihood estimator and admits a geometric characterization as the Euclidean projection onto the constraint set. Although this formulation is canonical, existing analyses of the LSE typically rely on local approximations near the true parameter and therefore do not capture how the estimator’s risk depends on the global geometry of the constraint set. When two compact, convex constraint sets are nested and the smaller set contains the true parameter, one might expect the corresponding LSE to have lower risk. We show that this intuition can fail: there exist settings in which the LSE associated with the smaller constraint has strictly larger risk than that associated with the larger constraint, both pointwise at a fixed parameter and at the level of worst-case risk. 1.1 Problem Setup We now introduce the mathematical framework that will serve as the basic object of study throughout the paper. Let \Theta_{S},\Theta_{L}\subset\mathbb{R}^{d} be compact, convex sets satisfying \Theta_{S}\subset\Theta_{L}, and suppose the unknown parameter \theta^{\star} lies in \Theta_{S} . We observe a single realization from the Gaussian sequence model (1) \displaystyle Y=\theta^{\star}+\sigma Z,\qquad Z\sim N(0,I_{d}), where \sigma>0 denotes the noise level. Our goal is to estimate \theta^{\star} from the noisy observation Y . For a compact, convex set \Theta\subset\mathbb{R}^{d} , define the Euclidean projection of a point y\in\mathbb{R}^{d} onto \Theta by \Pi_{\Theta}(y):=\arg\min_{\theta\in\Theta}\|y-\theta\|^{2}. Since \Theta is compact and convex, the minimizer exists and is unique. The estimator \Pi_{\Theta}(Y) is the LSE, which coincides with the maximum likelihood estimator under the constraint \theta^{\star}\in\Theta. We consider the two LSEs \widehat{\theta}_{S}:=\Pi_{\Theta_{S}}(Y),\qquad\widehat{\theta}_{L}:=\Pi_{\Theta_{L}}(Y), and evaluate their performance using the squared-error risk R_{\sigma}(\theta^{\star};\Theta):=\mathbb{E}_{\theta^{\star}}\bigl\|\widehat{\theta}-\theta^{\star}\bigr\|^{2}. Throughout the paper, the notation R_{\sigma}(\theta^{\star};\Theta) always denotes the risk of the estimator \Pi_{\Theta}(\theta^{\star}+\sigma Z). The inclusion \Theta_{S}\subset\Theta_{L} suggests a natural comparison. The estimator \widehat{\theta}_{S} enforces a stronger constraint and thus incorporates more prior information than \widehat{\theta}_{L} , while remaining correctly specified whenever \theta^{\star}\in\Theta_{S} . This leads to the central question of this work: can strengthening the constraint ever increase the estimation risk? In particular, do there exist settings in which R_{\sigma}(\theta^{\star};\Theta_{S})>R_{\sigma}(\theta^{\star};\Theta_{L}), a phenomenon we refer to as risk reversal . Beyond this pointwise comparison, we also ask whether an analogous reversal can occur at the level of worst-case risk over the constraint set. Remark 1.1 (Multiple observations). Throughout this work we state results for a single observation of ( 1 ) only for notational simplicity and without loss of generality. If instead one observes n\geq 1 independent samples Y_{1},\dots,Y_{n}\sim N(\theta^{\star},\sigma^{2}I_{d}) , the LSE is simply the orthogonal projection of the sample mean \mkern 1.5mu\overline{\mkern-1.5muY\mkern-1.5mu}\mkern 1.5mu onto the constraint set. Since \mkern 1.5mu\overline{\mkern-1.5muY\mkern-1.5mu}\mkern 1.5mu is Gaussian with mean \theta^{\star} and variance \tfrac{\sigma^{2}}{n}I_{d}, the n sample problem is exactly equivalent to the single-sample model with effective noise level \tfrac{\sigma}{\sqrt{n}}. In particular, all diverging-noise results of this paper continue to hold verbatim whenever the effective noise level diverges. ∎ y_{1} y_{2} A_{1} \widehat{\theta}_{S}=(0,0) A_{12} \widehat{\theta}_{S}=(\frac{y_{1}+cy_{2}}{1+c^{2}},\frac{c(y_{1}+cy_{2})}{1+c^{2}}) A_{2} \widehat{\theta}_{S}=(\tfrac{1}{c},1) v_{1}=(0,0) v_{2}=(\tfrac{1}{c},1) \Theta_{S} \star \theta^{\star} y_{1} y_{2} A_{3} \widehat{\theta}_{L}=(0,1) A_{23} \widehat{\theta}_{L}=(y_{1},1) A_{2} \widehat{\theta}_{L}=(\tfrac{1}{c},1) A_{13} \widehat{\theta}_{L}=(0,y_{2}) \Theta_{L} \widehat{\theta}_{L}=(y_{1},y_{2}) A_{1} \widehat{\theta}_{L}=(0,0) A_{12} \widehat{\theta}_{L}=(\frac{y_{1}+cy_{2}}{1+c^{2}},\frac{c(y_{1}+cy_{2})}{1+c^{2}}) v_{1}=(0,0) v_{3}=(0,1) v_{2}=(\tfrac{1}{c},1) \star \theta^{\star} Figure 1 : Geometry of the LSE over constraint sets \Theta_{S}=\operatorname{conv}(v_{1},v_{2}) (left) and \Theta_{L}=\operatorname{conv}(v_{1},v_{2},v_{3}) (right). The shaded regions illustrate the partition of the sample space based on the location of the constrained solution: regions A_{j} denote observations projected onto vertex v_{j} , while regions A_{ij} denote observations projected onto the line segment connecting v_{i} and v_{j} . 1.2 Risk reversal: an explicit example We now describe the core phenomenon identified in this work by constructing a concrete example in which risk reversal occurs. Specifically, we construct a simple two-dimensional example in

Read the complete text