Everywhere Learning: Artificial Intelligence with Pointwise Constraints

Ignacio Boero*, Ignacio Hounie*, Luiz Chamon, Alejandro Ribeiro · 2026

Read Everywhere Learning: Artificial Intelligence with Pointwise Constraints by Ignacio Boero*, Ignacio Hounie*, Luiz Chamon, Alejandro Ribeiro (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01557.txt.

Introduction

Boero and co-authors define everywhere learning, a stricter training paradigm. Standard AI training minimizes average loss over data. Everywhere learning demands loss constraints hold with probability one. Each point in the data distribution must satisfy them. An approximate duality theory supports their generalization analysis. It shows how close empirical and statistical solutions sit. Dual variables reweigh data toward hard-to-satisfy points.

Generalization turns on a mismatch of mass concentration. It compares where data mass sits versus where constraints bind hardest. A sparse L1 penalty on relaxations can control that gap. An experiment in agentic classification illustrates the merits. Language model tasks supply that test case. The paradigm suits settings where average performance is not enough.

Catalog record

CREATORIgnacio Boero*, Ignacio Hounie*, Luiz Chamon, Alejandro Ribeiro
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE125 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01557

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Everywhere Learning: Artificial Intelligence with Pointwise Constraints Ignacio Boero*, Ignacio Hounie*, Luiz Chamon, Alejandro Ribeiro Abstract Everywhere learning is a new paradigm whereby Artificial Intelligence (AI) systems are trained to satisfy loss constraints with probability one over the data distribution. This is in contrast to the standard paradigm of training AI systems to minimize average losses. We develop an approximate duality theory to substantiate a generalization analysis that establishes the proximity between solutions of empirical and statistical everywhere learning problems. Our results show that dual variables reweigh the data distribution towards points in which loss constraints are more difficult to satisfy and that generalization is controlled by the mismatch between the concentration of mass of the data distribution and the concentration of mass on points where constraints are more difficult to satisfy. We further show that we can control generalization with a sparse L1 penalty on constraint relaxations. We illustrate the merits of everywhere learning with an experiment in agentic classification for language model tasks. I Introduction We study the learnability of everywhere learning problems, a paradigm in which AI systems are trained to satisfy constraints with probability one over the data distribution. This paradigm departs from standard statistical risk minimization (SRM), which trains models by minimizing the expected value of a loss [ 32 ] . While SRM is effective for optimizing a single objective, it is not well suited for modern AI systems, whose deployment requires satisfying multiple, often competing, design criteria [ 34 , 37 , 23 ] . Everywhere learning separates objectives—what we want a model to optimize—from requirements—what the model needs to satisfy. This prevents trade-offs in which violations of one requirement is compensated by improvements in other requirement, as occurs in many extensions of SRM that aggregate multiple losses through weighted sums [ 31 , 41 , 13 , 15 , 14 ] . Existing work has studied constrained learning formulations in which requirements are imposed on expectation over the data distribution [ 38 , 25 , 20 , 10 , 16 ] . However, enforcing constraints only in expectation provides no guarantee on how often violations occur, nor on how large they can be at individual data points. A constraint may be satisfied on average while certain regions of the input space systematically fail to meet the requirement. These regions may correspond to specific users, operating regimes, or safety-critical scenarios. The issue becomes even more pronounced with multiple constraints, as different regions may violate different requirements, so all constraints can be satisfied in expectation even though no region satisfies them all simultaneously. This motivates imposing constraints pointwise, enforcing each requirement to hold almost everywhere over the corresponding data distribution. Everywhere learning, like any statistical learning problem, assumes that data distributions are unknown, and are accessed only through samples. We therefore need to understand when the statistical problem can be approximated, with high probability, from finitely many samples. To this end, we introduce a probably approximately correct constrained (PACC) notion of learnability for everywhere learning (Definition 3 ). This definition extends classical PAC learnability [ 36 ] by requiring both approximate optimality of the objective value and approximate feasibility of the pointwise constraints. However, unlike in standard SRM, uniform convergence of the objective and constraint losses is not sufficient to guarantee learnability (Example 1 ). This motivates the study of learnability for everywhere learning problems. For that, we turn to the dual domain. We study when the dual problem can itself be solved from finite samples, a property we call dual learnability (Definition 4 ). We then show how dual learnability transfers back to PACC learnability of the primal problem. The dual formulation introduces a multiplier \lambda(\bm{z}) that reweights the constraint violation at each sample \bm{z} . We show that a sufficient condition for dual learnability is that the optimal multiplier has bounded L_{\infty} norm (Theorem 1 ). This condition has a natural interpretation. The multiplier \lambda(\bm{z}) measures the sensitivity of the constrained optimum to perturbations of the constraint at \bm{z} , normalized by the probability of observing that sample. Large values of \lambda(\bm{z}) therefore identify regions that have high impact on the constrained optimum but are unlikely to appear in a finite sample set (Subsection III-D ). Although natural, the boundedness condition on \lambda^{\star} need not hold in practice. The constrained optimum may be highly sensitive to regions that have small probability under the data distribution. We therefore enforce the condition directly by restricting the dual variables to have bounded L_{\infty} norm. This restriction has a simple primal interpretation. It is equivalent to introducing an L^{1} -penalized relaxation of the pointwise constraints, which encourages the relaxation to be sparse (Proposition 2 ). This yields a principled relaxation strategy in which better generalization is obtained by sparsely relaxing the pointwise constraints and modifying the original problem only where necessary. Moreover, this relaxation is simple to implement in practice because it amounts to clipping the empirical dual variables (Theorem 2 ). We then relate dual learnability back to primal learnability. In convex problems, strong duality allows dual guarantees to transfer directly to PACC learnability (Theorem 3 ), and solving either the empirical primal problem or the empirical dual problem yields the same statistical guarantees (Proposition 2 ). Modern AI parametrizations, however, are typically nonconvex. For this setting, we show that primal learnability can still be recovered up to an approximation term controlled by the richness of the hypothesis class. In particular, when \mathcal{H} approximates its convex hull well, the duality gap is small and the empirical primal rule is near-PACC learnable (Theorem 4 ). For the empirical dual rule, the guarantee requires additional care because primal recovery is no longer automatic. We prove that there exists a Lagrangian minimizer satisfying near-PACC guarantees, with an additional term depending on the empirical complementary-slackness error (Proposition 3 ). We conclude with numerical experiments that validate the practical benefits of everywhere learning. The results show that enforcing pointwise constraints can outperform weighted-sum and penalty-based methods, that the proposed dual regularization can improve generalization, and that dual methods recover high-quality solutions even in nonconvex settings. I-A Related Work Constrained Learning . The constrained learning literature studies learning frameworks where predictors are trained to satisfy requirements instead of minimizing an objective. Although recent works have demonstrated the practical value of learning under pointwise constraints [ 27 , 30 ] , the theoretical understanding of such formulations remains limited. In particular, the work in [ 8 ] establishes learning guarantees for both expected-value and \mathcal{D} -almost everywhere constraints, but under a weaker notion of learnability, whereas [ 10 ] strengthens these guarantees only in the case of expected-value constraints. Our work builds on these results by extending the stronger learnability guarantees of [ 10 ] to the setting of \mathcal{D} -almost everywhere constraints. Robust Optimization . Robust optimization studies decision-making under uncertainty by seeking solutions that remain feasible, and often near-optimal, for all realizations of uncertain parameters within a prescribed uncertainty set [ 3 , 5 , 35 ] . At a high level, our framework shares with robust optimization the objective of controlling adverse outcomes. However, the distinction is that in robust optimization the uncertainty is exogenous and encoded through a

Read the complete text