Position: Adopt Constraints Over Fixed Penalties in Deep Learning

Juan Ramirez, Meraj Hashemizadeh, Simon Lacoste-Julien † · 2025

Read Position: Adopt Constraints Over Fixed Penalties in Deep Learning by Juan Ramirez, Meraj Hashemizadeh, Simon Lacoste-Julien † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20628.txt.

Introduction

Juan Ramirez, Meraj Hashemizadeh, and Simon Lacoste-Julien argue that deep learning should use hard constraints rather than fixed penalties for its requirements, in work from 2025. Models increasingly face set limits, such as safety levels, fairness caps, and sparsity budgets. In practice those limits are usually added to the loss with a fixed weight.

The authors give three reasons that habit fails. In non convex problems the penalized and constrained versions need not match. A weight sets a trade off, not the limit itself, so finding a good weight takes costly trial runs. And a penalty turns a firm rule into a soft one that training can trade away. Constraints keep the rule firm.

Catalog record

CREATORJuan Ramirez, Meraj Hashemizadeh, Simon Lacoste-Julien †
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20628

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Position: Adopt Constraints Over Fixed Penalties in Deep Learning Juan Ramirez, Meraj Hashemizadeh, Simon Lacoste-Julien † Abstract Recent efforts to develop trustworthy AI systems have increased interest in learning problems with explicit requirements, or constraints . In deep learning, however, such problems are often handled through fixed weighted-sum penalization : the constraints are added to the task loss with fixed coefficients, and the resulting scalarized objective is minimized. This position paper argues that fixed penalization is often ill-suited for deep learning problems with non-negotiable requirements for several reasons. First, in non-convex settings, the penalized and constrained problems are generally not equivalent, so solving the former need not solve the latter. Second, fixed penalization weakens hard requirements into soft penalties to be traded off against task performance. Third, choosing penalty coefficients to indirectly solve the constrained problem often involves costly trial and error, because changing them alters the penalized objective itself, and hence can mean solving the wrong problem altogether. We therefore argue that, when a deep learning problem specifies non-negotiable requirements, the constrained formulation itself should be the starting point, not the surrogate problem defined by fixed penalization. The appropriate solution strategy should then be chosen based on the problem’s structure and scale. 1 Introduction Recent advances in deep learning have enabled the deployment of AI systems across a wide range of domains, including high-stakes applications such as credit risk assessment [ 5 ] , medical imaging [ 45 ] , and autonomous driving [ 1 ] . As these systems become more capable and widely used, concerns about reliability, fairness, interpretability, and safety have become increasingly central [ 15 ] , as have calls for stronger oversight and regulation [ 19 ] . At the same time, many requirements arising from these concerns are now stated in thresholded, externally meaningful terms: acceptable safety levels, fairness limits, compute or sparsity budgets, and related quantities that must be monitored explicitly. A natural way to address such requirements is to encode them directly into training. This leads to constrained learning : rather than optimizing predictive performance alone, one seeks models that optimize performance subject to explicit requirements [ 22 , 9 , 12 ] . When these quantities represent genuine requirements rather than soft preferences, the constrained formulation is appealing because it builds compliance into the problem specification itself, thereby making success auditable. In such settings, formulation choice is not merely a matter of implementation, because different formulations need not agree on which models qualify as acceptable. In practice, constraints in deep learning are often handled through fixed weighted-sum penalization : one drops the hard constraints, adds them to the objective with fixed coefficients, and minimizes the resulting scalarized loss [ 7 , 30 , 33 , 6 ] ; see § 2 . This approach is easy to implement, requires little beyond differentiability, and fits naturally into standard unconstrained training pipelines. That convenience, however, often comes at a substantial cost. Our position is as follows: When a deep learning problem specifies genuine non-negotiable requirements, fixed weighted-sum penalization is often the wrong default. In such cases, one should start from the constrained formulation, and then choose a solution strategy based on the problem’s structure and scale. First, in non-convex settings, fixed weighted-sum penalization is not in general equivalent to the constrained formulation, so solving the penalized problem need not solve the intended constrained one. Second, choosing penalty coefficients is not an ordinary hyperparameter-tuning problem. Because the coefficient determines the trade-off being optimized, a poor choice can mean solving the wrong problem altogether rather than merely solving the intended constrained problem poorly. In practice, this often turns coefficient selection into costly repeated trial-and-error retraining. Third, more fundamentally, fixed penalization weakens hard requirements into soft penalty terms that can be traded off against task performance. As a result, feasibility is no longer built into the formulation or the optimization procedure itself, but instead becomes something to be checked only after training. In § 2 , we formalize fixed weighted-sum penalization and explain why it can appear justified, especially in convex settings. In § 3 , we show why this justification breaks down in the non-convex regime relevant to deep learning. In § 4 , we discuss how tailored constrained methods can address these failures, using the Lagrangian approach as an illustration. We then turn to the main counterarguments and practical implications: § 5 discusses the limits of our position and the challenges that remain in constrained deep learning, while § 6 distills the discussion into practical guidance. Scope. We do not argue that every learning problem should be cast as constrained. When a quantity is better viewed as a soft preference, a regularizer, or one objective among several, penalties or multi-objective formulations may be more appropriate. Our critique begins once the problem incorporates non-negotiable requirements: in that case, we argue against solving it in deep learning through fixed weighted-sum penalization . Moreover, this is not a critique of penalties in general, which remain valid in many settings. Finally, our use of Lagrangian methods is illustrative; the appropriate constrained optimization method depends on the specific problem structure and scale. Related Work. Our position draws on several existing strands of work. In multi-objective optimization, it is classical that weighted-sum scalarizations can fail to recover all optimal trade-offs in non-convex settings [ 16 , §4] . In deep learning, fixed penalties have long been criticized [ 13 , 14 , 21 ] , and their tuning burden often motivates tailored constrained optimization alternatives [ 9 , 48 , 12 , 22 ] . Contribution. This paper advances a formulation-level claim about constrained deep learning: when a quantity encodes a genuine non-negotiable requirement, handling it through a fixed weighted-sum penalty is often a mismatch between the problem specification and the optimization problem actually being solved. Prior critiques of fixed penalties in deep learning have largely appeared in fragmented form, including classical observations about weighted-sum scalarization in non-convex optimization, tunability critiques of linearly combined losses, and application-specific arguments for constrained methods in areas such as fairness, sparsity, and safety. We bring these observations together into a unified formulation-level perspective and distill a practical decision framework for choosing among penalties, constraints, and constrained optimization methods. 2 Fixed Weighted-Sum Penalization Let f:\mathcal{X}\rightarrow\mathbb{R} , \bm{g}:\mathcal{X}\rightarrow\mathbb{R}^{m} , and \bm{h}:\mathcal{X}\rightarrow\mathbb{R}^{n} be differentiable functions over \mathcal{X}\subseteq\mathbb{R}^{d} . We consider the constrained optimization problem \min_{\bm{x}\in\mathcal{X}}\,f(\bm{x})\quad\text{s.t.}\quad\bm{g}(\bm{x})\preceq\bm{\epsilon}_{\bm{g}}\quad\text{and}\quad\bm{h}(\bm{x})=\bm{\epsilon}_{\bm{h}}, (1) where \bm{\epsilon}_{\bm{g}} and \bm{\epsilon}_{\bm{h}} are the target constraint levels, \preceq denotes element-wise inequality, and \bm{g} and \bm{h} are the inequality and equality constraints, respectively. A point \bm{x}\in\mathcal{X} is feasible if it satisfies the constraints, and a feasible point \bm{x}^{*} is a constrained minimizer if f(\bm{x}^{*})\leq f(\bm{x}) for all feasible \bm{x} . In machine learning, f typically captures task performance, while the constraints encode requirements such as sparsity [ 22 ] , fairness [ 9 ] , or safety [ 12 ] . The key distinction from multi-objective optimization is that here \bm{g} and \bm{h} are not quantities to be optimized: they represent requirements to be satisfied . Solutions that violate them are inadmissible, regardless of their objective value. 1 1 1 In practice, small numerical violations of the constraints are often tolerated. A common way to tackle Equation 1 in

Read the complete text