Softmax is 1 / 2 1/2 -Lipschitz: A tight bound across all ℓ p \ell_{p} norms
Pravin Nair [email protected] · 2025
Read Softmax is 1 / 2 1/2 -Lipschitz: A tight bound across all ℓ p \ell_{p} norms by Pravin Nair [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23012.txt.
Introduction
Pravin Nair proves a sharper bound on how much the softmax function can stretch its inputs. Softmax appears in classification, attention, reinforcement learning, and game theory. Most guarantees treated its Lipschitz constant as one in the Euclidean norm, which may overstate how unstable models can be.
Nair shows the constant is one half, and that this holds for every Lp norm with p at least one. He also studies the local constant at each point. A tighter bound can improve robustness proofs and convergence checks for the many algorithms that rely on softmax, at no extra cost in practice.
Catalog record
| CREATOR | Pravin Nair [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-23012
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Softmax is 1 / 2 1/2 -Lipschitz: A tight bound across all ℓ p \ell_{p} norms Pravin Nair [email protected] Abstract The softmax function is a basic operator in machine learning and optimization, used in classification, attention mechanisms, reinforcement learning, game theory, and problems involving log-sum-exp terms. Existing robustness guarantees of learning models and convergence analysis of optimization algorithms typically consider the softmax operator to have a Lipschitz constant of 1 1 with respect to the ℓ 2 \ell_{2} norm. In this work, we prove that the softmax function is contractive with the Lipschitz constant 1 / 2 1/2 , uniformly across all ℓ p \ell_{p} norms with p ≥ 1 p\geq 1 . We also show that the local Lipschitz constant of softmax attains 1 / 2 1/2 for p = 1 p=1 and p = ∞ p=\infty , and for p ∈ ( 1 , ∞ ) p\in(1,\infty) , the constant remains strictly below 1 / 2 1/2 and the supremum 1 / 2 1/2 is achieved only in the limit. To our knowledge, this is the first comprehensive norm-uniform analysis of softmax Lipschitz continuity. We demonstrate how the sharper constant directly improves a range of existing theoretical results on robustness and convergence. We further validate the sharpness of the 1 / 2 1/2 Lipschitz constant of the softmax operator through empirical studies on attention-based architectures (ViT, GPT-2, Qwen3-8B) and on stochastic policies in reinforcement learning. 1 Introduction The softmax function has applications across diverse areas of machine learning and optimization. In classification models, model output logits is normalized using the softmax function to form a probability distribution ( Goodfellow et al. 2016 ) . In Transformer architectures, softmax normalizes attention scores to compute weighted combinations of input token features, enabling high-quality feature refinement ( Vaswani et al. 2017 ) . These transformer architectures have shown state-of-the-art performance in various applications in natural language processing ( Tunstall et al. 2022 ) , computer vision ( Khan et al. 2022 ) , reinforcement learning ( Li et al. 2023 ) , etc. Since softmax is the gradient of the log-sum-exp function, its properties are crucial for analyzing optimization algorithms ( Gao & Pavel 2017 ; Nachum et al. 2017 ; Nesterov 2005 ) . In reinforcement learning, softmax is used to convert action-value estimates (Q-values) to stochastic policies. This results in a probability distribution over actions that favours actions with higher Q-values and still assigns probabilities to suboptimal actions. In entropy-regularized reinforcement learning, the softmax operator arises naturally as the solution of the policy optimization problem ( Sutton & Barto 2018 ; Nachum et al. 2017 ) . The softmax operator maps any real vector to a probability distribution. Formally, the softmax function \sigma_{\lambda}:\mathbb{R}^{n}\to\Delta_{n}^{\circ} with inverse–temperature parameter \lambda>0 is defined as \sigma_{\lambda}({\bm{x}})_{i}\;=\;\frac{\exp(\lambda x_{i})}{\sum_{j=1}^{n}\exp(\lambda x_{j})},\qquad i=1,\dots,n, (1) where \Delta_{n}=\{{\bm{u}}\in\mathbb{R}^{n}:u_{i}\geq 0,\sum_{i}u_{i}=1\} is the probability simplex and \Delta_{n}^{\circ}=\{{\bm{u}}\in\mathbb{R}^{n}:u_{i}>0,\sum_{i}u_{i}=1\} denotes its interior. Since \exp(x_{i})>0 for all x_{i}\in\mathbb{R} , \sigma_{\lambda}({\bm{x}}) never attains the boundary of the simplex \partial\Delta_{n}=\{{\bm{u}}\in\mathbb{R}^{n}:u_{i}=0\text{ for some }i\} . The coefficient \lambda measures the smoothness of the output distribution. A larger \lambda makes the distribution more peaked on the largest entries, and a smaller \lambda makes it smoother with more evenly spread probability values. The standard softmax function is when \lambda=1 . Accurate analysis of softmax’s Lipschitz constant has important applications across machine learning and optimization. It enables accurate robustness analysis of attention mechanisms, where Lipschitz bounds can be used to set hyperparameters that ensure stable training ( Qi et al. 2023 ) . In game-theoretic learning, modeling action selection via softmax allows Lipschitz continuity to be directly leveraged for establishing convergence rates toward Nash equilibria ( Gao & Pavel 2017 ) . Many existing generalization results for learning models require computing the Lipschitz constant of the loss function with respect to either network input or network parameters. When softmax appears in the final layer or in attention, an accurate Lipschitz analysis becomes essential for deriving sharp generalization bounds ( Asadi & Abbe 2020 ) . In classification networks, robustness and domain adaptation depend on how sensitive output probabilities are to input perturbations, which is again controlled by the Lipschitz constant of softmax ( Chen et al. 2022 ) . The log-sum-exp (LSE) function is a smooth approximation of the maximum operator, and the gradient of LSE is the softmax function. Hence, in optimization problems with LSE-based objective functions, step-size rules for gradient descent and smoothness-based convergence rates can be sharpened using a tight softmax Lipschitz bound. Lipschitz constant of softmax is also used in analysing several other frameworks, including meta-learning ( Jeon et al. 2024 ) , structural causal models ( Le Priol et al. 2021 ) , prior-data fitted networks ( Nagler 2023 ) , and sparse training ( Lei et al. 2024 ) . The above-mentioned diverse set of applications highlights that deriving a tight Lipschitz constant of softmax has a substantial practical impact. Despite the theoretical and empirical significance of softmax’s Lipschitz constant, an accurate analysis is missing in the literature. A number of works ( Gao & Pavel 2017 ; Laha et al. 2018 ; Asadi & Abbe 2020 ; Gouk et al. 2021 ; Le Priol et al. 2021 ; Chen et al. 2022 ; Nagler 2023 ; Jeon et al. 2024 ; Lei et al. 2024 ) assume the softmax function to be 1 -Lipschitz with respect to the \ell_{2} norm. This assumption is often attributed to Proposition 4 in Gao & Pavel 2017 . In some works, this is mentioned as a straightforward fact, perhaps because showing the Lipschitz constant to be bounded by 1 is trivial. We also note that Xu et al. 2022 reports a bound of 1/4 for the softmax’s Lipschitz constant. The main motivation for this work is the discrepancy between the commonly assumed Lipschitz constant of the softmax function and the smaller values observed in our empirical studies on transformer architectures. This prompted us to derive a tighter bound for the Lipschitz constant of the softmax function. In this regard, our contributions are as follows: • We prove that softmax is 1/2 -Lipschitz across all \ell_{p} norms ( p\geq 1 ), thereby improving the usually assumed bound of 1 . • Furthermore, we show the tightness of this bound by proving that the local Lipschitz constant is attained for p=1 and p=\infty as 1/2 , while for p\in(1,\infty) the local Lipschitz constants remain strictly below 1/2 and the supremum is only approached in the limit. • We demonstrate that our tight Lipschitz bound improves the existing theoretical results in robustness and convergence. We also validate our derived Lipschitz constant through experiments on large-scale transformer models and reinforcement learning policies. 2 Preliminaries In this section, we provide the necessary background for our main results. We start with the definition of the \ell_{p} norm for vectors and the corresponding induced operator norm for matrices. Definition 2.1 ( \ell_{p} norm) . For a vector {\bm{x}}=(x_{1},x_{2},\dots,x_{n})\in\mathbb{R}^{n} and for 1\leq p<\infty , the \ell_{p} norm of {\bm{x}} is defined as \|{\bm{x}}\|_{p}\;=\;\left(\sum_{i=1}^{n}|x_{i}|^{p}\right)^{\!1/p}. For p=\infty , the \ell_{\infty} norm is defined as \|{\bm{x}}\|_{\infty}\;=\;\max_{1\leq i\leq n}|x_{i}|. We can also define the operator norm of a matrix {\bm{A}}=(A_{ij})\in\mathbb{R}^{m\times n} induced by the \ell_{p} norm as \|{\bm{A}}\|_{p}\;:=\;\sup_{{\bm{v}}\neq 0}\frac{\|{\bm{A}}{\bm{v}}\|_{p}}{\|{\bm{v}}\|_{p}}\;=\;\sup_{\|{\bm{v}}\|_{p}=1}\|{\bm{A}}{\bm{v}}\|_{p}. Throughout