The Curvature Rate λ \lambda : A Scalar Measure of Input-Space Sharpness in Neural Networks

Jacob Poschl · 2025

Read The Curvature Rate λ \lambda : A Scalar Measure of Input-Space Sharpness in Neural Networks by Jacob Poschl (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01438.txt.

Introduction

Jacob Poschl, in a single author paper, measures how sharply a neural network's output curves in input space. Curvature affects generalization, robustness, and how reliably a network responds to small input changes. But standard sharpness metrics live in parameter space, where they can be costly, hard to interpret, and sensitive to reparameterization.

His curvature rate is the exponential growth rate of higher order input derivatives, estimated as the slope of log derivative size against derivative order. For classical functions it recovers the inverse radius of convergence, and for bandlimited signals the spectral cutoff. On neural networks it tracks high frequency structure emerging in the decision boundary during training. A simple regularizer based on it matched a leading sharpness method's accuracy while giving flatter input space geometry and better calibrated confidence.

Catalog record

CREATORJacob Poschl
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-01438

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Curvature Rate λ \lambda : A Scalar Measure of Input-Space Sharpness in Neural Networks Jacob Poschl Abstract Curvature influences generalization, robustness, and how reliably neural networks respond to small input perturbations. Existing sharpness metrics are typically defined in parameter space (e.g., Hessian eigenvalues) and can be expensive, sensitive to reparameterization, and difficult to interpret in functional terms. We introduce a scalar curvature measure defined directly in input space: the curvature rate λ \lambda , given by the exponential growth rate of higher-order input derivatives. Empirically, λ \lambda is estimated as the slope of log ⁡ ‖ D n ​ f ‖ \log\|D^{n}f\| versus n n for small n n . This growth-rate perspective unifies classical analytic quantities: for analytic functions, λ \lambda corresponds to the inverse radius of convergence, and for bandlimited signals, it reflects the spectral cutoff. The same principle extends to neural networks, where λ \lambda tracks the emergence of high-frequency structure in the decision boundary. Experiments on analytic functions and neural networks (Two Moons and MNIST) show that λ \lambda evolves predictably during training and can be directly shaped using a simple derivative-based regularizer, Curvature Rate Regularization (CRR). Compared to Sharpness-Aware Minimization (SAM), CRR achieves similar accuracy while yielding flatter input-space geometry and improved confidence calibration. By grounding curvature in differentiation dynamics, λ \lambda provides a compact, interpretable, and parameterization-invariant descriptor of functional smoothness in learned models. 1. Introduction Curvature—understood as the sensitivity of a learned function to small perturbations in inputs or parameters—plays a key role in generalization, robustness, and optimization stability. Empirically, solutions found in flatter regions of the loss landscape tend to generalize better, while highly curved regions correlate with overfitting and brittle behavior. Despite this importance, the literature seems to lack a single interpretable scalar that captures curvature consistently across architectures and training regimes. Existing sharpness metrics typically rely on Hessian spectra or parameter-space perturbations, which are expensive to compute, sensitive to reparameterization, and difficult to interpret in functional terms. We instead seek a curvature quantity that is intrinsic to the function itself and directly measurable during training. To this end, we introduce \lambda , a scalar defined from differentiation dynamics—the exponential growth rate of higher-order input derivatives. Rather than examining curvature through the Hessian, we study how the function behaves under repeated differentiation: \lambda\approx\text{slope}(\log||D^{n}f||\text{ vs. }n) where D^{n}f denotes the norm of the n-th input derivative, estimated via a linear fit over small n . Intuitively, positive \lambda indicates derivative growth and sharp, high-frequency structure, while negative \lambda indicates decay and smoothness. This definition recovers classical quantities across analysis and signal theory: • For analytic functions, \lambda=-\log R , the inverse radius of convergence. • For bandlimited signals, \lambda=\log\Omega , the spectral edge Thus \lambda provides a unified growth-rate interpretation of smoothness and frequency content. We show that the same principle extends to neural networks: \lambda tracks the emergence of decision-boundary complexity and reflects input-space curvature independent of parameterization. We validate \lambda on analytic benchmarks and neural networks (Two Moons and MNIST), showing that \lambda evolves predictably during training and can be controlled via a derivative-based regularizer. This Curvature Rate Regularization (CRR) produces flatter geometry and improved calibration while maintaining accuracy and performs competitively with Sharpness-Aware Minimization (SAM). Contributions This work: • Introduces \lambda , a scalar curvature measure defined through differentiation dynamics. • Shows that \lambda unifies analytic radius, spectral bandwidth, and neural sharpness under one principle. • Demonstrates that \lambda can be efficiently estimated and used to control functional curvature during training. By grounding curvature in derivative growth, \lambda provides a conceptual link between analysis and deep learning and offers a practical mechanism for characterizing and shaping model behavior. 2. Theoretical Framework 2.1. Differentiation Dynamics We characterize curvature through differentiation dynamics , the behavior of a function under repeated differentiation. Let f:\mathbb{R}^{d}\to\mathbb{R} be smooth, and denote its n -th input derivative by D^{n}f . Define the derivative-growth sequence v_{n}=\|D^{n}f\|_{X}, where \|\cdot\|_{X} is a function-space norm (typically L_{2} or Frobenius). We define the curvature rate \lambda as the exponential growth rate of v_{n} : \lambda_{X}(f)=\limsup_{n\to\infty}\frac{1}{n}\log v_{n}. Positive \lambda indicates exponential growth of derivatives and sharp, high-frequency structure; negative \lambda indicates decay and smoothness. In practice, \lambda is estimated from finite orders by a linear fit: \lambda\approx\text{slope}\big(\log\|D^{n}f\|_{X}\text{ vs. }n\big),\qquad n=1\ldots 4. 2.2. Classical Connections Analytic Functions. For an analytic function f(x)=\sum_{n=0}^{\infty}a_{n}x^{n} , the radius of convergence R satisfies \frac{1}{R}=\limsup_{n\to\infty}|a_{n}|^{1/n}. Since f^{(n)}(0)=n!a_{n} , the derivative norms satisfy \lambda=-\log R. Bandlimited Signals. If f(x)=\int_{-\infty}^{\infty}F(\omega)\,e^{i\omega x}\,d\omega with F(\omega)=0 for |\omega|>\Omega , then D^{n}f(x)=\int(i\omega)^{n}F(\omega)\,e^{i\omega x}\,d\omega, and \lambda=\log\Omega. Examples. A Gaussian e^{-x^{2}/2} yields \lambda=0 (derivatives grow subexponentially). A rational function 1/(1-x) yields \lambda=+\infty (factorial derivative growth). 2.3. Implications Thus \lambda recovers the inverse radius of convergence in complex analysis and the spectral edge in harmonic analysis. Differentiation dynamics provides a single exponential-growth measure that characterizes smoothness in analytic functions, bandwidth in signals, and sharpness in neural networks. 3. Related Work Early empirical studies observed that large-batch training often converges to sharp minima associated with poorer generalization, whereas small-batch training tends to reach flatter minima with better performance [ 7 ] . This motivated optimization methods such as Sharpness-Aware Minimization (SAM), which minimizes worst-case loss in a local neighborhood to encourage flatter solutions and improve robustness [ 4 ] . Theoretical analyses based on PAC-Bayesian generalization bounds posit that flatter minima correspond to larger volumes of low-loss parameters, yielding tighter generalization guarantees [ 9 ] . However, parameter-space flatness is not intrinsic: reparameterizations can make the same function appear arbitrarily flat or sharp, particularly in ReLU networks [ 3 ] . Subsequent work introduced scale-invariant flatness measures to restore correlations with generalization [ 12 ] . A complementary line of work studies generalization through the frequency structure of neural functions. Neural networks exhibit spectral bias, learning low-frequency components before high-frequency ones [ 10 , 2 ] . In the Neural Tangent Kernel regime, convergence dynamics are governed by kernel eigenvalues, further linking generalization to spectral decomposition in function space [ 6 ] . Curvature has also been characterized through the Hessian spectrum. Empirical analyses show that trained networks have highly anisotropic curvature, with a bulk of near-zero eigenvalues and only a few significant directions [ 5 , 11 ] . This structure complicates the use of single scalar sharpness metrics and highlights the difficulty of summarizing curvature in parameter space. Position of This Work Existing approaches predominantly measure curvature in parameter space and are sensitive to parameterization or require expensive second-order computations. In contrast, we introduce \lambda , a curvature measure defined through the growth rate of higher-order input derivatives. \lambda operates directly in function space , characterizing curvature via differentiation dynamics rather than Hessian eigenstructure. This growth-rate viewpoint unifies the inverse radius of convergence for analytic functions, the spectral edge of bandlimited signals, and sharpness in neural networks under a single mathematical principle. 4. Interpretation for Neural Networks The curvature rate \lambda measures the growth of higher-order input derivatives and thus quantifies the functional complexity of the learned mapping. For neural networks, large \lambda corresponds to rapidly varying decision boundaries and high-frequency structure in the input space, while small or negative \lambda indicates smooth, slowly varying behavior. This aligns with the spectral bias of neural networks:

Read the complete text