A lift for input-convex neural network training
Ali Siahkoohi, Anirudh Thatipelli · 2026
Read A lift for input-convex neural network training by Ali Siahkoohi, Anirudh Thatipelli (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24274.txt.
Introduction
Siahkoohi studies input-convex neural net training. Convexity demands non-negative inter-layer weights. Projected gradient descent re-crosses the boundary endlessly under mini-batch noise, so the projection never settles. Direct softplus tunes a free latent weight through a positivity map whose gradient fades for negative weights. The proposed lift swaps the latent weight for a slack plus a batch-dependent body, coupling updates to batch gradients. Where softplus stalls at the shoulder, the lift fits targets better and reaches the same reconstruction roughly three times sooner.
Catalog record
| CREATOR | Ali Siahkoohi, Anirudh Thatipelli |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 115271 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.24274
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A lift for input-convex neural network training Ali Siahkoohi, Anirudh Thatipelli Abstract Input-convex neural networks (ICNNs) are widely used for a range of learning tasks—log-concave density estimation, convex-potential normalizing flows, optimal transport, and transport-map inversion for high-dimensional Bayesian posteriors. All of these tasks share a structural constraint: the inter-layer weights of the ICNN must remain non-negative. The standard recipe for enforcing it, projected gradient descent (PGD) onto the non-negative cone, applies a hard, non-smooth projection—the stiff-penalty limit of an ADMM-style constraint splitting—and its classical convergence guarantees do not transfer to the non-smooth ICNN training landscape; the differentiable alternative, softplus reparametrization, instead attenuates the gradient exponentially in the weight magnitude, stalling training with dead inter-layer weights and plateaued loss. To address this limitation, and inspired by the parameter-extension lifts of PDE-constrained inverse problems, we propose the lift : instead of constraining the inter-layer weights directly, we train an unconstrained hypernetwork that emits them from a permutation-invariant summary of the input batch. This adds a source of stochasticity to the training dynamics that softens the loss landscape, letting the iterates escape the gradient-attenuated region where direct softplus stalls. We trace this softening to three structural ingredients—a learnable bias acting as slack , a hypernetwork body that conditions on the target batch, and a cross-covariance coupling the two through batch stochasticity—and prove each one necessary: deleting any single ingredient collapses the cross-covariance that carries the softening. By means of log-concave energy-based modeling at scales from one-dimensional toy targets to image-flavored latents, and convex-potential normalizing flows on a 21-dimensional tabular benchmark, we show that the lift reaches a lower test loss than both PGD and direct softplus, and turns a plateau-bounded training trajectory into a valley-descending one . (a) Parameter-space view. (b) Loss-space view. Figure 1: Three positivity reparametrizations on log-concave EBM training, three different fates (21-dimensional tabular target; test negative log-likelihood reported at each method’s lowest-validation-loss checkpoint). (a) Loss landscape on a two-dimensional PGD-anchored slice ( Section 5.3 ), the converged hypernet at the origin (gold star); (b) held-out validation loss versus iteration on the same run. Hypernet descends through the basin to the deepest loss; PGD lands at the cone boundary; direct softplus is trapped on the readout shoulder ( Section 2.2 ), an order of magnitude above the other two. The lift’s margin above PGD is the landscape-smoothing benefit of Lemma 1 ; the gap above direct is the Kramers escape of Corollary 1 . Code to partially reproduce the results is available on GitHub . 1 Introduction Input-convex neural networks (ICNNs) ( 4 ) parametrize the convex scalar fields that drive several modern learning tasks in probabilistic modeling and Bayesian inference. They underlie the negative log-density of a log-concave ( 40 ; 41 ) energy-based model, the convex potential whose gradient defines a convex-potential normalizing flow ( 21 ) , the convex potential of a PCP-Map ( 51 ; 12 ) for transport-map posterior sampling, and the Brenier potential of ICNN-parametrized optimal transport ( 32 ; 27 ) . These transport-map constructions sit within a broader move toward generative posterior sampling for high-dimensional Bayesian inverse problems, a setting in which score-based and diffusion-model samplers have also been applied to seismic imaging ( 5 ; 42 ) . Across the ecosystem the ambient dimension ranges from a single coordinate on toy targets to thousands of pixels on image-density and PCP-Map applications, and all four applications share one structural feature: input-convexity demands positivity of the inter-layer weights, i.e., {\bm{\theta}}\succeq\bm{0} . The dominant practical recipe for enforcing this constraint is projected gradient descent (PGD) ( 4 ) : an unconstrained step on {\bm{\theta}} followed by the projection {\bm{\theta}}\leftarrow\max({\bm{\theta}},0) . The projection is non-differentiable on the active set, where the iterates concentrate, and PGD’s classical convergence guarantees ( O(1/k) for smooth convex problems with a closed convex constraint and O(1/\sqrt{k}) to stationarity for smooth non-convex problems ( 36 ; 8 ) ) assume a Lipschitz-smooth objective that the ICNN training landscape violates there, so they do not transfer. A differentiable alternative reparametrizes the inter-layer weights as {\bm{\theta}}=\psi(\tilde{\bm{\theta}}) with \tilde{\bm{\theta}}\in\mathbb{R}^{d} unconstrained and \psi a monotone non-negative map, of which there are two genuine families—the smooth one, softplus, and the non-smooth max-based one, the ReLU-type readouts \psi(\tilde{\theta})=\max(\tilde{\theta},\epsilon) that project onto the non-negative cone (the cone projection being the \epsilon=0 case). Each enforces {\bm{\theta}}\succeq\bm{0} by construction, and each introduces a chain-rule prefactor \psi^{\prime}(\tilde{\bm{\theta}}) that multiplies every downstream gradient and collapses on an extended region of parameter space: the softplus prefactor \psi^{\prime}(\tilde{\theta}) vanishes smoothly as \tilde{\theta}\to-\infty ( 20 ) , while the ReLU-type prefactor is identically zero on the gated set and matches PGD at the cost of differentiability. Stochastic gradient descent escapes this attenuated region only on a time scale that grows exponentially with the inverse noise level ( Corollary 1 )—the Kramers–Arrhenius ( 28 ; 16 ; 54 ) regime, fundamentally slower than the polynomial classical PGD rates. Existing remediations treat the symptom rather than the structure: specialized initialization schedules ( 20 ) each tame an individual failure mode; plain alternating direction method of multipliers (ADMM) with positivity ( 10 ) enforces the constraint by a closed-form prox on the slack but leaves the primal block without data-conditioned reparametrization, and its stiff-penalty limit \rho\to\infty reduces structurally to PGD on the cone. Inspired by the parameter-extension lifts of full-waveform inversion (FWI) ( 44 ; 47 ) —where a stiff variable is recast in a larger space to turn a hard non-convex problem into a smoother one—we propose the lift . Rather than constraining the inter-layer weights directly, we train an unconstrained hypernetwork ( 15 ) that emits them from the input batch. Because the batch is resampled at every step, the emitted weights fluctuate as training proceeds—an additional source of stochasticity in the training dynamics. Unlike ordinary mini-batch gradient noise, this fluctuation is not suppressed by the gradient attenuation that stalls direct softplus, and it softens the loss landscape around the region where training would otherwise plateau. We trace the effect to three structural ingredients of the construction—a learnable slack, the batch-conditioned body, and the stochastic coupling between them—and prove that none is dispensable ( Theorem 1 ). On a one-dimensional log-concave energy-based model (EBM) and a convex-potential normalizing flow on a 21-dimensional tabular benchmark, the lift descends to a lower test loss than both PGD and direct softplus, turning a plateau-bounded trajectory into a valley-descending one ( Figure 1 ). 1.1 Contributions (1) The lift. We propose the hypernetwork-emitted ICNN parametrization of equation ( 1 ), which introduces an additional source of stochasticity into the training dynamics that effectively smooths the loss landscape around the readout shoulder. We then read the parametrization as a split-variable decomposition ( Section 3 , Section 4.2 ), an interpretation that delivers the lift’s conditioning advantage without invoking the rate-theoretic assumptions that an ADMM convergence analysis would require. (2) Three jointly necessary structural ingredients. We identify three ingredients of the lift’s conditioning advantage—an identity-Jacobian slack , a data-conditioned body , and a non-vanishing cross-covariance between them—and prove that each is necessary for an operationally measurable cross-covariance estimator ( Theorem 1 ). An implicit strong-convexification result links the cross-covariance to an added curvature modulus on the loss landscape, scoped to the stochastic-readout