Convergence of Riemannian Stochastic Gradient Descents: Varying Batch Sizes And Nonstandard Batch Forming

Hao Wu · 2026

Read Convergence of Riemannian Stochastic Gradient Descents: Varying Batch Sizes And Nonstandard Batch Forming by Hao Wu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-06350.txt.

Introduction

Hao Wu proves convergence theorems for Riemannian stochastic gradient descent, a search method on curved spaces, in cases where the basic chance space changes from one step to the next. Such shifting chance spaces arise in practical variants of the method. Applied, he derives convergence results for the method with varying batch sizes and for unbiased batch forming schemes, showing both variants converge under the new theorems.

Catalog record

CREATORHao Wu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70247
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.06350

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Convergence of Riemannian Stochastic Gradient Descents: Varying Batch Sizes And Nonstandard Batch Forming Thanks: The author would like to thank Conglong Xu and Peiqi Yang for interesting conversations. Hao Wu Abstract . We establish convergence theorems for Riemannian stochastic gradient descents in which the underlying probability spaces vary from iteration to iteration. As applications, we deduce convergence results for Riemannian stochastic gradient descents with varying batch sizes and unbiased batch forming schemes. 2010 Mathematics Subject Classification Primary 41A60, 53Z50, 62L20, 68T05 1. Background 1.1. Bonnabel’s theorem In [ 6 ] , Bonnabel established an almost sure convergence theorem for Riemannian stochastic gradient descents. Before stating this theorem, let us first recall the definition of retractions on manifolds. Definition 1.1 . [ 1 , Definition 4.1.1] Let \mathcal{M} be a differentiable manifold. A retraction on \mathcal{M} is a C^{2} map R:T\mathcal{M}\rightarrow\mathcal{M} such that, for every x\in\mathcal{M} , the restriction R_{x}=R|_{T_{x}\mathcal{M}} satisfies • R_{x}(\mathbf{0}_{x})=x , where \mathbf{0}_{x} is the zero vector in T_{x}\mathcal{M} , • dR_{x}(\mathbf{0}_{x})=\mathrm{id}_{T_{x}\mathcal{M}} under the standard identification T_{\mathbf{0}_{x}}T_{x}\ M\cong T_{x}\mathcal{M} , where dR_{x} is the differential of R_{x} and \mathrm{id}_{T_{x}\mathcal{M}} is the identity map of T_{x}\mathcal{M} . Now we can state Bonnabel’s theorem. Theorem 1.2 . [ 6 , Theorem 2] Assume that (1) \mathcal{M} is a connected Riemannian manifold, 1 1 1 [ 6 , Theorem 2] assumes furthermore that the injective radius of \mathcal{M} is uniformly bounded below by a positive number. This is unnecessary since [ 6 , Theorem 2] also assumes that all iterates are contained in a compact subset of \mathcal{M} . (2) R:T\mathcal{M}\rightarrow\mathcal{M} a retraction (3) F:\mathcal{M}\rightarrow\mathbb{R} is a C^{3} cost function, (4) (\Omega,\mathcal{F},\mu) is a probability space, where \Omega is the sample space, \mathcal{F} is the event space and \mu is the probability measure, (5) H:\mathcal{M}\times\Omega\rightarrow T\mathcal{M} is a function satisfying that (a) for every \omega\in\Omega , H(\bullet,\omega):\mathcal{M}\rightarrow T\mathcal{M} is a tangent vector field on \mathcal{M} , (b) for every x\in\mathcal{M} , \mathbb{E}(H(x,\omega)):=\int_{\Omega}H(x,\omega)d\mu=\nabla F(x) , (6) \{\omega_{t}\}_{t=0}^{\infty} is a sequence of independent random variables taking values in \Omega with identical probability distribution \mu , (7) \{\gamma_{t}\}_{t=0}^{\infty} is a sequence of positive learning rates satisfying (1.1) \sum_{t=0}^{\infty}\gamma_{t}^{2}<\infty\text{ and }\sum_{t=0}^{\infty}\gamma_{t}=\infty, (8) x_{0}\in\mathcal{M} and \{x_{t}\}_{t=0}^{\infty}\subset\mathcal{M} is the sequence of iterates in \mathcal{M} generated by (1.2) x_{t+1}=R_{x_{t}}(-\gamma_{t}H(x_{t},\omega_{t}))\text{ for }t\geq 0, (9) there is a compact set K\subset\mathcal{M} such that (a) \{x_{t}\}_{t=0}^{\infty}\subset K , (b) there is an A>0 such that \|H(x,\omega)\|_{x}\leq A for all x\in K and \omega\in\Omega , where \|\bullet\|_{x} is the Riemannian norm of \mathcal{M} on T_{x}\mathcal{M} . Then F(x_{t}) converges almost surely and \|\nabla F(x_{t})\|_{x_{t}}\rightarrow 0 almost surely. Since K is compact, the sequence \{x_{t}\} in Theorem 1.2 has a convergent subsequence, which almost surely converges to a stationary point of F . 1.2. Batch sizes in stochastic gradient descents The scope of Theorem 1.2 is beyond basic Riemannian stochastic gradient descents. For example, for a mini-batch Riemannian stochastic gradient descent 2 2 2 Since all the stochastic gradient descents in the rest of this manuscript are mini-batch. We drop the word “mini-batch” for simplicity from now on. with fixed batch size b>0 , that is, when updating rule ( 1.2 ) is replaced by (1.3) x_{t+1}=R_{x_{t}}\left(-\frac{\gamma_{t}}{b}\sum_{j=tb}^{(t+1)b-1}H(x_{t},\omega_{j})\right)\text{ for }t\geq 0, one can get convergence results by applying Theorem 1.2 to the averaged random gradient (1.4) \overline{H}^{[b]}:\mathcal{M}\times\Omega^{b}\rightarrow T\mathcal{M}\text{ given by }\overline{H}^{[b]}(x,\omega_{1},\dots,\omega_{b})=\frac{1}{b}\sum_{j=1}^{b}H(x,\omega_{j}), where the probability measure on \Omega^{b} is of course \mu^{\times b} . More recently, there were a good amount of results about improving the efficiency of stochastic gradient descents by allowing the batch sizes to vary. See for example [ 2 , 4 , 8 , 9 , 10 , 11 , 13 , 17 , 19 , 21 , 22 , 23 ] for studies in this direction. These often involve adaptive batch sizes that increase as the cost functions decrease. Theorem 1.2 does not directly apply to stochastic gradient descents with varying batch sizes. Many authors in this direction performed convergence analysis for their choices of adaptive batch sizes. Notably, Bottou, Curtis and Nocedal established a general mean square convergence theorem [ 8 , Corollary 4.12] for stochastic gradient descents in Euclidean spaces with arbitrarily varying batch sizes. The main benefit of using larger or increasing batch sizes is the reduction of the variance of the updating vectors, which makes stochastic gradient descents converge faster. See for example [ 20 ] . Several more sophisticated batch forming schemes were introduced to further reduce the variance of the updating vectors. Examples of such schemes can be found in [ 14 , 15 , 18 , 28 , 29 ] . 1.3. Contributions In this manuscript, we mildly generalize the bounded variance convergence argument to prove convergence theorems in the spirit of Theorem 1.2 for Riemannian stochastic gradient descents with varying batch sizes or unbiased non-standard batch forming schemes. More precisely, we work on the following topics. (1) For the bounded variance convergence argument to work, the random inputs \{\omega_{t}\}_{t=0}^{\infty} need to be independent of each other. However, we observe that there is really no need for them to have the same distribution, or to even take values in the same probability space. Based on this observation, we generalize Theorem 1.2 to Theorem 2.4 below, which allows \omega_{t} to take values in different probability spaces as t varies. Our proof of Theorem 2.4 is mostly the standard bounded variance convergence argument with some cosmetic modifications. (2) Li and Orabona proved an almost sure convergence theorem [ 12 , Theorem 2] for stochastic gradient descents in Euclidean spaces with a specific adaptive learning rate formula. This was generalized to Riemannian stochastic gradient descents with fixed batch sizes in [ 25 , 27 ] , where it is also observed that Li and Orabona’s adaptive learning rates outperform deterministic learning rates satisfying the standard condition ( 1.1 ) in some Riemannian stochastic gradient descents. In the current manuscript, we further generalize [ 12 , Theorem 2] to Theorem 2.5 below, which allows the random input \omega_{t} to take values in different probability spaces as t varies. Our proof for Theorem 2.5 closely follows the arguments by Li and Orabona in [ 12 ] . (3) As long as the batches are formed independently and the expectation of the updating vector from each batch is the gradient of the cost function, one can apply Theorems 2.4 and 2.5 to get reasonably general convergence results for stochastic gradient descents. We present examples of such applications for three different batch forming schemes in Corollaries 2.7 , 2.9 and 2.11 . (4) Moreover, we apply the idea of “confinements” from [ 24 , 27 ] to stochastic gradient descents with varying batch sizes. This idea is rooted in [ 6 , 7 ] . As a result, in some special and yet widely applicable cases, instead of assuming that all iterates are contained in a compact set based on some empirical evidence, we can conclusively prove this compactness assumption without running the algorithm. 2. Convergence Theorems In this section, we state our convergence theorems. The proofs of these theorems are contained in Section 3 below. 2.1. Retraction-dependent Lipschitz tangent vector fields Before stating our convergence theorems, let us introduce a notion of retraction-dependent Lipschitz tangent vector fields. Recall that

Read the complete text