Multiscale Training of Convolutional Neural Networks
Shadab Ahamed [email protected], Niloufar Zakariaei [email protected], Eldad Haber [email protected], Moshe Eliasof [email protected] · 2025
Read Multiscale Training of Convolutional Neural Networks by Shadab Ahamed [email protected], Niloufar Zakariaei [email protected], Eldad Haber [email protected], Moshe Eliasof [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12739.txt.
Introduction
Shadab Ahamed, Niloufar Zakariaei, Eldad Haber, and Moshe Eliasof present 2025 methods to train image networks faster by working across scales. Training on high-resolution images is held back by the cost of gradient maths on the finest grid. Most steps are spent at full size, even though coarse versions of the task carry much of the same signal.
One method writes the fine-grid gradient as a telescoping sum of gradients at ever coarser grids. Big batches go to the cheap coarse levels, which keeps the noise of the estimate the same while cutting fine work fourfold per scale step. A full method then solves the task at coarse scale first and hot-starts the next finer level. Tests on denoising, deblurring, and upscaling tasks cut compute by four to sixteen times with no real loss.
Catalog record
| CREATOR | Shadab Ahamed [email protected], Niloufar Zakariaei [email protected], Eldad Haber [email protected], Moshe Eliasof [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-12739
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multiscale Training of Convolutional Neural Networks Shadab Ahamed [email protected], Niloufar Zakariaei [email protected], Eldad Haber [email protected], Moshe Eliasof [email protected] Abstract Training convolutional neural networks (CNNs) on high - resolution images is often bottlenecked by the cost of evaluating gradients of the loss on the finest spatial mesh. To address this, we propose Multiscale Gradient Estimation (MGE) , a Multilevel Monte Carlo - inspired estimator that expresses the expected gradient on the finest mesh as a telescopic sum of gradients computed on progressively coarser meshes. By assigning larger batches to the cheaper coarse levels, MGE achieves the same variance as single - scale stochastic gradient estimation while reducing the number of fine mesh convolutions by a factor of 4 with each downsampling. We further embed MGE within a Full - Multiscale training algorithm that solves the learning problem on coarse meshes first and “hot - starts” the next finer level, cutting the required fine mesh iterations by an additional order of magnitude. Extensive experiments on image denoising, deblurring, inpainting and super - resolution tasks using UNet, ResNet and ESPCN backbones confirm the practical benefits: Full-Multiscale reduces the computation costs by 4-16 × \times with no significant loss in performance. Together, MGE and Full - Multiscale offer a principled, architecture - agnostic route to accelerate CNN training on high - resolution data without sacrificing accuracy, and they can be combined with other variance - reduction or learning - rate schedules to further enhance scalability. 1 1 1 Code available at: https://github.com/ahxmeds/multiscale-gradient-estimation Reviewed on OpenReview: https://openreview.net/forum?id=HTQuEZwEHw 1 Introduction In this work, we consider the task of learning a functional y({\bf x})=\phi(u({\bf x})) , where {\bf x} is a position (in 2D {\bf x}=({\bf x}_{1},{\bf x}_{2}) , u({\bf x})\in{\cal U} and y({\bf x})\in{\cal Y} are families of functions. To this end, let us assume that we have discrete samples from {\cal U} and {\cal Y} , that is {\bf u}_{i}^{h}=u_{i}({\bf x}_{h}),{\bf y}^{h}_{i}=\phi(u_{i}({\bf x}_{h})) for i=1,\ldots,M , associated with some resolution h . A common approach to learning the function is to parameterize the problem, typically by a deep network, and replace \phi with a function f(\cdot,\cdot) that accepts the vector {\bf u}^{h} and learnable parameters {\bm{\theta}} which leads to the problem of estimating {\bm{\theta}} such that {\bf y}^{h}_{i}\approx f({\bf u}^{h}_{i},{\bm{\theta}}) for i=1,\ldots,M . To evaluate {\bm{\theta}} , the following stochastic optimization problem is formed and solved, \min_{{\bm{\theta}}}\ {\mathbb{E}}_{{\bf u}^{h},{\bf y}^{h}}{\ell}\left(f({\bf u}^{h},{\bm{\theta}}),{\bf y}^{h}\right), (1) where \ell(\cdot,\cdot) is a loss function (typically mean squared error). Standard approaches use variants of stochastic gradient descent (SGD) to estimate the loss and its gradient for different samples of ({\bf u}^{h},{\bf y}^{h}) . In deep learning with convolutional neural networks, the parameter {\bm{\theta}} (the convolutional weights) has identical dimensions, independent of the resolution. Although variants of SGD (e.g. Adam, AdamW, etc.) are widely used, their computational cost can become prohibitively high as the mesh-size h decreases, especially when evaluating the function f on a fine mesh for many samples {\bf u}_{i}^{h} . This challenge is worsened if the initial guess {\bm{\theta}} is far from optimal, requiring many costly iterations for large data sizes M . One way to avoid large meshes is to use small crops of the data where large images are avoided, however, this can degrade performance, especially when a large receptive field is required for learning ( Araujo et al., 2019 ) . Background and Related Work. Computational cost reduction can be achieved by leveraging different resolutions, a fundamental concept to multigrid and multiscale methods. These methods have a long history of solving partial differential equations and optimization problems ( Trottenberg et al., 2001 ; Briggs et al., 2000 ; Nash, 2000 ) . Techniques like multigrid ( Trottenberg et al., 2001 ) and algorithms such as MGopt ( Nash, 2000 ; Borzi, 2005 ) and Multilevel Monte Carlo ( Giles, 2015 ; Giles, 2008 ; Van Barel and Vandewalle, 2019 ) are widely used for optimization and differential equations. In deep learning, multiscale or pyramidal approaches have been used for image processing tasks such as object detection, segmentation, and recognition, where analyzing multiple resolutions is key ( Scott and Mjolsness, 2019 ; Chada et al., 2022 ; Elizar et al., 2022 ) . Recent methods improve standard CNNs for multiscale computations by introducing specialized architectures and training methods. For instance, the work by He and Xu (2019) uses multigrid methods in CNNs to boost efficiency and capture multiscale features, while Eliasof et al. (2023b) focuses on multiscale channel space learning, and van Betteray et al. (2023) unifies both. Li et al. (2020) introduced the Fourier Neural Operator, enabling mesh-independent computations, and Wavelet neural networks were explored to capture multiscale features via wavelets ( Fujieda et al., 2018 ; Finder et al., 2022 ; Eliasof et al., 2023a ) . While often overlooked, it is important to note that these approaches, can be divided into two families of approaches that leverage multiscale concepts. The first is to learn parameters for each scale, and a separate set of parameters that mix scales, as in UNet ( Ronneberger et al., 2015 ) . The second , called multiscale training , enables the approximation of fine-scale parameters using coarse-scale samples ( Haber et al., 2017a ; Wang et al., 2020 ; Ding et al., 2020 ; Ganapathy and Liew, 2008 ) . The second approach aims to gain computational efficiency, as it approximates fine mesh parameters using coarse meshes, and it can be coupled with the first approach, and in particular with UNets. Our approach. This work advances the multiscale training paradigm by rigorously adapting Multilevel Monte Carlo (MLMC) methods to the non-convex landscape of CNN training. While concepts like MLMC and multigrid are well-established in numerical analysis, their application to stochastic gradient estimation for deep CNNs is non-trivial and requires specific theoretical justification. We distinguish our contributions beyond a simple recombination of existing methods in three key ways. First, we establish the theoretical bounds for CNN gradient estimation. We explicitly derive the error bounds for the Multiscale Gradient Estimation (MGE) estimator, proving that under standard Lipschitz conditions, the difference between gradients on fine and coarse meshes decays as \mathcal{O}(h) . This provides the necessary theoretical guarantee for applying variance reduction to convolutional weights, ensuring that the optimization does not diverge when mixing scales, a specificity not addressed in standard MLMC literature. Second, we provide a rigorous analysis of subsampling strategies. Unlike prior empirical works ( Haber et al., 2017b ) , we mathematically demonstrate that the choice of downsampling is critical: we prove that coarsening-based subsampling yields an error that vanishes as resolution increases ( \mathcal{O}(2^{L}h) ), whereas cropping-based strategies result in a constant error bound ( \mathcal{O}(1) ) regardless of resolution. This offers a principled guideline for multiscale training paradigm design. Finally, we embed these theoretical insights into a Full-Multiscale training algorithm. By combining the variance reduction of MGE with a coarse-to-fine “hot-start” initialization, we demonstrate an architecture-agnostic framework that accelerates training by orders of magnitude for tasks ranging from denoising to super-resolution, without significant loss in performance over key metrics. Although our method exploits the convolutional operation, it does not explicitly address alternative mechanisms like attention. Consequently, while the framework could, in principle, be adapted to attention, it