Principled Curriculum Learning using Parameter Continuation Methods
Harsh Nilesh Pathak, Randy Paffenroth · 2025
Read Principled Curriculum Learning using Parameter Continuation Methods by Harsh Nilesh Pathak, Randy Paffenroth (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22089.txt.
Introduction
Harsh Pathak and Randy Paffenroth propose a stepwise training path for neural nets based on parameter continuation. The idea grows from homotopy methods, which solve hard problems by way of easy ones. It also mirrors school style learning, where tasks rise in steps.
Their method is backed by theory and tested on both guided and unguided tasks. It beat a famed standard trainer on how well nets did on fresh data. The gain held across task kinds in their tests. The work gives curriculum learning a firmer math base than ad hoc step lists.
Catalog record
| CREATOR | Harsh Nilesh Pathak, Randy Paffenroth |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22089
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Principled Curriculum Learning using Parameter Continuation Methods Harsh Nilesh Pathak, Randy Paffenroth Abstract In this work, we propose a parameter continuation method for the optimization of neural networks. There is a close connection between parameter continuation, homotopies, and curriculum learning. The methods we propose here are theoretically justified and practically effective for several problems in deep neural networks. In particular, we demonstrate better generalization performance than state-of-the-art optimization techniques such as ADAM for supervised and unsupervised learning tasks. † † affiliationnotice: Equal contribution 1 Introduction Deep learning applications have seen remarkable progress in recent years ( LeCun et al. 2015 ; Goodfellow et al. 2016 ; Pathak et al. 2018 ) . However, the performance of neural networks is highly dependent on hyper-parameter choices such as loss function, network architecture design, activation function, training strategy, optimizer, initialization, and many other considerations. Unfortunately, many of these choices can lead to highly non-convex optimization problems that then need to be solved for the training of the deep neural network. Another domain in which highly non-convex problems arise is dynamical systems. In fact, the word “chaotic” ( Kathleen et al. 1997 ) has become synonymous with the properties of some such systems. Accordingly, herein we draw inspiration from the study of dynamical systems and propose to analyze deep neural networks from the perspective of homotopy methods and parameter continuation algorithms. For our proposed training methods, we transform standard deep neural networks using homotopies ( Pathak & Paffenroth 2025 ; Nilesh Pathak & Paffenroth 2019 ; Pathak 2024 ; Pathak 2018 ) . Such homotopies allow one to decompose the complex optimization problem into a sequence of simpler problems, each of which is provided with a good initial guess based upon the solution of the previous problem. Accordingly, in this work, we show how one can analyze the evolution of extrema based on the numerical continuation of some homotopy parameter for neural networks. These concepts are not new ( Allgower & Georg 2003 ) and have been used in other fields of mathematics such as discrete and continuous dynamical systems. However, these techniques have not been widely used for analyzing deep neural networks even though they provide many advantages. 1.1 Standard Training for Neural Networks \theta^{*}=\underset{\theta}{argmin}\hskip 5.69046pt\frac{1}{N}\sum_{i=1}^{N}L(y_{i},\hat{y_{i}}) (1) Given a task, dataset and a network architecture the standard techniques for training the neural network is to apply optimization techniques to a problem similar to the one given by equation ( 1 ), where \hat{y}=f(x;\theta) is the output of the neural network. Classically, a variant of a minibatch gradient descent optimizer, perhaps with momentum term ( Duchi et al. 2011 ; Hinton et al. 2012 ; Kingma & Ba 2014 ) , is iteratively applied to find the optimal network parameters. Unfortunately, a deep neural network’s cost surface usually consists of many critical points ( Goodfellow et al. 2016 ) such as local minima, saddle points and degenerate minima and saddle region. Thus, getting to the quality minimum with very low generalization performance is an active area of research. 1.2 Continuation Methods for Neural Networks Parameter continuation methods ( Allgower & Georg 2003 ; Soviany et al. 2022 ; Pathak 2018 ) take a different approach than the standard training. As introduced, continuation methods utilize homotopies to decompose the original problem to a continuum of tasks to work with a family of minima and thus, starts by finding a minimum (or critical) point for the simpler optimization problem. Then, the optimization problem is gradually changed from the easy problem to the challenging problem of interest . The critical point is adjusted as the optimization problem is changed, leading to finding a critical point of the challenging problem. In particular, given a challenging minimization problem \underset{\theta\in\mathbb{R}^{m}}{min}L(\theta) , one can embed this problem into a larger class of problems using a homotopy such as : \underset{\theta\in\mathbb{R}^{m},\lambda\in[0,1]}{min}\lambda L(\theta)+(1-\lambda)M(\theta) (2) where M(\theta) is some problem where a good initialization \theta^{0} is known, which is in the basin of attraction of some critical point. Given \theta^{0} and setting \lambda=0 the above optimization problem can be solved using any first or second order gradient methods to converge at the critical point. The parameter \lambda can then be “continued in” by increasing in small steps until \lambda=1 is reached and a critical point is found of the problem of interest L . Of course, many questions present themselves. Under what circumstances can we guarantee that we will eventually find a solution where \lambda=1 ? What is an appropriate “small” step size? What if L has many critical points and M has only one? Such questions are precisely those that arise and are addressed by continuation method theory. Such homotopy embedding and continuation methods have long served as useful tools in modern mathematics ( Allgower & Georg 2003 ; Klein 1883 ; Bendixson 1901 ; Leray & Schauder 1934 ) . The use of deformations to solve nonlinear systems of equations may be traced back at least to ( Lahaye 1934 ) . 2 Curriculum and Continuation Methods We discussed two approaches to solving a non-convex optimization problem; First, a direct method where data is fed randomly and initialization of parameters is also random. Second, a continuation-based approach where we start with a simpler (possibly convex) problem which is gradually transformed to the highly non-convex problem. In this section, we want to shed some light on another popular approach which is originally inspired by continuation methods i.e. curriculum learning ( Bengio et al. 2009 ) . In general, curriculum learning suggests feeding data in a meaningful order; similar to humans who learn the tasks with increasing difficulty. Many researchers observed better generalization performance after introducing curriculum strategies to existing SOTA neural architectures ( Soviany et al. 2021 ; Hacohen & Weinshall 2019 ; Karras et al. 2017 ; Weinshall & Cohen 2018 ; Wang et al. 2019 ) . The authors ( Soviany et al. 2021 ; Pathak 2018 ) broadly classify curriculum strategies as, (1) by using meaningful order of samples (data curriculum) , and (2) by altering some carefully chosen model configuration (model curriculum) . For a detailed study on recent curriculum strategies we recommend this paper ( Soviany et al. 2021 ) . Despite the better performance of curriculum learning, it has not been widely accepted by the Deep Learning community ( Soviany et al. 2021 ) . Even in NLP, Active learning is more popular ( Chandrasekaran et al. 2020 ) . One of the possible reasons that the curriculum needs to integrate well the in-hand optimization task is that the difficulty of devising such strategies may be domain-dependent and may also require careful human intervention. However, instead of one’s intuition, we study curriculum strategies through the lens of Implicit Function Theorem (IFT) ( Allgower & Georg 2003 ) . In this paper, we attempt to close the gap between curriculum learning and continuation methods. In particular, if we define a single parameter \lambda to employ data or model curriculum, then we discuss: Question: What is the best parametrization ( \lambda ) to find a family of minima for complex problems like Neural Networks? In the recent literature, researchers have chosen several directions for applying curriculum learning in neural