A condensing approach to multiple shooting neural ordinary differential equation
Siddharth Prabhu, Srinivas Rangarajan, Mayuresh Kothare · 2025
Read A condensing approach to multiple shooting neural ordinary differential equation by Siddharth Prabhu, Srinivas Rangarajan, Mayuresh Kothare (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00724.txt.
Introduction
Siddharth Prabhu, Srinivas Rangarajan, and Mayuresh Kothare bring a multiple shooting method to neural ordinary differential equations. Their 2025 paper works on models that learn the rules of a changing system from data. In multiple shooting, a trajectory is cut into short pieces that can be solved in parallel, then stitched by equality constraints.
That stitching is the hard part for neural models, which is why the method saw little use there. The authors use a condensing trick to fold the stitch rules into the training problem. First-order optimizers such as Adam can then train the model directly. The payoff is stability: multiple shooting handles wiggly and long trajectories far better than a single long integration.
Catalog record
| CREATOR | Siddharth Prabhu, Srinivas Rangarajan, Mayuresh Kothare |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00724
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A condensing approach to multiple shooting neural ordinary differential equation Siddharth Prabhu, Srinivas Rangarajan, Mayuresh Kothare Abstract Multiple-shooting is a parameter estimation approach for ordinary differential equations. In this approach, the trajectory is broken into small intervals, each of which can be integrated independently. Equality constraints are then applied to eliminate the shooting gap between the end of the previous trajectory and the start of the next trajectory. Unlike single-shooting, multiple-shooting is more stable, especially for highly oscillatory and long trajectories. In the context of neural ordinary differential equations, multiple-shooting is not widely used due to the challenge of incorporating general equality constraints. In this work, we propose a condensing-based approach to incorporate these shooting equality constraints while training a multiple-shooting neural ordinary differential equation (MS-NODE) using first-order optimization methods such as Adam. Keywords Multiple-Shooting \cdot NeuralODE \cdot Equality Constraint Neural Network \cdot Automatic Differentiation 1 Introduction Neural ordinary differential equation (NODE) ( Chen et al. 2018 ; Rackauckas et al. 2020 ) approximates the dynamics of the system as a neural network, which is integrated to get the trajectory of the states. The parameters of NODE are obtained by solving an optimization problem using a single-shooting approach ( Vassiliadis et al. 1994a ; Vassiliadis et al. 1994b ) . Single-shooting approach, however, is difficult to converge and unstable, especially for long and highly complex oscillatory dynamics. Multiple-shooting addresses these issues by dividing the trajectory into small intervals, which can be handled in parallel, while adding continuity equations as equality constraints to the optimization problem ( Bock and Plitt 1984 ; Diehl et al. 2006 ) . A pictorial representation of the different shooting approaches is shown in Figure 1 . While these additional equality constraints can easily be incorporated in a typical Newton-based nonlinear optimization solver, incorporating them while training a neural network using first-order optimization techniques is not straightforward. (a) (b) Figure 1: A pictorial representation of single-shooting (a) and multiple-shooting (b) in the context of parameter estimation of ordinary differential equations. The green points are measurements, while the solid black line shows the models prediction. Previous work by Massaroli et al. 2021 incorporates multiple-shooting as differentiable layers in a neural network. However, their main focus is on solving forward simulations in parallel using a modification of the parareal method ( Maday and Turinici 2002 ) , while the training of the neural network parameters is still performed using a single-shooting approach. Turan and Jäschke 2022 solves a constraint multiple shooting problem by reformulating it as an unconstrained augmented Lagrangian problem and applying either first-order or quasi-second-order optimization techniques. Bradley and Boukouvala 2021 introduces a multiple-shooting-like approach in which each shooting interval begins with a measurement used as the initial condition, thereby removing the need for explicit equality constraints. While they were able to successfully train such a NODE model on the training data, they were unable to extrapolate the model to unseen test data. Additionally, when the measurements are noisy, such an approach may break down because the endpoint of the previous shooting interval may not align with the noisy measurement, which serves as the initial condition of the next shooting interval. Some other works ( Chen et al. 2024 ; Rangarajan et al. 2022 ; Beucler et al. 2021 ) , though not in the context of multiple shooting, have incorporated equality constraints during neural network training, a formulation to which the multiple-shooting problem can be reduced. However, these approaches are limited to either linear or linearly separable equality constraints, both of which are unlikely to arise in typical multiple-shooting scenarios. In this paper, we take a condensing-based ( Bock and Plitt 1984 ; Albersmeyer and Diehl 2010 ) approach for training NODE subject to continuity constraints (shooting gap equals zero) arising from a multiple-shooting formulation. This procedure ensures that the updates to the continuity variables ( \Delta x ) and the parameters ( \Delta p ) satisfy the first-order Taylor expansion of the equality constraints at each iteration. We demonstrate the applicability of this algorithm on several complex oscillatory dynamical systems and compare its performance with that of a model trained using naive single-shooting. 2 Method In MS-NODE, we want to solve the following optimization problem \displaystyle\begin{split}&\min_{x_{1},x_{2},\cdots x_{m},p}\ \sum_{k=1}^{m}\phi(x_{k},p)\\ \text{subject to}&\\ &\quad x_{1}=\hat{x}_{1}\\ &\quad x_{k+1}=x_{k}+\int_{t_{K}}^{t_{k+1}}f(x_{k},p)\quad\text{for}\quad k=1,2,\cdots,m-1\end{split} (1) where x_{k}\in\mathbb{R}^{n} are the states, \hat{x}_{0} is the measured initial condition, \phi is the loss function and the equality constraints are the boundary constraints for each multiple shooting block. Let F_{k}(x_{k},p)=x_{k}+\int_{t_{k}}^{t_{k+1}}f(x_{k},p),\quad G=\begin{bmatrix}x_{1}-\hat{x}_{1}\\ x_{2}-F_{1}(x_{1},p)\\ \vdots\\ x_{m}-F_{m-1}(x_{m-1},p)\end{bmatrix}_{mn\times 1},\quad\text{and}\quad\Phi=\sum_{i=1}^{m}\phi(x_{i},p) (2) where G is a matrix with m constraints and each constraint has dimension n , F_{k} can be obtained using appropriate time-stepping integration schemes. For convenience, we will drop the brackets in front of F_{k} . The Lagrangian of the optimization problem in equation 3 is written as L(x,p,\lambda)=\Phi+\lambda^{T}G (3) and the KKT conditions are as follows \displaystyle\begin{split}&L_{x}=\Phi_{x}+\lambda^{T}G_{x}=0\\ &L_{p}=\Phi_{p}+\lambda^{T}G_{p}=0\\ &L_{\lambda}=G=0\end{split} (4) where L_{x}\in\mathbb{R}^{mn\times 1},L_{p}\in\mathbb{R}^{p\times 1},L_{\lambda}\in\mathbb{R}^{mn\times 1} is the partial derivative of L with respect to x , p , and \lambda respectively. Similarly G_{x}\in\mathbb{R}^{mn\times mn},G_{p}\in\mathbb{R}^{mn\times p} is the partial derivative of G with respect to x , and p respectively. The Newton step for the system of equations in 4 is as follows \begin{bmatrix}L_{xx}&L_{xp}&G_{x}^{T}\\ L_{px}&L_{pp}&G_{p}^{T}\\ G_{x}&G_{p}&0\\ \end{bmatrix}\begin{bmatrix}\Delta x\\ \Delta p\\ \Delta\lambda\\ \end{bmatrix}=-\begin{bmatrix}L_{x}\\ L_{p}\\ G\\ \end{bmatrix} (5) Note that while training a neural network second-order optimization methods such as the Newton method are not used. In case of first-order optimization methods such as Gradient descent, which is often used for training a neural network, equation 5 can be rewritten as \begin{bmatrix}I_{xx}&0&G_{x}^{T}\\ 0&I_{pp}&G_{p}^{T}\\ G_{x}&G_{p}&0\\ \end{bmatrix}\begin{bmatrix}\Delta x\\ \Delta p\\ \Delta\lambda\\ \end{bmatrix}=-\begin{bmatrix}L_{x}\\ L_{p}\\ G\\ \end{bmatrix} (6) where I_{xx}\in\mathbb{R}^{mn\times mn} , I_{pp}\in\mathbb{R}^{p\times p} are identity matrix and 0 is a zero matrix of appropriate dimensions. Solving the system of equations gives \displaystyle\begin{split}&I_{xx}\Delta x+G_{x}^{T}\Delta\lambda=-L_{x}\\ &\Delta x=-\left[G_{x}^{T}\Delta\lambda+L_{x}\right]\\ \end{split} (7) \displaystyle\begin{split}&I_{pp}\Delta p+G_{p}^{T}\Delta\lambda=-L_{p}\\ &\Delta p=-\left[G_{p}^{T}\Delta\lambda+L_{p}\right]\end{split} (8) \displaystyle\begin{split}&G_{x}\Delta x+G_{p}\Delta p=-G\\ &-G_{x}\left[G_{x}^{T}\Delta\lambda+L_{x}\right]-G_{p}\left[G_{p}^{T}\Delta\lambda+L_{p}\right]=-G\\ &\Delta\lambda=\left[G_{x}G_{x}^{T}+G_{p}G_{p}^{T}\right]^{-1}\left[G-G_{x}L_{x}-G_{p}L_{p}\right]\end{split} (9) Substituting for \Delta\lambda in equation 7 and 8 gives \displaystyle\begin{split}&\Delta x=-\left[G_{x}^{T}\left[G_{x}G_{x}^{T}+G_{p}G_{p}^{T}\right]^{-1}\left[G-G_{x}L_{x}-G_{p}L_{p}\right]+L_{x}\right]\\ &\Delta p=-\left[G_{p}^{T}\left[G_{x}G_{x}^{T}+G_{p}G_{p}^{T}\right]^{-1}\left[G-G_{x}L_{x}-G_{p}L_{p}\right]+L_{p}\right]\\ \end{split} (10) Instead of calculating the matrix explicitly and then taking its inverse in equation 9 , we use the conjugate gradient method, which only uses the cheaper Hessian-vector products and can be calculated efficiently. To do so, we introduce four new algorithms to efficiently deal with Jacobian-vector-product and vector-Jacobian-product of G_{x} , and G_{p} . Note that each of these algorithms can exploit GPU’s by using primitives defined in Wen-Mei et al. 2022 Algorithm 1 Computing \left[G_{p}v\right] Tangent vector v\in\mathbb{R}^{p\times 1} Y\leftarrow\text{Empty}(v)\in\mathbb{R}^{m\times n} \triangleright Initialize empty matrix Y Y[1]\leftarrow 0 \triangleright Initialize the first row of matrix to zero for k=2,3,\cdots,m do Y[k]\leftarrow\frac{\partial F_{k-1}}{\partial p}v end for y\leftarrow\text{Reshape}(Y)\in\mathbb{R}^{mn\times 1} \triangleright Vectorize the matrix return y Algorithm 2 Computing \left[v^{T}G_{p}\right] Co-tangent vector v\in\mathbb{R}^{mn\times 1} V\leftarrow\text{Reshape}(v)\in\mathbb{R}^{m\times n} y\leftarrow 0\in\mathbb{R}^{p\times 1} \triangleright Initialize empty vector y for k=2,3,\cdots,m do y\leftarrow y+V[k]^{T}\frac{\partial F_{k-1}}{\partial p} end for return y Algorithm 3 Computing \left[G_{x}v\right] Tangent vector v\in\mathbb{R}^{mn\times 1} V\leftarrow\text{Reshape}(v)\in\mathbb{R}^{m\times n} Y[1]\leftarrow V[1] \triangleright Initialize the first row of the matrix for k=2,3,\cdots,m do \triangleright Forward pass Y[k]\leftarrow V[k]-\frac{\partial F_{k-1}}{\partial x_{k-1}}Y[k-1] end for y\leftarrow\text{Reshape}(Y)\in\mathbb{R}^{mn\times 1} \triangleright Vectorize the matrix return y Algorithm 4 Computing \left[v^{T}G_{x}\right] Co-tangent vector v\in\mathbb{R}^{mn\times 1} V\leftarrow\text{Reshape}(v)\in\mathbb{R}^{m\times n} Y\leftarrow\text{Empty}(v)\in\mathbb{R}^{m\times n} \triangleright Initialize empty matrix Y Y[m]\leftarrow-V[m] \triangleright Initialize the last row of the matrix for k=m,m-1,\cdots,2 do \triangleright