Weight-Parameterization in Continuous Time Deep Neural Networks for Surrogate Modeling
Haley Rosso, Lars Ruthotto, Khachik Sargsyan · 2025
Read Weight-Parameterization in Continuous Time Deep Neural Networks for Surrogate Modeling by Haley Rosso, Lars Ruthotto, Khachik Sargsyan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22045.txt.
Introduction
Rosso, Ruthotto, and Sargsyan studied how to parameterize the changing weights of continuous-time neural networks in twenty twenty-five. Such models, like neural ordinary differential equations, serve as fast surrogates for complex physical systems. Their weights evolve over time, and learning them expressively yet stably under computing limits is a central challenge.
The authors constrain weight evolution to a low-dimensional subspace spanned by polynomial basis functions. They compare simple monomial bases with Legendre polynomials, which are mutually orthogonal, inside two architectures and two training schemes. Across three high-dimensional benchmark problems, Legendre parameterizations trained more stably, cost less to compute, and matched or beat both monomial and unconstrained models in accuracy. The findings clarify how basis choice trades expressivity against training efficiency in time-dependent networks.
Catalog record
| CREATOR | Haley Rosso, Lars Ruthotto, Khachik Sargsyan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22045
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Weight-Parameterization in Continuous Time Deep Neural Networks for Surrogate Modeling Haley Rosso, Lars Ruthotto, Khachik Sargsyan Abstract Continuous-time deep learning models, such as neural ordinary differential equations (ODEs), offer a promising framework for surrogate modeling of complex physical systems. A central challenge in training these models lies in learning expressive yet stable time-varying weights, particularly under computational constraints. This work investigates weight parameterization strategies that constrain the temporal evolution of weights to a low-dimensional subspace spanned by polynomial basis functions. We evaluate both monomial and Legendre polynomial bases within neural ODE and residual network (ResNet) architectures under discretize-then-optimize and optimize-then-discretize training paradigms. Experimental results across three high-dimensional benchmark problems show that Legendre parameterizations yield more stable training dynamics, reduce computational cost, and achieve accuracy comparable to or better than both monomial parameterizations and unconstrained weight models. These findings elucidate the role of basis choice in time-dependent weight parameterization and demonstrate that using orthogonal polynomial bases offers a favorable tradeoff between model expressivity and training efficiency. keywords neural ODEs, weight parameterization, surrogate modeling, residual neural networks 1 Introduction The development of efficient surrogate models is essential for accelerating simulations of high-dimensional, computationally expensive systems, particularly those governed by ordinary differential equations (ODEs). Surrogate modeling aims to replace complex physical or numerical models with cheaper approximations that preserve essential input-output behavior. This is particularly important for tasks where thousands of model evaluations may be required, such as uncertainty quantification Sudret 2019 and data assimilation Kovachki et al. 2021 . In recent years, deep neural networks (DNNs) have gained prominence as expressive and efficient surrogates in scientific machine learning contexts, especially for nonlinear systems where traditional reduced-order models may struggle Wang et al. 2025 ; Raissi et al. 2019 ; Lu et al. 2021 . Among these, continuous-time DNNs, such as neural ordinary differential equations (neural ODEs) and the continuous limit of residual networks (ResNets), offer promising surrogate modeling architectures. By modeling the forward pass of a neural network as the solution of an ODE, these architectures enable temporal smoothness and provide a structured framework for incorporating physical constraints or known system dynamics, improving interpretability and alignment with real-world processes. In comparison to standard deep networks, continuous-time DNNs are particularly well-suited for modeling dynamical systems, as the network’s evolution over time is governed by an explicit differential equation, allowing one to trace how inputs evolve through the network over time. For surrogate modeling of ODE systems, prior information about the underlying physics can be incorporated into the network architecture, such as symmetries or conservation laws, which can lead to more accurate models Wang et al. 2025 ; Raissi et al. 2019 . Continuous-time DNNs also provide a natural lens through which to apply techniques from control theory and numerical analysis. As a result, they are increasingly used to approximate dynamical systems or to emulate the solution operators of high-dimensional PDEs Chen et al. 2018 ; Ruthotto 2024 ; Günther et al. 2021 . While both neural ODEs and the continuous limit of ResNets make use of the same continuous framework, they are not equivalent. The first works specifically studying the continuous limit of ResNets, to our knowledge, are E 2017 ; Haber and Ruthotto 2017 ; in Haber and Ruthotto 2017 , the learning problem is given as an optimization problem where the weights are updated using the Gauss-Newton preconditioned conjugate gradient (PCG) method. The continuous limit of a ResNet, often derived from interpreting discrete residual layers as time steps, allows the weights to vary across layers and, by extension, over time Ruthotto 2024 ; Haber and Ruthotto 2017 . On the other hand, neural ODEs typically assume that the network dynamics are governed by a differential equation with time-invariant weights, which are optimized through an adjoint-based continuous-time training procedure Chen et al. 2018 . This key difference affects both model expressivity and numerical behavior. For instance, discretizing a neural ODE using an explicit scheme such as forward Euler or Runge-Kutta may yield poor approximations, especially with large step sizes, since its static-weight formulation cannot capture the same layer-to-layer variability as a time-varying ResNet Ott et al. 2020 ; Sander et al. 2022 . These structural differences also influence training pipelines; ResNets are often discretized first and then optimized, whereas neural ODEs are typically optimized in continuous time and then discretized for inference. Our work addresses both approaches using a unified framework based on time-dependent weight parameterization. In any case, the benefits offered by continuous-time DNNs come with a cost. Training continuous-time deep neural networks such as neural ODEs or deep ResNets often involves optimizing high-dimensional parameter spaces, particularly when time-varying weights are discretized at many points or parameterized with expressive basis functions. These difficulties are exacerbated in surrogate modeling for PDE-governed systems, which often involve stiff dynamics themselves, such as in convection–diffusion–reaction models Günther et al. 2021 ; Chen et al. 2018 ; Sudret 2019 . To mitigate these challenges, one promising strategy is to parameterize the time-dependent weights of the network using a small, pre-determined number of basis functions. Such weight parameterization can be interpreted as a form of model reduction, enforcing smoothness in the network while reducing the dimensionality of the parameter space and improving generalization. For example, B-spline parameterizations have been used to regularize training and promote stability Günther et al. 2021 , while polynomial basis functions such as Legendre or Chebyshev polynomials offer orthogonality properties that support efficient optimization Davis et al. 2020 . Despite these advantages, the impact of weight parameterization on surrogate modeling performance — particularly across different continuous-time architectures and polynomial basis functions — remains underexplored. In addition, most studies either discretize the network first and then optimize (common in ResNets), or solve an optimal control problem in continuous time (typical in neural ODEs) Li et al. 2019 ; Benning et al. 2019 ; Ruthotto 2024 , but few compare these approaches side-by-side under different parameterization regimes. Moreover, many works do not explicitly treat time as a network input or explore the use of orthogonal basis functions for temporal weights, especially in high-dimensional, PDE-based surrogate modeling tasks. In this work, we address these gaps by studying polynomial weight parameterization for continuous-time deep neural networks in the context of surrogate modeling for PDE systems. We explore two families of basis functions, monomials and Legendre polynomials, and apply them to both ResNets and neural ODEs. In our test cases, we also include a Hamiltonian-inspired system with forward propagation resembling a symmetric, second-order ODE modeled after the continuous version of a ResNet layer in Haber and Ruthotto 2017 . This system bears resemblance to a ResNet, with weights that vary with each layer, but utilizes a Verlet time integrator. To evaluate the efficacy of our proposed approaches, we considered surrogate modeling tasks tied to real-world scenarios such as climate modeling and physical phenomena. Specifically, we utilized the popular, high-dimensional training data sets ELM (E3SM Land Model), a component of the DOE Energy Exascale Earth System Model (E3SM) project, alongside convection diffusion reaction (CDR) and direct current resistivity (DCR). The CDR is a system of PDEs which measure various physical phenomena, and the DCR data represents an inverse conductivity problem using a PDE that models electric potential Newman et