Variational autoencoder for inference of nonlinear mixed effect models based on ordinary differential equations

Zhe Li, Mélanie Prague, Rodolphe Thiébaut, Quentin Clairon \emailx [email protected], [email protected], [email protected], [email protected] Univ. Bordeaux, INSERM BPH, U1219, Inria SISTM team, VRI, France · 2026

Read Variational autoencoder for inference of nonlinear mixed effect models based on ordinary differential equations by Zhe Li, Mélanie Prague, Rodolphe Thiébaut, Quentin Clairon \emailx [email protected], [email protected], [email protected], [email protected] Univ. Bordeaux, INSERM BPH, U1219, Inria SISTM team, VRI, France (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17400.txt.

Introduction

Zhe Li, Mélanie Prague, Rodolphe Thiébaut, and Quentin Clairon present a new way to fit drug and disease models. Their models are nonlinear mixed-effects models built on ordinary differential equations. Data come from many subjects over time. The standard fit tool is likelihood-based and leans on Monte Carlo steps. It can be slow and fragile in mid-size problems.

The team uses a variational autoencoder instead. Their design is identifiability-aware, so it flags traits the data cannot pin down. That honesty matters in health research. Wrong certainty can mislead dose and care choices. A faster, aware fit tool could widen use of rich models in clinics and trials.

Catalog record

CREATORZhe Li, Mélanie Prague, Rodolphe Thiébaut, Quentin Clairon \emailx [email protected], [email protected], [email protected], [email protected] Univ. Bordeaux, INSERM BPH, U1219, Inria SISTM team, VRI, France
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-17400

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Variational autoencoder for inference of nonlinear mixed effect models based on ordinary differential equations Zhe Li, Mélanie Prague, Rodolphe Thiébaut, Quentin Clairon \emailx [email protected], [email protected], [email protected], [email protected] Univ. Bordeaux, INSERM BPH, U1219, Inria SISTM team, VRI, France Abstract We propose a variationl autoencoder approach for identifiability-awared parameter estimation in nonlinear mixed-effects models based on ordinary differential equations (NLME-ODEs) using longitudinal data from multiple subjects. In moderate dimensions, likelihood-based inference via the stochastic approximation EM algorithm (SAEM) is widely used, but it relies on Markov Chain Monte-Carlo (MCMC) to approximate subject-specific posteriors. As model complexity increases or observations per subject are sparse and irregular, performance often deteriorates due to a complex, multimodal likelihood surface which may lead to MCMC convergence difficulties. We instead estimate parameters by maximizing the evidence lower bound (ELBO), a regularized surrogate for the marginal likelihood. A shared encoder amortizes inference of subject-specific random effects by avoiding per-subject optimization and the use of MCMC. Beyond pointwise estimation, we quantify parameter uncertainty using observed-information-based variance estimator and verify that practical identifiability of the model parameters is not compromised by nuisance parameters introduced in the encoder. We evaluated the method in three simulation case studies (pharmacokinetics, humoral response to vaccination, and TGF- β \beta activation dynamics in asthmatic airways) and on a real-world antibody kinetics dataset, comparing against SAEM baselines. \pagerange Variational autoencoder for inference of nonlinear mixed effect models based on ordinary differential equations – References \artmonth Feburary 10.13140/RG.2.2.32869.46567 Volume: XX keywords Longitudinal population data analysis;Neural Networks;Variational bayeisien inference; 1 Introduction Ordinary differential equation (ODE) models are widely used in systems biology, mathematical medicine, virology, vaccinology, and pharmacometrics due to their interpretability and predictive power. However, classical ODEs typically describe an average trajectory and do not capture inter-individual variability in longitudinal population data, which are often sparse, irregularly sampled, and noisy. To address this, ODEs are commonly embedded in nonlinear mixed-effects (NLME) models, yielding NLME-ODEs [ Lavielle, 2014 ] , where subject-specific random effects capture heterogeneity while preserving a shared mechanistic structure. More formally, NLME-ODEs consider a population of n subjects. For subject i\in\{1,\dots,n\} , observations are collected at (possibly irregular) times t_{ij} , j=1,\dots,n_{i} , where n_{i} denotes the number of measurements for subject i ; we write Y_{ij} for the measurement taken at time t_{ij} . The dynamics of the subject i is described by a D_{X} -dimensional ODEs: \begin{cases}\dfrac{dX_{i}(t)}{dt}=f_{\theta_{i}}\bigl(t,X_{i}(t)\bigr),\\ X_{i}(0)=x_{i,0}.\end{cases} where f_{\theta_{i}} is a D_{X} -dimensional vector field defining the system dynamics, \theta_{i} is a vector of subject-specific parameters, and x_{i,0} denotes the initial condition. Inter-individual variability is modeled through a mixed-effects formulation \theta_{i}=g(\bar{\theta},b_{i}), where \bar{\theta} represents the population-level mean and b_{i} denotes the subject-specific deviation around this mean, also known as the random effects. Here, g(\cdot) is a (possibly nonlinear) transformation function, for example through additive or log-normal parameterizations to enforce constraints such as positivity. We assume b_{i}\sim p_{\phi}(b_{i})=\mathcal{N}(0,\Omega) , where b_{i}=(b_{i1},\ldots,b_{id_{b}})^{\top}\in\mathbb{R}^{D_{b}} is the D_{b} -dimensional vector of random effects and \Omega\in\mathbb{R}^{D_{b}\times D_{b}} is a symmetric positive-definite covariance matrix. To guarantee positive definiteness, we parameterize \Omega via its Cholesky factor \Omega=L_{\Omega}L_{\Omega}^{\top} . Our goal is to estimate the population parameters \phi=(\bar{\theta},\,L_{\Omega},\,L_{\Sigma}) of dimension D_{\phi} from the noisy and discrete observations Y_{ij} generated according to the following observation model: Y_{ij}=h_{\theta_{i}}(X_{i}(t_{ij}))+\epsilon_{ij},\qquad\epsilon_{ij}\sim\mathcal{N}(\mathbf{0},\,\Sigma). (1) Here, h_{\theta_{i}} is the observation function mapping the i -th subject state represented by the ODEs solution X_{i} to the measure Y_{ij} and \Sigma\in\mathbb{R}^{D_{n_{i}}\times D_{n_{i}}} is the covariance matrix of the measurement noise which is parameterized by its Cholesky factor \Sigma=L_{\Sigma}L_{\Sigma}^{\top} . As in classical inference problems, NLME-ODEs parameters can be estimated by maximum likelihood, marginalizing over the random effects b_{i} viewed as latent variables. However, maximizing the marginal likelihood requires repeated integration over b_{i} , which is generally intractable in nonlinear ODE models. Early approaches such as first-order (FO) and first-order conditional estimation (FOCE) rely on local Taylor/Laplace approximations to obtain a tractable likelihood [ Pinheiro and Bates, 1995 ] . These approximations can be inaccurate under strong nonlinearities, non-Gaussian random effects, or sparse individual data, settings common in NLME-ODEs; adaptive Gaussian quadrature can improve accuracy [ Prague et al., 2013 ] but becomes computationally prohibitive as the number of random effects grows. A widely used method is the stochastic approximation EM (SAEM) algorithm [ Delyon et al., 1999 ; Lavielle and Mentré, 2007 ] , which avoids explicit numerical integration by sampling random effects from their conditional distribution (typically via MCMC) and updating parameters through stochastic approximation. SAEM is often more accurate and stable than FO/FOCE or quadrature for nonlinear models and is implemented in Monolix and Pumas [ Simulations Plus, 2024 ; Rackauckas et al., 2020 ] . Nonetheless, for sparse data and/or high-dimensional random effects, SAEM may become computationally expensive and less reliable, motivating scalable variational alternatives. SAEM inherits several limitations from its reliance on MCMC in the E-step. In sparse settings, the conditional posterior of the random effects is weakly informed by the data, which can lead to slow mixing and poor exploration. As the dimension of the random effects increases, MCMC becomes increasingly expensive and more sensitive to multimodality, potentially converging to local modes and yielding non-identifiable solutions. Bayesian MCMC-based inference or MAP estimation faces similar computational and convergence issues in the same regimes [ Lunn et al., 2000 ; Carpenter et al., 2017 ] . More fundamentally, NLME-ODEs likelihoods are often highly non-convex because ODE solutions depend nonlinearly on the parameters, producing rugged objective landscapes and compounding identifiability and optimization difficulties. These challenges motivate optimization-based alternatives that can regularize inference and scale to complex models. Variational inference (VI) replaces stochastic sampling with a tractable, parametric approximation of the posterior, optimized by maximizing a lower bound. In particular, variational autoencoders (VAEs) use neural networks to parameterize this approximation, offering a scalable and flexible framework for NLME-ODEs inference. This coupling between NLME-ODEs and neural networks have already been investigated for other purposes. Qian et al. [2021] proposed to integrates expert pharmacological ODEs into Neural ODEs to provide additional insights to the clinicians in disease-progression modeling. Bräm et al. [2025] introduced low-dimensional neural ODEs implemented directly in Monolix [ Simulations Plus, 2024 ] , showing how inter-individual variability can be incorporated into neural ODEs parameters while preserving pharmacometric interpretability. Janssen et al. [2024] compared FO/FOCE and VI approaches for mixed-effect estimation in Deep Compartment Models. Arruda et al. [2024] developed an amortized inference framework in which a conditional normalizing flow is trained on simulated data generated from the parametric ODE model to approximate the parameter posterior distribution. Martensen et al. [2024] introduced a Deep Nonlinear Mixed-Effect [ PumasAI, 2025 ] modeling framework in which a neural network term is embedded directly into the ODE to represent unknown system dynamics. Roeder et al. [2019] performed amortised Bayesian inference for NLME-ODEs through a conditional VAE with population-, group-, and individual-level latent blocks. Despite these advances, most deep or variational extensions of NLME-ODEs emphasize predictive performance and scalability over statistical interpretability and identifiability. Amortized approaches [ Roeder et al., 2019 ; Arruda et al., 2024 ] provide efficient posterior approximations but do not explicitly address identifiability. Conversely, methods that embed neural networks into pharmacometric ODEs [ Bräm et al., 2025 ; Martensen et al., 2024 ] typically rely on SAEM or FOCE, and thus

Read the complete text