Optimization Benchmark for Diffusion Models on Dynamical Systems

Fabian Schaipp · 2025

Read Optimization Benchmark for Diffusion Models on Dynamical Systems by Fabian Schaipp (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19376.txt.

Introduction

Fabian Schaipp benchmarks optimization algorithms for training diffusion models on dynamical systems, a setting missing from most optimizer comparisons. Diffusion models are usually judged on text or images, so claims about new optimizers may not carry over to denoising tasks on flow trajectories.

Two newer methods, Muon and SOAP, prove highly efficient substitutes for the standard AdamW, reaching notably lower final loss. Schaipp also rechecks known training phenomena in this setting, including how learning-rate schedules shape training and how large the gap between adaptive methods and plain gradient descent really is. Practitioners get guidance grounded in their own task type.

Catalog record

CREATORFabian Schaipp
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-19376

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Optimization Benchmark for Diffusion Models on Dynamical Systems Fabian Schaipp Abstract The training of diffusion models is often absent in the evaluation of new optimization techniques. In this work, we benchmark recent optimization algorithms for training a diffusion model for denoising flow trajectories. We observe that Muon and SOAP are highly efficient alternatives to AdamW (18% lower final loss). We also revisit several recent phenomena related to the training of models for text or image applications in the context of diffusion model training. This includes the impact of the learning-rate schedule on the training dynamics, and the performance gap between Adam and SGD . 1 Introduction Over the last decade, the focus of optimization research has seen a shift towards applications in image classification and language modeling, particularly LLM pretraining. The training of diffusion models , despite their impressive success and wide range of applications, is usually absent from empirical validation in optimization research. Even the most extensive efforts on optimization benchmarking ( Schmidt et al. 2021 ; Dahl et al. 2023 ; Kasimbeg et al. 2025 ) do not contain results on diffusion models. Further, it remains unclear whether newly proposed methods, such as SOAP ( Vyas et al. 2025 ) or Muon ( Jordan et al. 2024 ) , are equally effective outside of LLM pretraining. In this work, we validate whether recent trends in optimization for deep learning transfer to the training of diffusion models. In particular, our benchmark problem concerns training a diffusion model for denoising trajectories of dynamical systems, where the training data is obtained from fluid dynamics simulations. Our benchmark problem originally has been used for score-based data assimilation ( Rozet & Louppe 2023 ) ; compared to the setting of LLM pretraining, it is different in terms of model architecture and loss function, data domain and training regime (multi/single epoch). In order to run multiple seeds and hyperparameter configurations for all methods, our computational constraints only allow for relatively small-scale problems ( \sim 23M parameters). Despite this limitation with respect to scale, the modeling technique from our benchmark problem has been successfully applied to diffusion-based data assimilation for regional and global weather and climate simulation ( Manshausen et al. 2024 ; Schmidt et al. 2025 ; Andry et al. 2025 ) . The findings of this benchmark might therefore be relevant and interesting to researchers who are training diffusion models for these scientific applications. Benchmarking in optimization for machine learning. The most extensive optimization benchmarking effort in recent years has been the AlgoPerf: Training Algorithms benchmark ( Dahl et al. 2023 ; Kasimbeg et al. 2025 ) . It consists of a variety of workloads, such as image classification and reconstruction, speech recognition, language translation, molecular property prediction, and click-through rate prediction. With Shampoo ( Gupta et al. 2018 ; Anil et al. 2020 ) being one of the competition winners, the benchmark sparked renewed interest in dense matrix preconditioning techniques and led to the development of new algorithms, such as SOAP ( Vyas et al. 2025 ) and Muon ( Jordan et al. 2024 ) . Semenov et al. 2025 and Wen et al. 2025 recently conducted extensive benchmarking for LLM pretraining. Here, we study whether these new methods can also shine for our diffusion training task. In particular, we compare the performance of SOAP , Muon and ScheduleFree ( Defazio et al. 2024 ) to the baseline method AdamW ( Loshchilov & Hutter 2019 ) . Performance gap between Adam and SGD . In contrast to image classification with convolutional networks, where SGD and Adam perform equally well (if properly tuned), it is well known that SGD does not easily 1 1 1 Recent works show that SGD can close the gap to Adam also for language tasks when using very small batch sizes ( Srećković et al. 2025 ; Marek et al. 2025 ) , or when applying Adam only on the weights of the embedding layers ( Zhao et al. 2025 ) . achieve the same performance as Adam for language modeling tasks ( Zhang et al. 2020 ; Kunstner et al. 2023 ) . Kunstner et al. 2024 further showed that imbalance of the class labels is sufficient to observe a gap between Adam and SGD . It remains unclear in which way other factors (for example, components of the model architecture) can have the same effect. Here, we investigate whether SGD can close the gap to Adam for an instance of diffusion model training, where the argument of class imbalance is not applicable. Summary and main findings. Muon and SOAP prove to be highly efficient also for diffusion model training. Despite their higher runtime per step compared to AdamW , they achieve lower final loss values. ScheduleFree almost matches AdamW in terms of loss (without the need for scheduling), however we observe inferior generative quality. Similar effects can be observed for the wsd schedule, which leads us to the conjecture that the entire training trajectory (and not only the final loss) is important for the quality of the trained diffusion model. We also observe a clear gap between Adam and SGD , which in this case can not be attributed to class imbalance. 2 Experimental Setup Our experimental setup for training the diffusion model is following closely the setup of Rozet & Louppe 2023 : they train a U-Net model ( Ronneberger et al. 2015 ) which learns the score function of a dynamical system trajectory, obtained from the velocity field governed by the Navier-Stokes equations with Kolmogorov flow ( Kochkov et al. 2021 ) . Using the standard DDPM approach ( Ho et al. 2020 ) , the score function is learned by denoising data points sampled from the true distribution. We refer to Section A.1 in the appendix for a detailed description of architecture and training data. Hyperparameter tuning. For each optimizer, we tune learning rate and weight decay separately (see Fig. 9 for a detailed view on the grid search). In general, we run three different seeds for each setting, and average all metrics across seeds. If not specified otherwise, we run for 1024 epochs with a linear-decay learning-rate schedule. Compared to Rozet & Louppe 2023 , we add warmup and gradient clipping by default (which lead to a minute reduction of the loss). A summary of the default hyperparameter settings is given in Table 1 . Computational cost. A single run over 1024 epochs with AdamW takes roughly one hour one a single NVIDIA A100 GPU (this includes the end-of-epoch evaluations). In total, we executed \sim 600 training runs, and utilized \sim 830 A100-hours in total. All experiments are conducted with Pytorch ( Paszke et al. 2019 ) of version 2.5.1 . 3 Results Naming conventions. We use Adam and AdamW interchangeably. ScheduleFree always refers to the AdamW version presented by Defazio et al. 2024 . Figure 1 : (Left) Final validation loss (averaged over the last five epochs) for each method and learning rate. Enlarged dot marks best learning rate. (Right) Validation loss curve for the best found setup for each method. Legend indicates learning rate (lr) and weight decay (wd) values. To obtain smoother curves we plot exponential moving averages with

Read the complete text