Improving Full Waveform Inversion in Large Model Era
Yinan Feng, Peng Jin, Yuzhe Guo, Yinpeng Chen, Youzuo Lin · 2026
Read Improving Full Waveform Inversion in Large Model Era by Yinan Feng, Peng Jin, Yuzhe Guo, Yinpeng Chen, Youzuo Lin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-00377.txt.
Introduction
Feng and four colleagues scale a billion parameter model for full waveform inversion, which rebuilds underground speed maps from surface seismic waves. Small models trained on narrow data sets tend to overfit and fail on realistic geology. The team trains only on simulated and fairly plain data, scaling model size, data variety, and training method together. Their model leads on OpenFWI and closes much of the gap on six hard geology tests, inferring structures absent from training and lifting similarity scores markedly.
Catalog record
| CREATOR | Yinan Feng, Peng Jin, Yuzhe Guo, Yinpeng Chen, Youzuo Lin |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36036 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.00377
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improving Full Waveform Inversion in Large Model Era Yinan Feng, Peng Jin, Yuzhe Guo, Yinpeng Chen, Youzuo Lin Abstract Full Waveform Inversion (FWI) is a highly nonlinear and ill-posed problem that aims to recover subsurface velocity maps from surface-recorded seismic waveforms data. Existing data-driven FWI typically uses small models, as available datasets have limited volume, geological diversity, and spatial extent, leading to substantial concerns about overfitting. Although they perform well on synthetic datasets, current methods fail to generalize to more realistic geological structures. In this work, we show that a model trained entirely on simulated and relatively simple data can generalize remarkably well to challenging and unseen geological benchmarks. We provide a working recipe that tames a billion-parameter model for FWI through coordinated scaling across three axes: model capacity, data diversity, and training strategy. Our model achieves state-of-the-art performance on OpenFWI and significantly narrows the generalization gap in data-driven FWI. Across six challenging geophysical benchmarks, including Marmousi, 2D SEG/EAGE Salt and Overthrust, 2004 BP, Sigsbee, and SEAM Phase I, it infers complex structures absent from the training set and delivers significant performance improvements (SSIM from 0.5844 to 0.7669). Overall, our results demonstrate that with an appropriate scaling strategy, large models trained on simple synthetic data can achieve substantial generalization to more complex and realistic geological structures. 1 Introduction Figure 1 : Reconstruction results on challenging realistic benchmarks. BigFWI tends to collapse toward over-smoothed, mean-shaped solutions that miss key interfaces and salt bodies. In contrast, our model produces sharper boundaries and geologically meaningful structures. Subsurface imaging is critical in scientific and industrial applications, including earthquake monitoring 33 ; 32 , greenhouse gas storage 20 ; 37 , medical imaging 14 ; 23 , and oil and gas exploration 34 ; 36 . At its core, Full Waveform Inversion (FWI) serves as a representative method, which aims to reconstruct subsurface velocity maps from seismic data recorded on the surface, governed by the acoustic wave equation: \frac{\partial^{2}p(x,z,t)}{\partial t^{2}}=v^{2}(x,z)\nabla^{2}p(x,z,t)+f(x,z,t), (1) where p(x,z,t) is the pressure wavefield at spatial location (x,z) and time t , v(x,z) is the 2D vertical velocity map, \nabla^{2} is the Laplacian operator, and s(x,z,t) is the source term. In practice, seismic data are often the waveform data collected at the surface (i.e., p(x,z=0,t) ). While FWI has the potential to produce high-resolution velocity maps, the inverse problem itself is inherently non-linear and ill-posed. In addition, the traditional iterative solvers based on gradient descent or adjoint-state methods are computationally demanding and sensitive to initialization, noise, and local minima 34 ; 25 ; 13 . These challenges have motivated growing interest in data-driven deep learning methods. Recent advances in data-driven FWI have demonstrated the potential of deep neural networks to directly infer velocity maps from seismic data 3 ; 38 ; 43 ; 21 ; 11 ; 19 . However, these models typically operate at small to medium model scales. This is largely due to the limited scale of existing FWI datasets in terms of data volume, geological diversity, and spatial extent, which raises strong concerns about overfitting. Prior studies also report that modest increases in model size do not reliably improve performance 18 . Although current approaches achieve promising results on synthetic datasets, they struggle to generalize to more realistic geological structures, particularly those containing complex features such as salt bodies or strong heterogeneity. As illustrated in the second and third columns of Fig. 1 , a state-of-the-art end-to-end method, BigFWI 18 , often collapses into overly smoothed, mean-shaped solutions that fail to recover key interfaces or salt bodies, resulting in reconstructions that lack meaningful geological structure. In this work, we show that a large model trained entirely on simulated and relatively simple data can generalize remarkably well to challenging and unseen geological benchmarks. However, deploying large models for a scientific inverse problem is far from straightforward. Several fundamental questions arise. First, which architectural design can best capture seismic–velocity relationships, where accuracy and physical consistency are essential? Second, how can one mitigate data scarcity when existing datasets are orders of magnitude smaller than what large models typically require? Third, what learning strategies can effectively align the network with the underlying physical dependencies of wave propagation? Finally, how can post-processing be leveraged to enhance physical consistency and further improve reconstruction accuracy? We address these challenges by introducing a working recipe that tames a billion-parameter model for FWI through coordinated scaling across three axes: model capacity, data diversity, and training strategy. As illustrated in the rightmost column of Fig. 1 , our method delivers significantly better reconstructions, recovering sharper layer boundaries and geologically meaningful high-velocity regions that prior methods fail to capture. At the model level, our framework employs a one-billion-parameter transformer backbone that enables global contextual modeling between seismic and velocity tokens. Rather than relying on causal next-token generation, we adopt non-causal parallel decoding , allowing all velocity tokens to be generated simultaneously with full self-attention, significantly improving efficiency and accuracy. Complementing this backbone, we employ a ViT-VQGAN tokenizer 40 with an expanded bottleneck width that preserves fine geological details, avoiding the excessive information compression common in conventional VQGANs. The wide latent space is particularly well-suited for scientific inversion tasks where spatial fidelity is critical. To address data scarcity, we train a latent diffusion model 27 on the OpenFWI dataset 9 to synthesize additional velocity maps. Each generated sample is paired with forward-simulated seismic data, ensuring physical consistency between the two modalities. This process expands the training corpus from 408k to over five million velocity–seismic pairs and introduces hybrid geological structures that mix heterogeneous features across sub-datasets. By substantially increasing both the quantity and structural diversity of training data, this diffusion-driven augmentation effectively meets the data requirements of the large transformer without relying on any real or realistic geological models. Figure 2 : Overview of progressive model enhancements and final reconstruction quality. (a) Step-wise performance improvement as each component is introduced—from baseline to data augmentation, non-causal modeling, ViT VQGAN (without bottleneck), reinforcement learning alignment, and latent gradient refinement. (b) Illustration of the final reconstruction results, showing sharp structures and fine geological details across diverse scenarios. In terms of learning strategies, we adopt a two-stage training pipeline. In the supervised pre-training stage, the model learns token-wise mappings from seismic-conditioned inputs to velocity representations. In the RL-based post-training stage, we further enhance model robustness through reinforcement learning (RL) optimization 29 ; 44 . We formulate velocity token generation as a discrete policy optimization problem, and utilize map-level rewards that encourage geological continuity and physical plausibility, enabling the model to move beyond the limitations of purely supervised learning. In the post-processing stage, we first perform a physics-based gradient descent (GD) refinement to improve the physical consistency of the predicted velocity maps. This step optimizes the continuous latent embeddings in the VQGAN decoder with respect to forward-modeling residuals, enforcing consistency with the governing wave equation. To further mitigate the ill-posedness of FWI, we additionally employ stochastic sampling and ensemble aggregation to combine multiple reconstructions when evaluating on realistic and unseen datasets, thereby reducing ambiguity and reducing predictive uncertainty. Through extensive experiments, we demonstrate that each component of our large-model recipe contributes to improved performance and stability. Figure 2 summarizes the progressive gains and final reconstruction quality