Autoregressive deep learning for real-time simulation of soft tissue dynamics during virtual neurosurgery
Fabian Greifeneder, Wolfgang Fenz, Benedikt Alkin, Johannes Brandstetter, Michael Giretzlehner, Philipp Moser · 2026
Read Autoregressive deep learning for real-time simulation of soft tissue dynamics during virtual neurosurgery by Fabian Greifeneder, Wolfgang Fenz, Benedikt Alkin, Johannes Brandstetter, Michael Giretzlehner, Philipp Moser (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-13676.txt.
Introduction
Fabian Greifeneder and co-authors build a fast brain tissue simulator for virtual neurosurgery in 2026. True-to-life training tools must show how brain tissue bends and shifts. Tool-tissue contact makes complex, nonlinear shapes as the work goes on. Classic solvers can capture that, but they run too slowly for live use.
The team uses autoregressive deep learning, which predicts each next state from the last. That style can run fast once trained, step by step in real time. Trainees could then feel how tissue reacts as they work. Fast, true tissue models could make surgical training safer and richer.
Catalog record
| CREATOR | Fabian Greifeneder, Wolfgang Fenz, Benedikt Alkin, Johannes Brandstetter, Michael Giretzlehner, Philipp Moser |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-13676
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Autoregressive deep learning for real-time simulation of soft tissue dynamics during virtual neurosurgery Fabian Greifeneder, Wolfgang Fenz, Benedikt Alkin, Johannes Brandstetter, Michael Giretzlehner, Philipp Moser Abstract Accurate simulation of brain deformation is a key component for developing realistic, interactive neurosurgical simulators, as complex nonlinear deformations must be captured to ensure realistic tool–tissue interactions. However, traditional numerical solvers often fall short in meeting real-time performance requirements. To overcome this, we introduce a deep learning-based surrogate model that efficiently simulates transient brain deformation caused by continuous interactions between surgical instruments and the virtual brain geometry. Building on Universal Physics Transformers, our approach operates directly on large-scale mesh data and is trained on an extensive dataset generated from nonlinear finite element simulations, covering a broad spectrum of temporal instrument-tissue interaction scenarios. To reduce the accumulation of errors in autoregressive inference, we propose a stochastic teacher forcing strategy applied during model training. Specifically, training consists of short stochastic rollouts in which the proportion of ground truth inputs is gradually decreased in favor of model-generated predictions. Our results show that the proposed surrogate model achieves accurate and efficient predictions across a range of transient brain deformation scenarios, scaling to meshes with up to 150,000 nodes. The introduced stochastic teacher forcing technique substantially improves long-term rollout stability, reducing the maximum prediction error from 6.7 mm to 3.5 mm. We further integrate the trained surrogate model into an interactive neurosurgical simulation environment, achieving runtimes below 10 ms per simulation step on consumer-grade inference hardware. Our proposed deep learning framework enables rapid, smooth and accurate biomechanical simulations of dynamic brain tissue deformation, laying the foundation for realistic surgical training environments. 1 Introduction Surgical simulators are designed to offer safe, controlled, and anatomically realistic environments for surgical training, preoperative planning, and intraoperative guidance. These systems have demonstrated significant potential in enhancing both the technical skills and clinical reasoning of medical trainees, while also addressing key challenges such as variability in clinical training experiences and concerns about patient safety [ 47 , 28 ] . Surgical simulators are applied across a broad spectrum of clinical procedures, supporting tasks from endoscopic sinus surgery to hip arthroscopy [ 50 , 5 ] . In high-risk interventions (particularly those involving potential neurological damage or hemorrhage) virtual training environments are especially valuable, for example in neurosurgical contexts such as brain tumor resections and aneurysm clipping interventions [ 4 , 18 , 23 ] . A core requirement for neurosurgical simulators is the realistic simulation of soft tissue deformation [ 35 ] , which is essential for accurately capturing the biomechanical behavior of soft tissues under surgical manipulation [ 55 ] . These deformation processes are typically modeled using nonlinear elasticity theory, resulting in systems of partial differential equations (PDEs) that describe the tissue’s dynamic response to applied forces. Achieving fast and high-fidelity solutions to these equations remains a significant challenge [ 39 ] . Real-time interactivity, however, is crucial for delivering immediate visual feedback, enabling haptic responses, and allowing dynamic adjustments to surgical plans based on user-controlled tool-tissue interactions. Classical methods for solving systems of PDEs commonly rely on the finite element method (FEM) [ 10 ] . Achieving physiologically accurate simulations often requires nonlinear material formulations and the conversion of patient-specific geometries into high-resolution meshes, both of which demand computationally expensive iterative solvers. Moreover, the brain’s biomechanical properties (e.g., viscoelasticity, heterogeneity, and sensitivity to small-scale forces) make high-fidelity modeling particularly challenging. These factors necessitate high spatial and temporal resolution to realistically simulate responses to surgical actions. The complexity is further compounded by the introduction of surgical tool interactions, which constitute dynamic boundary conditions and contact mechanics [ 38 ] . These factors collectively result in significant computational overhead. Although explicit methods like Total Lagrange Explicit Dynamics (TLED) [ 37 ] offer performance benefits in certain scenarios, a trade-off exists between precision and computational efficiency, particularly for highly detailed simulations of complex tissue interactions. As a result, conventional numerical methods can be impractical for real-time applications in complex neurosurgical simulators, highlighting the need for alternative approaches. In recent years, deep neural network (DNN)-based surrogate models have emerged as computationally efficient complements or alternatives to traditional numerical PDE solvers. Such surrogate models have demonstrated significant potential in capturing complex physical phenomena across various domains of science and engineering [ 57 , 48 ] . Notable applications include fluid dynamics [ 43 , 52 ] , solid mechanics [ 14 ] , and weather forecasting [ 8 , 7 , 27 ] . Once trained on data pre-computed with numerical solvers, these surrogate models allow for highly accurate and fast predictions of simulation outcomes. These benefits make surrogate models an ideal candidate to be be integrated into surgical simulation pipelines, replacing the computationally intensive numerical solver components. Common surrogate models often build upon convolution-based architectures such as U-Nets [ 20 ] or ResNets [ 19 ] . These methods typically operate on regular grids and require projecting mesh-based data onto grids [ 36 ] . This projection step can lead to a loss of geometric detail, and imposes additional memory and computational burdens when dealing with complex geometries. As a result, convolution-based architectures can be cumbersome and inefficient when working directly with mesh-based simulation data. Many recent surrogate models rely on neural operators [ 30 , 26 , 2 , 31 , 53 , 32 ] , which are a class of machine learning models designed to learn mappings between function spaces, such as those arising in PDEs. Instead of approximating specific solutions at fixed input points, neural operators aim to learn the entire solution operator, i.e., a function that maps input functions (e.g., initial or boundary conditions) to output functions (e.g., solution field). Such capabilities make neural operators particularly well-suited for capturing the fine-grained, spatially continuous deformations. We chose Universal Physics Transformers (UPT) [ 2 ] as a backbone architecture due to their flexible encoder-approximator-decoder blocks and demonstrated applicability to multi-physics [ 3 ] and computational fluid [ 1 ] dynamics. While other backbones such as OFormer [ 29 ] , Transolver [ 54 ] , MeshGraphNet [ 42 ] , or DoMINO [ 44 ] could in principle be used, recent results [ 1 ] suggest that UPT offer superior performance and scalability compared to these alternatives. In this work, we develop a UPT-based surrogate model, tailored for predicting transient soft tissue deformation caused by continuous interactions of surgical instruments with the virtual brain mesh. While there has already been prior work on deep learning-accelerated simulation of soft tissue deformation [ 46 , 45 , 33 , 25 , 56 ] , these reports often rely on convolutional, cellular, or graph-based architectures, which can be computationally demanding and less scalable when applied to high-resolution meshes. To the best of our knowledge, no prior work has demonstrated autoregressive inference, which is a critical component for modeling transient tissue dynamics enabling a tightly coupled interaction loop between the surgical instrument and the virtual brain mesh. Our high-fidelity approach constitutes a promising path for future neurosurgical simulations, providing surgeons with a realistic training environment to practice and refine complex surgical interventions. 2 Methods 2.1 Overview of the autoregressive framework At the core of our methodology is an