SWoMo: Neuro-Symbolic World Model for Cataract Surgery Simulation

Ssharvien Kumar Sivakumar Akwele Johnson, Anirudh Dhingra, Yannik Frisch, Ghazal Ghazaei, Anirban Mukhopadhyay · 2026

Read SWoMo: Neuro-Symbolic World Model for Cataract Surgery Simulation by Ssharvien Kumar Sivakumar Akwele Johnson, Anirudh Dhingra, Yannik Frisch, Ghazal Ghazaei, Anirban Mukhopadhyay (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16530.txt.

Introduction

Sivakumar and five others present SWoMo, a neuro-symbolic model for cataract surgery splitting motion from realism. Surgical simulation matters for training surgeons and self-run agents, and world models scale practice to diverse ops. Current methods fail clinical bars including realism, contacts, and cases beyond training. The symbolic component, a simulator with scene graphs, models motion and tool-tissue interactions, while diffusion gives appearance. Inverse pairing reconstructs real surgical videos in the simulator for sim-to-real translation. Tests show gains, with generalization to unseen geometries and phase detection.

Catalog record

CREATORSsharvien Kumar Sivakumar Akwele Johnson, Anirudh Dhingra, Yannik Frisch, Ghazal Ghazaei, Anirban Mukhopadhyay
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23762
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.16530

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SWoMo: Neuro-Symbolic World Model for Cataract Surgery Simulation Ssharvien Kumar Sivakumar Akwele Johnson, Anirudh Dhingra, Yannik Frisch, Ghazal Ghazaei, Anirban Mukhopadhyay Abstract Realistic surgical simulation plays a crucial role in training novice surgeons and in the development of autonomous agents. World models can scale such simulation environments to realistic and diverse procedures by predicting future patient states conditioned on current observations and surgical actions. However, current state-of-the-art approaches often fail to satisfy key criteria required for clinical applicability, including visual realism, physically grounded interactions, and the ability to simulate scenarios beyond the training distribution. Hence, we introduce SWoMo, a neuro-symbolic world model for cataract surgery simulation that decouples motion generation from visual realism. The symbolic component, consisting of a rule-based simulator and scene graph representations, models motion dynamics and tool-tissue interactions, while a diffusion model produces realistic visual appearance, including textures and tissue deformations. We propose an inverse pairing strategy that reconstructs real surgical videos in the simulator to obtain paired simulated and real videos, which are then used to train our video diffusion model for the reverse objective of sim-to-real translation. Our experiments show both qualitative and quantitative improvements over prior work. We demonstrate that our simulator further satisfies the key criteria, including generalisation to unseen interaction geometries, improvements in downstream phase detection, and unsupervised video style transfer. The code, data, and model weights are available at: https://ssharvienkumar.github.io/SWoMo/ . 1 Introduction Surgery demands high precision, where irreversible actions must be carefully coordinated through structured spatial reasoning and causal understanding [ 6 ] . Simulated environments provide a safe and controllable setting in which such dependencies can be explored [ 8 ] , enabling applications ranging from risk-free, immersive surgical skills training for novice surgeons to the development of autonomous surgical robotic agents that can reason and plan interactively [ 23 , 25 ] . However, scaling such environments to realistic and diverse procedures requires models that integrate continuous perception with explicit representations of the geometry and dynamics of the surgical scene [ 3 ] . Surgical world models address this need by enabling interactive simulation and predictive modelling of future patient states conditioned on current observations and actions [ 11 , 20 ] , thereby unifying perception, dynamics, and action within a single framework [ 3 ] . However, the clinical applicability of a surgical world model hinges on satisfying three key criteria: (I) The simulated environment must achieve high visual realism to minimise the sim-to-real gap, as low fidelity limits policy transfer [ 18 ] . Yet this remains a major challenge for traditional simulators, particularly in modelling complex instrument-tissue interactions [ 20 , 28 ] . (II) Equally important is ensuring physically grounded interactions , so that both learning agents and human trainees acquire behaviours that transfer reliably to real high-stakes surgical settings [ 34 ] . (III) The dynamic representation of the world model must go beyond the training distribution , enabling the generation of plausible outcomes under unseen tool geometries, insertion angles, and novel surgical workflows. Unfortunately, state-of-the-art research typically excels on one criterion while failing to satisfy others. Methods that attempt to control video generative models via various conditioning signals [ 2 , 12 , 26 , 28 ] fall short of interactive surgical simulation due to the lack of step-wise action conditioning. As a result, they provide limited fine-grained control for agent or trainee intervention and cannot be considered true world models. Their behaviour is also tightly coupled to the training distribution, degrading significantly under out-of-distribution (OOD) conditions [ 6 ] . World-model based approaches, such as SurgWM [ 20 ] , inspired by Genie [ 4 ] , learn interactive environments from unstructured surgical videos, but limit interaction to latent action codes rather than direct tool control. DreamGen [ 17 ] and SurgWorld [ 13 ] generate paired video-action data using inverse dynamics, improving policy learning, yet they lack physical grounding. Unpaired image translation methods [ 24 , 32 ] have also been explored to address the sim-to-real gap, but remain insufficient for modelling complex spatiotemporal tool-tissue interactions. Figure 1: Neuro-symbolic World Model for interactive cataract surgery simulation that decouples surgical interaction dynamics from visual appearance. Hence, we introduce SWoMo, an interactive neuro-symbolic world model for cataract surgery, as shown in Figure 1 , in which symbolic scene graphs and a rule-based physics simulator explicitly encode structure and constraints, while a diffusion model provides high-fidelity visual synthesis. Our proposed method explicitly decouples the modelling of surgical motion and interaction dynamics from visual appearance . This allows precise tool and anatomical motions to be preserved through physically grounded simulation while enabling the generation of entirely novel scenarios beyond the training distribution. Visual realism, such as texture and tissue deformation, is learned through data-driven synthesis using a diffusion model. To achieve this, we introduce inverse pairing strategy in which tool and anatomical motions are extracted from real surgical videos and replayed in the simulator, producing large-scale paired simulated and real videos. These pairs are then used to train a video-to-video diffusion model that translates simulated renderings back into realistic surgical videos. The simulator also produces corresponding segmentations, which are used to construct scene graphs that efficiently encode the scene and object relationships in a structured graphical format. We show that conditioning the diffusion model on scene graphs is crucial for mitigating issues caused by residual misalignment between simulated and real videos. Optional initial frame or style reference conditioning is added for subject-specific surgical simulation. We further demonstrate generalisation to unseen surgical interactions, notable improvements in downstream phase detection, and unsupervised video style transfer across datasets. 2 Method Our surgical world model decouples motion generation from visual realism. Motion and interaction dynamics are governed by symbolic components comprising of scene graphs \mathcal{G} and a rule-based simulator T(\cdot) , while visual appearance is generated by a denoising diffusion model, employing a neural network \boldsymbol{\epsilon}_{\theta}(\cdot) . We begin by extracting the anatomical configuration k^{\text{anat}}_{t} and tool kinematics k^{\text{tool}}_{t} from real surgical video x_{1:T} , and use them to construct a digital twin of the eye and replay these signals in this simulator. At time step t , we define the simulator state as \bar{x}_{t}=\{{k^{\text{anat}}_{t},k^{\text{tool}}_{t}}\} . Actions correspond to explicit surgical tool motions, a_{t}=\Delta k^{\text{tool}}_{t} . State transitions are governed by T(\cdot) such that (\bar{x}_{t+1},\bar{m}_{t+1})=T(\bar{x}_{t},a_{t}) , where \bar{x}_{t+1} denotes the next state along with its rendered simulated frame, and \bar{m}_{t+1} is the corresponding segmentation. From each simulator state, a scene graph is generated, \mathcal{G}_{t}=S(\bar{x}_{t},\bar{m}_{t}), encoding object geometry and relational structure. Final observations are then generated according to p_{\theta}(x_{t:t+n}\mid\bar{x}_{t:t+n},\mathcal{G}_{t:t+n}) , parameterized by the diffusion model, which translates simulated sequence \bar{x}_{t:t+n} into realistic sequence x_{t:t+n} . Here, n denotes the number of consecutive simulated frames jointly provided as conditioning to the diffusion model. Inverse Pairing: In this section, we describe the extraction of k^{\text{anat}}_{t} and k^{\text{tool}}_{t} from real surgical video x_{1:T} . We start by segmenting pupil and iris employing nnU-Net [ 16 ] , and corresponding segmentations are fitted with ellipses to obtain compact geometric parameters k^{\text{iris}}_{t} and k^{\text{pupil}}_{t} encoding centroid, orientation, and axis lengths. Motion of the eye globe decomposes into global and local components. Global motion , arising from camera or patient movement, is estimated by annotating

Read the complete text