Actionable Real-Time Modeling of Surgical Team Dynamics via Time-Expanded Interaction Graphs
Vincenzo Marco De Luca, Antonio Longa, Giovanna Varni, Andrea Passerini · 2026
Read Actionable Real-Time Modeling of Surgical Team Dynamics via Time-Expanded Interaction Graphs by Vincenzo Marco De Luca, Antonio Longa, Giovanna Varni, Andrea Passerini (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04169.txt.
Introduction
Vincenzo Marco De Luca, Antonio Longa, Giovanna Varni, and Andrea Passerini model surgical team dynamics in real time, using graphs that expand over time. Surgical performance comes from craft plus non-technical skills such as talking and coordinating. Yet surgical AI mostly models visual workflow signals. In their setup, team members become time-indexed nodes. Messages between them become directed edges.
The expansion over time lets them model the interactions as they unfold. It still allows fast inference with a static graph neural network. The model predicts how far a procedure runs from its expected duration. Beyond prediction, a what-if analysis finds the smallest changes in how the team talks. It also finds plain behavioral variables tied to better predicted outcomes. Tests on recorded procedures show earlier flags for prolonged interventions. They also give coherent, actionable explanations.
Catalog record
| CREATOR | Vincenzo Marco De Luca, Antonio Longa, Giovanna Varni, Andrea Passerini |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.04169
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Actionable Real-Time Modeling of Surgical Team Dynamics via Time-Expanded Interaction Graphs Vincenzo Marco De Luca, Antonio Longa, Giovanna Varni, Andrea Passerini Abstract Surgical team performance arises from complex interactions between technical execution and non-technical skills, including communication and coordination dynamics. However, current surgical AI systems predominantly model visual workflow signals, lacking structured representations of intraoperative team interactions over time. We propose a real-time actionable approach for modeling surgical team dynamics using time-expanded interaction graphs, where team members are modeled as time-indexed nodes and communication exchanges define directed edges. This spatio-temporal expansion enables dynamic interaction modeling, while allowing efficient inference with a static graph neural network. The model predicts procedural efficiency as the deviation from the expected duration and supports real-time deployment. Beyond prediction, we perform a counterfactual analysis to identify minimal changes in communication structure and interpretable behavioral variables associated with improved predicted outcomes. Experiments on recorded surgical procedures show that structured modeling of team interactions improves early identification of prolonged interventions and provides coherent, actionable explanations. This work advances surgical AI toward real-time, team-aware, and actionable decision support in the operating room. keywords Surgical Data Science \sep Graph Learning \sep Actionable AI \sep Counterfactual Explainability \sep Team Modeling 1 Introduction Actionable AI refers to systems that go beyond prediction or explanation by providing recommendations that users can act upon to achieve desired outcomes while considering relevant trade-offs [ 22 ] . High-stakes scenarios particularly benefit from Actionable AI, where predictive accuracy alone is insufficient and decision support systems must translate model outputs into actionable feedback. The operating room (OR) exemplifies these settings due to its intrinsic complexity and the critical role played by teamwork in ensuring safe and efficient procedures [ 23 ] . Fields such as Surgical Data Science (SDS) aim to improve interventional healthcare through procedural data analysis [ 27 , 29 ] . However, current SDS research predominantly focuses on visual workflow modeling [ 7 , 13 ] and technical skill assessment [ 12 , 10 ] , while intraoperative team dynamics remain largely unexplored [ 11 , 8 , 16 ] , despite team performance emerging from complex interaction processes [ 5 , 30 ] . Moreover, existing studies that consider team dynamics are typically limited to predicting social signals, without providing actionable guidance to support performance improvement. Communication breakdowns, misaligned task allocation, and micro-coordination failures may accumulate and introduce temporal inefficiencies [ 25 ] that translate into increased postoperative risk [ 2 ] . A recent study on spinal surgery shows that 15% increase in operative time is associated with a 14% higher risk of complications [ 17 ] . Although not a direct measure of care quality, systematic deviations from the expected operative time provide a practical proxy for team performance [ 24 ] , with prolonged surgeries impacting both patient safety and costs, up to +160% per procedure [ 20 ] , and incurring substantial OR expenses, up to $37 per minute [ 3 ] . In this work, we frame operative time as a predictive and actionable signal of team performance. We model team interactions by jointly capturing temporal and relational dimensions from multimodal communication data. While temporal graph neural networks have demonstrated strong performance in modeling spatio-temporal data [ 14 ] , they are less suitable in small-data regimes such as surgical teams where the limited amount of data and the team size increase overfitting risk [ 15 ] . To address this challenge, we incorporate temporal dynamics directly into relational networks through the construction of time-expanded graphs. Finally, we introduce a counterfactual analysis framework to identify interpretable behavioral changes in team interactions associated with improved predicted outcomes. The main contributions of this paper can be summarized as follows: • A Time-Expanded Relational Neural Network ( TE-ReNN ) architecture that simultaneously models relations and temporal evolution for teams. • A two-level counterfactual algorithm to generate behavioral suggestions to enhance team performance. • An experimental evaluation on simulated surgical procedures showcasing the effectiveness of the proposed solution in both improving early identification of prolonged interventions and providing actionable counterfactuals. Figure 1 : Overview of the interaction modeling pipeline. Top row: multimodal time-series are segmented into fixed temporal windows (15 seconds). Middle row: for each window, a snapshot interaction graph G_{t} is built, in which nodes represent team members and edges encode broadcast verbal communication. Node features integrate interpretable paralinguistic (eGeMAPS), pose, and human–tool interaction. Bottom row: snapshot graphs are connected through identity-preserving temporal edges into a Time-Expanded Interaction Graph G^{TE} , enabling joint modeling of intra-team coordination and longitudinal individual dynamics across time. 2 Methodology Figure 1 illustrates the proposed pipeline of our approach, from multimodal feature extraction to the construction of the Time-Expanded Interaction Graph. The methodology is designed not only to capture team interaction dynamics but also to preserve an explicit link between model representations and modifiable human behaviors, a key requirement for Hybrid Human-AI collaboration. 2.1 Interaction Modeling Framework We model surgical team collaboration as a sequence of multimodal interaction graphs constructed over fixed temporal windows. The input consists of paralinguistic, motion, and human-object interaction time-series features extracted from audio-video recordings and annotations. Each modality is sampled at its own ad hoc rate. Time windows of 15 seconds are constructed from these time series. Let \mathcal{V} denote the set of team members in the OR. For each temporal window t , we build a snapshot interaction graph G_{t}=(V_{t},E_{t}) , where V_{t}\subseteq\mathcal{V} is the set of team members present in the OR during window t . Edges E_{t} encode verbal communication under a broadcast assumption. When a team member v_{i}\in V_{t} speaks during window t , we add edges (v_{i},v_{j}) for all v_{j}\in V_{t} , j\neq i . This representation models verbal communication as being directed to all present team members as a one-to-many interaction, rather than to a specific addressee. While simplified, this assumption reflects the shared acoustic environment of the OR, where speech is typically audible to all team members. Each node v\in V_{t} is associated with a multimodal feature vector integrating: • Paralinguistic features : loudness, alpha-ratio, and harmonics-to-noise ratio features from the eGeMAPS 2.0 set [ 6 ] . • Motion features : the position of each team member, and the average displacement and its standard deviation. • Human-tool interaction features : a triplet of values (team member-action-object) describing atomic actions in the OR (e.g., anesthesiologist-calibrating-instrument). These features are one-hot encoded. • Role features : the functional role of each team member (e.g., head surgeon, nurse). Roles’ features are one-hot encoded. 2.2 Behavioral State Representation To support interpretable and actionable suggestions in the form of counterfactual explanations, we focus on a subset of paralinguistic features whose meanings directly map to interpretable behavioral patterns. As previously mentioned, the paralinguistic features are the loudness, referring to the activation and engagement, the harmonics-to-noise ratio that measures vocal control and tension, and the alpha ratio accounting for the vocal dominance. Each combination of these three features is mapped to a discrete behavioral class. Activation Control Dominance Behavioral Class Absent Absent Absent Silent ∗ Low Low Low Withdrawn / Disengaged Low Low High Restrained Conflict Low High Low Calm–Cooperative Low High High Quiet Authority High Low Low Agitated / Over-aroused