Probabilistic Task Parameterization of Tool-Tissue Interaction via Sparse Landmarks Tracking in Robotic Surgery

Yiting Wang, Yunxin Fan, Fei Liu · 2025

Read Probabilistic Task Parameterization of Tool-Tissue Interaction via Sparse Landmarks Tracking in Robotic Surgery by Yiting Wang, Yunxin Fan, Fei Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-11495.txt.

Introduction

Wang, Fan, and Liu wrote in twenty twenty-five about modeling how surgical tools interact with tissue by tracking sparse landmarks. Robotic surgery needs accurate models of soft tissue deformation. Hand labeling every frame is slow, and rigid assumptions fail on real tissue. Their framework propagates expert-marked keypoints across endoscopic video frames. It copes with large tissue deformations during procedures. Clustered keypoints build local reference frames that move with the tissue.

Tool poses are tracked the same way and expressed in those frames. A task-parameterized Gaussian mixture model fuses data with clinical expertise. The model predicts relative tool-tissue poses for surgical motions. Understanding comes directly from video rather than extra sensors. The authors aim for better visual understanding of robotic operations. That could support training, automation, and safety checks in surgery.

Catalog record

CREATORYiting Wang, Yunxin Fan, Fei Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE9 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-11495

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Probabilistic Task Parameterization of Tool-Tissue Interaction via Sparse Landmarks Tracking in Robotic Surgery Thanks: * Denotes equal contribution. † \dagger Denotes the corresponding author. Thanks: 1 Department of Electrical Engineering and Computer Science, University of Tennessee Knoxville, Knoxville, TN 37996, USA [email protected], [email protected] . Thanks: 2 Department of Mechanical Engineering, Stanford University, Stanford, CA 94305, USA [email protected] .
Yiting Wang, Yunxin Fan, Fei Liu

Abstract
Accurate modeling of tool-tissue interactions in robotic surgery requires precise tracking of deformable tissues and integration of surgical domain knowledge. Traditional methods rely on labor-intensive annotations or rigid assumptions, limiting flexibility. We propose a framework combining sparse keypoint tracking and probabilistic modeling that propagates expert-annotated landmarks across endoscopic frames, even with large tissue deformations. Clustered tissue keypoints enable dynamic local transformation construction via PCA, and tool poses, tracked similarly, are expressed relative to these frames. Embedding these into a Task-Parameterized Gaussian Mixture Model (TP-GMM) integrates data-driven observations with labeled clinical expertise, effectively predicting relative tool-tissue poses and enhancing visual understanding of robotic surgical motions directly from video data.



I
Introduction

Robotic-assisted minimally invasive surgery (RMIS) offers improved precision and reduced human error, yet tool-tissue interaction remains a core challenge due to tissue deformability, anatomical variability, and strict safety constraints. Traditional methods often depend on dense manual annotations or rigid kinematic assumptions, limiting scalability and generalization across surgical contexts. Recent learning-based approaches have improved tissue tracking, but typically overlook the integration of domain-specific surgical priors—crucial for interpretability and robustness in clinical settings.

To address this, we propose a hybrid framework that combines sparse keypoint tracking with probabilistic task parameterization. Using LocoTrack
[
1
]
, we manually annotate and track sparse anatomical landmarks (e.g., tissue regions, tool tips) across video frames. These keypoints are clustered to generate dynamic tissue frames via PCA, capturing local deformation. Tool poses are expressed relative to these frames and modeled using a Task-Parameterized Gaussian Mixture Model (TP-GMM), which integrates both time and tissue context. Our formulation enables uncertainty-aware, context-sensitive trajectory prediction, incorporating surgical priors such as motion constraints and safety zones.

Fig. 1
:
framework diagram

II
Method

This section outlines our approach for modeling tool-tissue interactions using a time-conditioned TP-GMM, applied to the EndoNeRF dataset
[
2
]
. We learn a model of tool motion in SE(2), capturing 2D position and orientation directly from endoscopic video. The method relies on sparse anatomical keypoints and dynamically derived local tissue frames. The overall pipeline is summarized in Algorithm
1
, enabling smooth and context-aware motion prediction conditioned on surgical tool dynamics and tissue geometry.

II-A

Framework Overview

We model tool–tissue interactions using 2D keypoints tracked from endoscopic video via LocoTrack. Guided by surgical domain knowledge, we cluster tissue landmarks into semantically meaningful regions (e.g., tissue layers or boundaries), with the number and structure of clusters reflecting anatomical priors. Local tissue frames are then derived from these regions using PCA, and tool poses are expressed relative to the resulting dynamic reference frames. A task-parameterized Gaussian Mixture Model (TP-GMM) is trained on these frame-relative trajectories, enabling probabilistic modeling that captures spatiotemporal variability, uncertainty, and multi-modality in tool motion.

II-B

Tissue-Relative Frame Representation

A key challenge in surgical motion modeling lies in the nonrigid, time-varying nature of tissue. To address this, we construct local reference frames
\mathbf{T}_{\text{ref}}(t)
from sparse tool and tissue landmarks, as detailed in Algorithm
1
. Tool poses
{c(t),q(t)}
are transformed into this frame, yielding a deformation-invariant representation. The resulting time- and tissue-conditioned trajectories are used to train a TP-GMM for smooth and context-aware motion prediction.

Algorithm 1

Tool-Tissue Interaction Modeling via Time-Conditioned TP-GMM

1:

Input:
Tracked tool landmarks
\mathcal{X}_{\text{tool}}(t)
and tissue landmarks
\mathcal{X}_{\text{tissue}}(t)
for
t=1,\ldots,T

2:

Output:
Predicted tool poses
\hat{x}_{\text{tool}}(t)
for
t\in[0,1]

3:

for

t=1
to
T

do

4:

\{\mathcal{C}_{i}(t)\}\leftarrow\mathtt{DomainCluster}(\mathcal{X}_{\text{tissue}}(t))
,
i=1,\ldots,k

5:

\left(\mu_{i}(t),\Sigma_{i}(t)\right)\leftarrow\mathtt{computeGaussian}(\mathcal{C}_{i}(t))

6:

c_{\text{ref}}(t)\leftarrow 1/k\sum\mu_{i}(t)

7:

\mathbf{R}_{\text{ref}}(t)\leftarrow\mathtt{PCA}(\{\mu_{i}(t)\})

8:

\mathbf{T}_{\text{ref}}(t)\leftarrow\begin{bmatrix}\mathbf{R}_{\text{ref}}(t)&c_{\text{ref}}(t)\\
\mathbf{0}^{\top}&1\end{bmatrix}

\triangleright

Reference frame

9:

\left\{\mathbf{p}_{\text{rel}}(t),\ \mathbf{q}_{\text{rel}}(t)\right\}\leftarrow\left\{\mathbf{T}_{\text{ref}}^{-1}(t)\cdot\begin{bmatrix}c(t)\\
1\end{bmatrix},\ q_{\text{ref}}^{-1}(t)\cdot q(t)\right\}

10:

\mathcal{P}(t)\leftarrow\left[t,\mathbf{p}_{\text{rel}}(t),\mathbf{q}_{\text{rel}}(t)\right]

11:

end

for

12:

\Theta\leftarrow\mathtt{TP\_GMM\_Train}(\mathcal{P}^{\top})

\triangleright

Train with EM

13:

for

t\in[0,1]

do

14:

\hat{x}(t)\leftarrow\mathtt{GMR}(\Theta,t)

\triangleright

Predict position

15:

\hat{q}(t)\leftarrow\mathtt{SLERP}(\Theta,t)

\triangleright

Interpolate orientation

16:

end

for

17:

return

\{\hat{x}_{\text{tool}}(t)\}=\{[\hat{x}(t),\hat{q}(t)]\}

II-C

Learning with TP-GMM and Prediction

To model temporal tool motion, we train a time-conditioned TP-GMM on frame-relative trajectories
\mathcal{P}(t)
via Expectation Maximization. Each component is conditioned on time and tissue context, enabling the model to capture uncertainty and multimodal patterns in surgical demonstrations. In the inference stage, the tool positions
\hat{x}(t)
are predicted using Gaussian mixture regression (GMR), and the orientations
\hat{q}(t)
are computed using spherical linear interpolation (SLERP), ensuring smooth and consistent pose predictions. The resulting trajectories support the execution of tasks that adapt to robotic cutting or suturing.

III
Results

We evaluate the proposed time-conditioned TP-GMM on the EndoNeRF dataset for a 2D cutting task. The model is trained on 128 frames and tested on the remaining 28 frames. Performance is quantified by Euclidean distance between predicted and ground-truth positions. Orientation is assessed using angular errors. The model achieves smooth and visually consistent pose predictions over time.

Fig. 2
:
Comparison of ground truth (red) and predicted (blue) tool-tip trajectories across three frames.

III-A

Quantitative Evaluation

We evaluated TP-GMM performance using Euclidean distance for position and angular difference for orientation. These metrics reflect how well the model fits the training data and generalizes to unseen deformations.

III-A
1
Position Error

The accuracy of the tool position is evaluated using the Euclidean distance between the predicted and ground-truth 2D pixelwise positions (in pixel). The mean errors for the training and test set are 12.97 px and 67.69 px, respectively. Notice that the larger testing error indicates overfitting and limited generalization to unseen tissue motions.

III-A
2
Angle Error

Orientation is evaluated using the angular error (in degrees), calculated as the absolute difference between the predicted and ground-truth tool angles. This metric is better suited than quaternion-based metrics in our 2D planar setup. The mean angular errors for the training and test set are 1.93° and 5.48°, respectively. This trend mirrors the position error, showing strong training performance but reduced accuracy on the test set.

Figure
3
presents angle error curves in normalized time (0–1) for both training and testing sets, highlighting how temporal dynamics affects prediction accuracy.

Fig. 3
:
Angular error for training (a) and testing (b)

IV
Discussion

The results highlight both strengths and limitations of the TP-GMM framework. Low training error (12.97 px) confirms its ability to model tool–tissue interactions, with GMR and SLERP ensuring smooth position and orientation predictions. However, high test error (67.69 px) and a 5.2× test/train ratio indicate overfitting, likely due to limited data diversity and over-parameterization from adaptive mixture components (
N=10\sim 30
).

Read the complete text