Evaluation of our new three-dimensional navigation system in robot-assisted partial nephrectomy Authors: Toshihiro Magaribuchi, Masanao Koeda, Shinya Somiya, Akihiro Hamada, Atsushi Sengiku, Kimihiko Masui, Takashi Kobayashi, Atsuro Sawada Published: 2026 ABSTRACT AbstractTo evaluate the accuracy of a high-precision navigation system that projects a three-dimensional (3D) kidney model onto surgical images by limiting its application to moments of minimal organ deformation during partial nephrectomy (PN). We analyzed 29 patients who underwent PN at Kyoto University Hospital and Kobe City Medical Center General Hospital in Japan. 3D models of the kidney and tumor were generated using DICOM data, whereas 3D point clouds of the surgical field were obtained using stereo camera recordings. Noise reduction processing was applied to the camera-derived point clouds. Registration between the computed tomography-derived models and camera-derived point clouds was performed using the closest iterative point, and the accuracy was assessed using the root mean squared error. We evaluated the effects of the point-cloud surface area and camera-to-target distance on the registration accuracy. Without noise reduction, the median registration error was 2.33 mm, whereas noise reduction improved the accuracy by 1.83 mm. The accuracy was significantly higher when the camera-to-target distance was shorter, with and without noise reduction. The surface area was inversely correlated with the accuracy without noise reduction, but no significant correlation was observed with noise reduction. Focusing on moments with minimal organ deformation, we demonstrated that high-precision surgical navigation is achievable in PN using actual surgical recordings. This may contribute to improved tumor localization and the preservation of renal function.Supplementary InformationThe online version contains supplementary material available at https://doi.org/10.1038/s41598-026-57504-3.Subject terms: Cancer, Engineering, Medical research, Oncology Introduction Partial nephrectomy (PN) is the standard treatment for small renal cell carcinoma1. In PN, it is essential to completely resect the tumor while minimizing damage to the normal renal tissue. A decrease in normal glomeruli is associated with impaired kidney function2. Therefore, precise tumor localization ensures adequate resection while preserving healthy tissues. However, accurately locating tumors becomes challenging in cases where the tumor is deeply embedded within the kidney, increasing the surgical complexity3. Ultrasound imaging is the most commonly used method for intraoperative tumor localization. Three-dimensional (3D) navigation systems have been developed to provide accurate localization. These systems use preoperative computed tomography (CT) data to generate 3D models of the kidney and tumors to assist in tumor localization. Augmented reality (AR)-based navigation involves projecting CT-derived 3D kidney models onto the intraoperative field, offering surgeons an accessible and effective tool4. Ukimura et al. first applied AR navigation to PN in 2008 and manually aligned 3D kidney models with 2D surgical views to identify tumor locations5. Subsequent studies have reported similar techniques, and a 2022 systematic review highlighted the potential of AR navigation to reduce perioperative complications such as opening the collecting system6. Despite these advantages, AR-based navigation in PN faces significant challenges when adapting to intraoperative kidney movement and deformation. Accurate CT or MRI data registration to an organ is feasible when the target organ is fixed and minimally deformed7. However, in PN, surgical manipulation causes kidney movement and deformation, and registering CT data in the kidney is highly difficult. Although various approaches to address deformation have been studied experimentally, reports of practical applications in actual surgeries are limited to methods that manually align 3D kidney models with 2D surgical views6. To address this issue, we devised a navigation method that operates exclusively when kidney deformation is minimal. Consequently, the anatomical consistency between CT-derived and intraoperative kidney shapes improved. Additionally, projecting a 3D kidney model onto a 3D point-cloud view of the surgical field has become more feasible. This enables higher-precision navigation than the traditional approach of manually aligning a 3D kidney model with a 2D surgical view. In a previous study, we evaluated the accuracy of this navigation system using a 3D-printed kidney model in a laboratory setting. Our results indicate that the accuracy improved with closer camera-to-target distances, with registration discrepancies of less than 2 mm at distances within 7 cm8. However, a limitation of the previous study was the lack of real surgical footage. Surgical videos often include images of other organs, potentially narrowing the surface area of the navigable kidney. Additionally, intraoperative irregularities, such as small surface indentations or reflections from lighting, can affect navigation accuracy, making it necessary to evaluate these factors. In this study, we evaluated the accuracy of our AR-based navigation system using real surgical footage, and the results were favorable. Our navigation system is especially effective in helping to rapidly and accurately identify deeply embedded renal tumors that are difficult to localize, even with preoperative CT alone. This clinical utility is a key advantage of the proposed approach. Materials and methods Objectives and participants This study aims to evaluate the accuracy of an AR-based navigation system using real surgical footage. The primary endpoint was navigation accuracy, defined as the degree of alignment between the CT-derived kidney model and the intraoperative 3D representation. The accuracy evaluation method is described in Section "Navigation accuracy evaluation". We included 29 patients who underwent RAPN between February 2020 and May 2024 at Kyoto University Hospital and Kobe City Medical Center General Hospital. Patients with incomplete surgical footage, insufficient preoperative imaging, conversion to radical nephrectomy, or multifocal tumors were excluded. Both retroperitoneal and transperitoneal approaches were used in this study. Navigation was applied during the minimal kidney manipulation phases, regardless of the approach. AR-based navigation The procedure for our AR-based navigation is explained in four steps (A–D), as shown in Fig. 1. Steps of AR-based navigation workflow. (A) Creation of a 3D kidney model using CT data. (B) 3D reconstruction of the surgical field using stereo images. (C) Noise reduction from the 3D reconstruction generated in (B). (D) Accurate alignment of the CT-derived 3D model with the 3D point cloud from the surgical field. Creation of kidney model 3D models of the kidney and tumor were created using SYNAPSE VINCENT (Fujifilm) and were created using the preoperative CT DICOM data. 3D models of the urinary tract and vasculature, prone to intraoperative deformation, were not created. Creation of a 3D point cloud from the surgical field via stereo-matching The Da Vinci Xi surgical system has stereo cameras on the left and right sides, and these two images provide surgeons with 3D images of the surgical field. To generate a 3D point cloud, disparity images were obtained from the left and right stereo camera images. Stereo matching was performed using a semiglobal block-matching (SGBM) algorithm implemented in OpenCV9. SGBM identifies the points between the left and right images and calculates their disparity. The resulting disparity image represents the distance between corresponding points based on color and brightness. The following transformation was applied to generate a 3D point cloud from the disparity image: In this transformation, X, Y, and Z are the coordinates of the 3D point cloud; u and v are the coordinates in the disparity image; d is the disparity; cx and cy are the coordinates of the intersection between the optical axis and the imaging plane of the left camera; f is the focal length; and B is the baseline distance of the stereo camera. Only the kidney and tumor surfaces were used for registration, and regions other than the kidney and tumor surfaces were removed from the images used for stereo-matching (Fig. 2). The segmentation process was performed manually, requiring approximately 1–2 min per case. Segmentation of kidney and tumor surfaces. The top row shows the point cloud generated from the unprocessed stereo images of the surgical footage, including surrounding organs. After removing non-essential regions, the bottom row displays the point cloud, leaving only the kidney and tumor surfaces from the stereo images. This reduction process allowed for selective area adjustment. Additionally, for accuracy evaluation, the surface area of the generated point cloud and the minimum distance between the point cloud and the camera were measured. The surface area of the point cloud was calculated using the open-source software MeshLab. The minimal kidney manipulation phase was defined as the period during which the kidney remained in its anatomical position within the renal fossa without any direct contact or manipulation by surgical instruments, as visually confirmed by surgical footage. Noise reduction in the generated 3D point cloud Noise inevitably occurs in point clouds generated through stereo matching, particularly on surfaces perpendicular to the viewpoint of the camera and areas with strong light reflections. In real surgical footage, reflections in the wet intraperitoneal environment, surface irregularities from surrounding adipose tissue, and surgical manipulation reduce the accuracy of the registration due to noise. To address this problem, we implemented a noise reduction function. As noise tends to extend in the direction tangential to the camera, the normal direction of the point cloud was calculated from the brightness gradient, and points perpendicular to the camera were removed. Additionally, the residual noise in the surrounding area was removed by deleting one pixel around these points. This method effectively reduces noise while preserving the kidney and tumor morphology (Fig. 3). Comparison of 3D point cloud with and without noise reduction. The top row illustrates the captured images at the time of point cloud reading, where increased black areas in the noise-removed images indicate portions excluded from the reading. The bottom row represents the resulting 3D point clouds, demonstrating the effects of noise reduction on the generated data. In the bottom row, the noise-reduced point cloud (right) preserves the morphology of the kidney surface while effectively suppressing peripheral noise, in contrast to the point cloud without noise reduction (left). Registration of CT-derived and stereo camera-derived point clouds The registration between the kidney model point cloud derived from CT and the point cloud generated via stereo matching was performed using the Point-to-Plane Iterative Closest Point (ICP) algorithm10. The Point-to-Plane ICP algorithm minimizes the distance between two point clouds to achieve registration. The primary objective is determining the optimal rigid transformation (translation and rotation) to align the two point clouds. The algorithm proceeds as follows. For each point in the point cloud P, the closest point in the point cloud Q was identified. For each point in the point cloud P, the closest point in the point cloud Q was identified. A rigid transformation that minimizes the error function E(T) is calculated, where Pi​ and Qi​ are the corresponding points in point clouds P and Q, respectively, T is the rigid transformation matrix, ni is the normal vector obtained from the plane near Pi, and N is the number of corresponding points.2 A rigid transformation that minimizes the error function E(T) is calculated, where Pi​ and Qi​ are the corresponding points in point clouds P and Q, respectively, T is the rigid transformation matrix, ni is the normal vector obtained from the plane near Pi, and N is the number of corresponding points.2 The rigid transformation is applied to point cloud P. The rigid transformation is applied to point cloud P. Steps 1 to 3 are repeated until convergence is achieved. Steps 1 to 3 are repeated until convergence is achieved. While registration uses point clouds of the kidney and tumor surfaces, other point clouds can be displayed after registration to assess the positional relationship between the surgical environment and the CT-derived kidney model. This confirmed the tumor location within the current surgical footage (Fig. 4) (Supplementary Fig. 1). ICP-registered 3D model with surgical footage. (A) Surgical footage. (B) Kidney model created using CT. (C, D) Images registered via ICP, with the tumor highlighted in pink. C shows the surrounding organs displayed, while D shows them hidden. The organ-textured model is derived from surgical footage, while the model represented by the point cloud and the pink-colored tumor is the CT-derived kidney model. The display of surrounding organs can be toggled on or off, and the viewing position and direction can be adjusted arbitrarily, enabling visualization of the location of embedded tumors that are not visible in the surgical footage. Navigation accuracy evaluation The discrepancy between the two point clouds after registration was evaluated using the root mean squared error (RMSE), which served as the primary measure of navigation error and improved accuracy in this study. The RMSE was defined by the following equation11, and a lower RMSE indicated higher navigation accuracy. where N is the total number of corresponding points, R is a 3 × 3 rotation matrix, pi​ represents the coordinates of point i in the first point cloud, qi represents the coordinates of the corresponding point in the second point cloud, and t is the translation vector of the point cloud. A smaller RMSE indicates a higher registration accuracy. Additionally, the correlation between the registration accuracy and the minimum distance from the camera to the target and the surface area of the point cloud generated from the stereo images was assessed using linear regression analysis. The evaluations were conducted with and without noise reduction. Results The median navigation accuracy (in mm) was 2.33 (0.51–5.36) without noise reduction and 1.83 (0.548–2.93) with noise reduction, showing a significant improvement in accuracy with noise reduction (p < 0.001). The median distance between the camera and the target (in mm) was 46.73 (28.07–71.9) without noise reduction and 47.73 (28.74–72.35) with noise reduction, with no significant difference (p = 0.39). The median surface area (in mm2) of the point cloud was 1821.99 (494.45–4755.38) without noise reduction and 965.36 (330.68–2363.92) with noise reduction, showing a significant reduction in surface area due to noise reduction (p < 0.001) (Fig. 5A). Regarding the relationship between camera-to-target distance and registration accuracy, closer distances were associated with significantly higher accuracy with and without noise reduction (p = 0.015 and p = 0.0025, respectively) (Fig. 5B). Additionally, in the absence of noise reduction, smaller point-cloud surface areas generated using stereo matching were significantly associated with higher navigation accuracy (p < 0.001). However, no significant correlation was observed between the point cloud surface area and the navigation accuracy when noise reduction was applied (p = 0.29) (Fig. 5C). Comparison of navigation accuracy with and without noise reduction under different camera-object distances. (A) Box plots comparing navigation accuracy, distance, and point cloud surface area with and without noise reduction. Outliers are indicated by dots in the surface area box plot. (B) Scatter plot showing the relationship between camera-to-target distance and navigation accuracy. (C) Scatter plot showing the relationship between the surface area of camera-derived point clouds and navigation accuracy. No significant associations were observed between tumor characteristics (tumor size and location), patient demographics (age, sex, and BMI), or navigation accuracy (Supplementary Table 1). Discussion During surgery, limited or obstructed visibility of the target organ increases the risk of organ damage. To address this issue, surgical navigation systems have been developed and are now widely used, particularly in head and neck surgery and orthopedic surgery7. These applications benefit from the stability of the target organs, which undergo minimal deformation or movement during the procedure. Consequently, 3D anatomical structures derived from CT or MRI can be directly used for navigation. However, in surgical fields where organs deform or move due to intraoperative manipulation, the direct use of CT or MRI data has become challenging, and navigation systems have yet to achieve widespread clinical adoption. The standard navigation method for PN involves projecting a 3D kidney model created from preoperative CT onto a surgical view using AR6. While the kidney moves and deforms during surgery, most AR navigation systems reported in practice rely on nondeformable models, making them incapable of addressing organ deformation12. Research on deformable models includes finite element methods13 and techniques combining shape matching with cluster-based deformation to create non-rigid models14. Zhang et al. developed a real-time navigation system using deformable registration with Coherent Point Drift to adapt to intraoperative organ deformation and movement15. However, these approaches remain limited to experimental or animal studies, and no reports exist on their practical implementation in human surgeries. In real surgeries, the dynamic environment and ongoing changes make it technically challenging to accurately account for deformations. Consequently, the existing reports only describe the projection of rigid 3D kidney models onto flat 2D surgical views. In this study, we adopted a navigation method that operated during periods of minimal kidney deformation. This approach eliminates the need to address deformation, allowing navigation to account only for other 3D changes, such as movement and rotation. Consequently, 3D kidney models can be projected onto 3D surgical views, providing more precise guidance than AR systems that project 3D models onto 2D views. This method is particularly effective because there are moments during PN when kidney deformation is minimal. Stereo matching was used to reconstruct the surgical views. Stereo matching uses the disparity between images captured using stereo cameras to reconstruct a 3D space, similar to how humans perceive depth through binocular vision. While methods using monocular cameras, such as structure from motion, have been reported16, stereo cameras in robotic endoscopic surgeries make stereo matching a reasonable choice for 3D reconstruction of surgical footage. Among the stereo-matching methods, we selected Semi-Global Block Matching (SGBM)9. SGBM reduces computational costs compared with global matching while producing smoother disparity maps with less noise than block matching, making it highly suitable for surgical navigation, where lightweight computation and accuracy are essential. We used the closest iterative point to align the reconstructed 3D surgical views with the 3D kidney model and Iterative Closest Point (ICP). Developed by Besl and McKay in 1992, the ICP iteratively minimizes the distance between corresponding points in two-point clouds17. This method enables accurate registration and is effective for rigid models such as those used in this study. Although stereo matching and ICP are not new techniques, their low computational burden makes them easy to implement without requiring high-performance PCs or additional equipment. This simplicity renders these methods suitable for clinical use. A comparison of this navigation system with existing approaches is represented in Supplementary Table 2. However, a direct quantitative comparison with conventional manual 2D-AR projection techniques remains challenging. Most previously reported approaches rely on subjective visual alignment and do not provide objective accuracy metrics such as RMSE or TRE. Therefore, an acceptable error margin has not yet been established for these conventional techniques. In this context, the present study contributes not only to a practical navigation approach but also to a quantitative evaluation framework for AR-based navigation accuracy using real surgical data. The reported registration error of approximately 2 mm should be interpreted as an objective reference value rather than a direct comparison with previous methods. Noise commonly arises when generating 3D point clouds from real-world data, and there is no perfect noise reduction method, although many approaches have been devised18. In this study, we applied a method that removed points perpendicular to the camera based on the normal direction of the point cloud and deleted one additional pixel around the removed area. This relatively simple approach is computationally efficient and provides satisfactory results. We observed a significant improvement in navigation accuracy after noise reduction. In this study, the median navigation accuracy was approximately 2.33 mm without stereo noise reduction and approximately 1.83 mm with noise reduction. Considering that tumors < 2 cm in size are rare in patients receiving PN19, an error margin of approximately 2 mm is sufficient for tumor localization. The accuracy improved with closer camera-to-target distances due to the higher point cloud accuracy generated through stereo matching at shorter distances. In our previous study, the camera of the Da Vinci system was limited to a minimum stereo-matching distance of 3 cm8, and the minimum camera-to-target distance in this study was 2.8 cm. Additionally, smaller point-cloud surface areas were associated with higher navigation accuracy without noise reduction, likely reflecting the impact of stereo noise on increasing the surface area. With noise reduction, no significant correlation was observed between the surface area and navigation accuracy, suggesting that navigation remains feasible even when the visible kidney surface is relatively limited during actual surgeries. Our cohort consisted of consecutive, nonselective patients receiving RAPN performed at two academic centers. Patient demographics and tumor characteristics, summarized in Supplementary Table 1, were comparable to those reported in a large multicenter study of 1,222 patients receiving RAPN20. This suggests that although our sample size was limited, it reflects a typical clinical population undergoing RAPN and supports the generalizability of our findings. This study has certain limitations. First, we evaluated the accuracy of the navigation using RMSE. Although RMSE is a widely used metric, it does not guarantee true correspondence between point pairs; therefore, it is not a perfect measure of registration accuracy. Nevertheless, the RMSE remains an appropriate and realistic evaluation method in this context, given the non-invasive nature of using surgical video footage. The use of fiducial markers or anatomical landmarks for more precise validation was not feasible in this study owing to its retrospective design and the limitations of rigid registration. Additionally, the initial alignment was performed manually, which may have introduced operator-dependent variability. Furthermore, the use of a rigid registration model does not account for potential intraoperative organ deformation, which may affect the navigation accuracy. The absence of vascular structure integration may limit the clinical applicability of the system, because vascular identification is essential in partial nephrectomy. Second, this was not a clinically validated study. Consequently, clinical outcomes such as surgical margin status, complication rates, and potential clinical challenges were not assessed, including the surgeon’s learning curve, system latency, and integration into robotic surgical platforms. Moreover, the current processing pipeline requires several minutes to complete, limiting its applicability to real-time intraoperative navigation. Real-time intraoperative validation of the navigation accuracy was not conducted because this study was based on a retrospective analysis of recorded surgical videos. In addition, although our cohort reflected a typical clinical population undergoing RAPN, the number of patients analyzed was relatively small (n = 29), which may have limited the statistical generalizability of the findings. Additionally, tumor complexity (e.g., RENAL score) was not evaluated in this study, and its potential impact on navigation accuracy remains unclear. Finally, there is a potential risk of misleading guidance if navigation is applied during unrecognized organ deformation, and additional intraoperative validation methods, such as ultrasound, are required. We plan to conduct large-scale clinical trials to address these issues. Furthermore, whether this navigation system can be integrated with real-time intraoperative imaging techniques (e.g., intraoperative MRI and fluoroscopy) remains to be evaluated. Conclusion We evaluated the accuracy of AR-based navigation for PN using real surgical footage and CT images. The error margin was approximately 2 mm, demonstrating that highly accurate navigation was achievable. Supplementary Information Below is the link to the electronic supplementary material. Supplementary Material 1 Supplementary Material 2 Supplementary Material 3 Acknowledgements We would like to express our sincere gratitude to the urology staff at Kobe City Medical Center General Hospital for their invaluable support in obtaining the image data used in this study. Author contributions T.M. and A.S. conceived the concept and study design. Data collection was performed by T.M., S.S., and A.H. Data analysis was conducted by T.M. Coordination of the study and guidance were provided by A. Sawada, A. Sengiku, and K.M. Technical support for the 3D navigation system was provided by M.K. Supervision was provided by T.K. and A. Sawada. Funding This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. Data availability The datasets generated and/or analyzed during the current study are not publicly available due to patient privacy restrictions, but de-identified data are available from the corresponding author upon reasonable request. Declarations Competing interests The author(s) declare no competing interests. Ethics approval The study was conducted under the approval of the Institutional Review Board of Kyoto University Hospital (Approval No. R1581: Observational study of urological diseases at Kyoto University Hospital and affiliated hospitals). Patient clinical data, including CT images, were used with IRB approval. The requirement for informed consent was formally waived by the Institutional Review Board of Kyoto University Hospital (Approval No. R1581) because this study used de-identified retrospective data. All methods were performed in accordance with the relevant guidelines and regulations, including the Declaration of Helsinki. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Contributor Information Takashi Kobayashi, Email: selecao@kuhp.kyoto-u.ac.jp. Atsuro Sawada, Email: atsurou_sawada@med.miyazaki-u.ac.jp. References Associated Data Supplementary Materials Supplementary Material 1 Supplementary Material 2 Supplementary Material 3 Data Availability Statement The datasets generated and/or analyzed during the current study are not publicly available due to patient privacy restrictions, but de-identified data are available from the corresponding author upon reasonable request.