Intelligent Virtual Sonographer (IVS): Enhancing Physician-Robot-Patient Communication

Tianyu Song ∗, Feng Li ∗, Yuan Bi, Angelos Karlas, Amir Yousefi, Daniela Branzan, Zhongliang Jiang, Ulrich Eck, Nassir Navab · 2025

Read Intelligent Virtual Sonographer (IVS): Enhancing Physician-Robot-Patient Communication by Tianyu Song ∗, Feng Li ∗, Yuan Bi, Angelos Karlas, Amir Yousefi, Daniela Branzan, Zhongliang Jiang, Ulrich Eck, Nassir Navab (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-13052.txt.

Introduction

Tianyu Song and co-authors give a robotic ultrasound system a virtual colleague who talks to everyone in the room, in work from 2025. Robotic scanning research usually studies either the doctor's control or the patient's comfort, not the conversation among all three parties.

The Intelligent Virtual Sonographer is an extended-reality agent powered by a language model, with speech recognition and synthesis. It speaks professionally to physicians, explains and reassures patients in plainer words, and relays the doctor's commands to the robot while narrating what the robot does. The authors present the system as a first step toward trust and clarity in robotic ultrasound, where patients accept the machine more readily when it explains itself.

Catalog record

CREATORTianyu Song ∗, Feng Li ∗, Yuan Bi, Angelos Karlas, Amir Yousefi, Daniela Branzan, Zhongliang Jiang, Ulrich Eck, Nassir Navab
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-13052

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Intelligent Virtual Sonographer (IVS): Enhancing Physician-Robot-Patient Communication Tianyu Song ∗, Feng Li ∗, Yuan Bi, Angelos Karlas, Amir Yousefi, Daniela Branzan, Zhongliang Jiang, Ulrich Eck, Nassir Navab Abstract The advancement and maturity of large language models (LLMs) and robotics have unlocked vast potential for human-computer interaction, particularly in the field of robotic ultrasound. While existing research primarily focuses on either patient-robot or physician-robot interaction, the role of an intelligent virtual sonographer (IVS) bridging physician-robot-patient communication remains underexplored. This work introduces a conversational virtual agent in Extended Reality (XR) that facilitates real-time interaction between physicians, a robotic ultrasound system(RUS), and patients. The IVS agent communicates with physicians in a professional manner while offering empathetic explanations and reassurance to patients. Furthermore, it actively controls the RUS by executing physician commands and transparently relays these actions to the patient. By integrating LLM-powered dialogue with speech-to-text, text-to-speech, and robotic control, our system enhances the efficiency, clarity, and accessibility of robotic ultrasound acquisition. This work constitutes a first step toward understanding how IVS can bridge communication gaps in physician-robot-patient interaction, providing more control and therefore trust into physician-robot interaction while improving patient experience and acceptance of robotic ultrasound. The code is available at https://github.com/stytim/IVS $*$ $*$ footnotetext: These authors contributed equally to this work. 1 Introduction Advancements in robotic ultrasound systems (RUSs) [ 17 , 3 , 15 ] and telemedicine [ 23 , 9 ] are transforming medical diagnostics by enabling automated and remote procedures. These technologies enhance accessibility to ultrasound imaging, particularly in rural or underserved areas, where trained sonographers may be unavailable. However, the integration of robotic ultrasound systems into clinical practice presents new challenges in communication, decision-making, and user acceptance [ 10 ] . A crucial aspect of this integration is the interaction between physicians, robotic systems, and patients —a communication dynamic that is often overlooked in existing research. While prior studies have explored virtual agents for either patient engagement [ 5 , 27 , 24 ] or physician assistance [ 19 , 18 ] , there remains a significant research gap in designing intelligent agents introduced in this work that facilitate communication among all three entities: the physician, the RUS, and the patient. This gap is critical, as effective coordination among these stakeholders is essential for diagnostic efficiency, physician trust, and patient acceptance. Without a structured communication framework, physicians may struggle to effectively convey instructions to the robotic system, patients may have difficulty understanding the physician’s intent, and physicians may misinterpret patient concerns. This lack of seamless multidirectional communication can lead to errors in robotic control, patient uncertainty, and decreased trust in the procedure. Recent research in Embodied Conversational Agents (ECAs) [ 6 , 7 ] has shown that virtual avatars with human-like behaviors, facial expressions, and speech synchronization enhance trust, engagement, and communication effectiveness, particularly in healthcare applications. ECAs have been used to guide patients through medical procedures [ 4 , 26 ] and support physician decision-making [ 21 ] , demonstrating their potential in clinical interactions. However, existing ECAs typically focus on single-user interactions, either with the patient or the physician, rather than facilitating real-time multidirectional communication between multiple stakeholders and a robotic system. Building on this foundation, we introduce an Intelligent Virtual Sonographer (IVS) designed to bridge the interaction gap between the physician, the RUS, and the patient. This agent operates in an Extended Reality (XR) environment, offering assistance and procedural control for the physician, and providing empathetic guidance and reassurance for patients. The agent facilitates bidirectional information exchange, ensuring that physicians receive real-time updates on patient conditions while patients remain informed about the robotic ultrasound procedure. Additionally, the agent enhances transparency and user trust by explicitly communicating the robotic system’s actions to both parties. Our approach leverages recent advancements in large language models (LLMs) to enable real-time, natural language interaction. The IVS facilitates multidirectional communication by translating physician instructions into robotic ultrasound commands and relaying system updates back to the physician. It allows patients to request pauses or pressure adjustments while keeping them informed about the procedure. Additionally, it ensures effective physician-patient communication by relaying patient concerns to the physician and conveying the physician’s intent to the patient. In this paper, we present the design, implementation, and evaluation of IVS. We assess its impact in bridging physician-robot-patient interactions and analyze its usability, communication effectiveness, perceived intelligence, and overall user satisfaction. Additionally, our system runs entirely locally to address medical privacy concerns, utilizing a relatively small LLM that operates efficiently without requiring a data center, yet still achieves strong performance. Our findings contribute to the broader field of LLM-driven AI agents in medical applications, particularly in telemedicine and robotic ultrasound-assisted diagnostics. 2 Method Our system implements an XR-driven robotic ultrasound workflow, as illustrated in Fig. 1 , allowing both the physician and patient to interact with virtual sonographers. The RUS autonomously plans and performs ultrasound scanning by extracting intent from natural conversations. The following sections provide a detailed overview of the RUS architecture and the IVS. Figure 1: System overview. (a) IVS interacting with the physician in XR. (b)-(d) Automatic trajectory generation visualized in XR. (e) RUS hardware setup with tracking camera, ultrasound probe, and robot. (f) Real-time vessel segmentation on the ultrasound image. (g) IVS interacting with the patient. The straight arrow represents the data transmission flow, the curvy arrow shows the interaction. 2.1 Robotic Ultrasound System To enable automatic scanning, the robotic ultrasound system is designed with a KUKA LBR iiwa 14 R820 robotic arm and a Siemens ACUSON Juniper ultrasound machine. The Siemens 12L3 ultrasound probe is rigidly attached to the robot’s end-effector using a 3D-printed probe holder. Additionally, an Intel Realsense D435i depth camera is mounted between the probe and the robot end-effector (Fig. 1 (e)) to capture the 3D point cloud from the patient side, which is then reprojected for the physician’s visualization. The hand-eye calibration [ 13 ] is used to transform the camera pose into the robot end-effector pose. Before scanning, the physician can instruct the IVS system to display the patient’s arm and the planned scanning trajectory. During this process, one RGB image along with ten consecutive depth images are acquired. To generate the scanning path, we employ the MediaPipe [ 20 ] framework to extract specific body features, including skin segmentation and hand landmark detection. First, the arm mask is obtained using skin segmentation with a trained model, SelfieMulticlass (Fig. 1 (b)). To better determine the scanning direction, HandLandmarker model is applied to identify the wrist point pose. Additionally, the arm mask’s skeleton is computed, starting from the wrist and extending to the widest part of the forearm. While these steps rely on the RGB image, generating the full scanning path requires 3D point information, which is extracted from the depth images. We capture ten depth images, extract the depth values at each selected path point, and apply a median filter to refine the depth estimation. Once the origins of each scanning point (t_{x},\ t_{y},\ t_{z}) are determined, 16 neighboring points for each selected point along the path are acquired and a plane is fitted to calculate the normal vector. This step ensures that the robot’s end effector maintains the smallest possible angle with respect to the normal

Read the complete text