Interactive Holographic Visualization for 3D Facial Avatar *Tri Tung Nguyen NGUYEN (Ritsumeikan University), Yasuyuki FUJII (Ritsumeikan University), Dinh Tuan TRAN (Ritsumeikan University), Joo-Ho LEE (Ritsumeikan University) Abstract Traditional methods for visualizing dynamic human expressions, particularly in medical training, often rely on flat-screen displays or static mannequins, which have proven inefficient for realistic simulation. In response, we propose a platform that leverages a 3D interactive facial avatar capable of displaying non-verbal feedback, including pain signals. This avatar is projected onto a stereoscopic, view-dependent 3D display, offering a more immersive and realistic simulated patient experience for pain assessment practice. However, there is no existing solution that dynamically predicts and projects interactive 3D facial avatars in real-time. To overcome this, we emphasize the need for a 3D display projection system that can project the facial avatar holographically, allowing users to interact with the avatar from any viewpoint. By incorporating 3D Gaussian Splatting (3DGS) and real-time view-dependent calibration, we significantly improve the training environment for accurate pain recognition and assessment. 1. Introduction Fig. 1: 3D Holographic Avatar Projection Overview . A novel pipeline combining 3D Gaussian Splatting and light-field shading enables real-time, photo-realistic facial avatar hologram animations, streamed to the 3D Looking Glass display. The OpenGL-based light-field shading program uses FLAME expression parameters with its 3D Gaussian primitive correspondence and the display calibration parameters to render a multi-view quilt that adapts to the audience’s viewpoint. Pain assessment is a key skill for healthcare professionals, essential for effective pain management and improving patient outcomes. Traditional training methods using mannequins or classroom materials often fail to reflect real clinical practice, leaving gaps in students’ knowledge and skills. Simulated patients provide a more realistic training environment, closely mimicking real-world scenarios. Unlike VR or MR systems, which are limited to individual users, 3D holographic displays allow multiple users to view the same hologram from different angles without headsets. This facilitates group learning and collaborative pain assessment, making it ideal for training. We focus on advanced holographic displays like the Looking Glass, which provide true 3D visualization through volumetric projections. When combined with 3D Gaussian Splatting viewers, these technologies offer flexible, immersive pain recognition training by enabling realistic facial avatars to be observed from various perspectives. Our contributions include: 1. We present a proof-of-concept pipeline that connects multimodal input through deep generative modeling to render non-verbal facial expressions via a 3D holographic display. 2. We propose solutions to address the challenges inherent in using stereoscopic and holographic displays as simulated patients for pain assessment, focusing on dynamic realism, visual comfort, and interaction fluidity. 2. Related Work 2.1 Simulated Patient Unlike classroom materials, which are often theoretical and abstract, the simulated patient approach - having professional certified actors roleplay as a patient in various standardized scenarios, allows trainees to engage with lifelike cases, enhancing their ability to recognize and assess pain in real patients [ 4 ] . However, employing human actors requires significant resources to compensate, which leads to the rise of virtual simulated patient development. Mixed reality solutions arise as a cheaper and more sustainable solution in which a virtual avatar is programmed with standardized scenarios for training purposes. Most recent notable mixed reality platforms for nursing training, such as GigXR’s Holopatient [ 2 ] or SimX’s VR [ 6 ] , employ VR headsets as the main medium to project the patient avatar to users. However, due to current technology limitations, VR headset is reported to hinder the training experience: most VR head-mounted display supports only single users, and VR glasses also cause discomfort, isolating feelings with extended use. On the other hand, a multi-view 3D display can naturally support group settings while maintaining realistic interaction between the training and the patient avatar. 3. System Design 3.1 Generative Facial Feedback As the pain-related dataset for simulated patients does not exist, for the first step, we revise the task into a more generalized non-verbal communication behavior endowment for robotic agents. For facial expression, we utilized the FLAME feature [ 3 ] that represents pose and expression facial features. Pain intensity future may be added into the latent space to adjust facial expression in the future. Given multi-modal stimulation inputs consist of speaker’s audio A_{s}(t) , facial features V_{s}(t) until time t along with listener’s past facial features V_{l}(t-k) until time (t-k) , our model predicts k -frames feedback facial features \hat{V}(t,k) . We chose 1D convolutional blocks to extract temporal features from modality inputs F_{a},F_{v},F_{l} . \displaystyle F_{a}(t) \displaystyle=F_{a}(A_{s}(t)) \displaystyle F_{v}(t) \displaystyle=F_{v}(V_{s}(t)) \displaystyle F_{l}(t-k) \displaystyle=F_{l}(V_{l}(t-k)) The objective is to find the generative model represented by G that predicts \hat{V} output: \displaystyle\hat{V}_{l}(t) \displaystyle=G\big(F_{a}(t),F_{v}(t),F_{l}(t-k)\big) \displaystyle H^{l}_{d} \displaystyle=DecoderLayer^{l}(H^{l-1}_{d}),H^{0}_{d}=X_{d}(0) For this study, we employed the Transformer architecture’s Encoder-Decoder modules to decode facial expression feedback from multi-modal encoded embedding. By applying linear transformation Z as the Decoder’s last layer, we transformed the decoded embedding into logits for action tokens output: Z(t)=W_{out}H^{l}_{d} For every time step in the prediction time window, the highest probability token class was elected as the output: \hat{y}(t)=\text{argmax}\hskip 2.84544pt{\mathbf{P}(t)} . The key challenge in facial feedback prediction is fusing multiple modalities (i.e., speaker’s audio/visual and listener’s visual) into one context embedding. Given two input modalities X and Y\in\mathbb{R}^{T\times D} , by concatenation along feature dimension D we have: F_{concat}(t)=\text{concat}(X(t),Y(t)) Fig. 2: Holographic rendering pipeline overview. Our system consists of a blend-shape model for the 3D Gaussian Splatting Avatar with a FLAME prior. Given an expression parameter \psi_{m} at frame \#m , the pipeline linearly blends pre-trained blend shapes {B_{K}} into an accumulated B^{*}_{m} primitive parameters. An iterative procedure rasterizes this 3D facial avatar from various viewpoints to reproduce a light field given an input viewpoint. The final output is a shaded quilt comprised by multiple discrete views that visualizes the expected 3D hologram on the target 3D display. By transforming the concatenated features into learned query, key, and value matrix W_{q} , W_{k} , and W_{v} , attention weight can be computed to add create cross-relationship to each modality’s time step, creating fused embedding C(t) : C(t)=\text{softmax}\big(\frac{QK^{T}}{\sqrt{d_{k}}}\big)V Algorithm 1 Calibrate Multi-View 3D Avatar 0: Virtual camera parameters camSize , aspect ratio ar , field-of-view fov(.) , viewer’s distance d , base view matrix V 0: Multi-view quilt views 1: Initialize empty array views [ ] 2: total\leftarrow 48 3: fov_{d}\leftarrow fov (d) 4: d_{cam}\leftarrow\frac{-camSize}{\tan(fov_{d}/2)} 5: for i\leftarrow 0 to (total-1) do 6: \alpha_{off}\leftarrow alpha_{max}*(\frac{i}{total-1}-0.5) 7: t_{off}\leftarrow d_{cam}*\tan(alpha) 8: 9: # Compute view matrix 10: V^{\prime}\leftarrow Translate( V,t_{off},camDist ) 11: 12: # Projection matrix for parallel projection. 13: P\leftarrow Perspective( fov,ar,z_{near},z_{far} ) 14: P[2][0]\leftarrow P[2][0]+t_{off}/(camSize*ar) 15: 16: # Render 2D portrait 17: v\leftarrow render( V^{\prime} , P ) 18: Append v to views [ ] 19: end for 20: 21: return views [ ] Finally, by splitting feature dimension now (D_{X}+D_{Y}) by D_{X} we had a new attention-aware embedding of X fused by Y . Fig. 3: 3D FLAME output features could be used to reconstruct 3D facial mesh model that works as prior for photorealistic 3D Gaussian Splatting facial avatar. Table 1: Baselines. Comparison of our approach with baseline models [ 7 , 1 ] on the test set. Appropriateness Diversity Realism Synchrony FRCorr ( \uparrow ) FRDist ( \downarrow ) FRDiv ( \uparrow ) FRVar ( \uparrow ) FRRea ( \downarrow ) FRSyn ( \cdot ) Ground truth 8.73 0.00 0.0000 0.0724 - 47.69 Random 0.05 237.23 0.1667 0.0833 - 44.10 Mime 0.38 92.94 0.0000 0.0724 - 38.54 MeanFr 0.00 97.86 0.0000 0.0000 - 49.00 Trans-VAE [ 7 ] 0.07 90.31 0.0064 0.0012 69.19 44.65 BeLFusion [ 7 ] 0.12 94.09 0.0379 0.0248 94.09 49.00 FSQ-Tformer [ 1 ] 0.31 84.93 0.1164 0.0348 34.66 47.42 Ours 0.52 86.23 0.1212 0.0442 25.78 48.57 ( \cdot ) means the closer to the ground truth, the better. indicates the best average performance among the heuristic baselines for the groups of metrics. 3.2 3D Facial Avatar Rendering In current facial avatar work, 2D rendering from generative model outputs is common. However, our research explores converting these outputs into 3D facial avatars compatible with multi-view stereoscopic displays like the Looking Glass Portrait (LGP). 3.2.1 3D Gaussian Splatting Avatar Inspired by recent work on 3D Gaussian Splatting Avatar [ 5 ] , we adapt this implementation into our current task to render expression transition from our predictor in Section 3.1 . The main advantages of 3D GaussianAvatar include its extremely fast rendering speed ( \geq 60 fps, tested on Nvidia RTX 4060 GPU). Given a FLAME expression/pose facial codes feature along with camera perspective, and a pre-trained avatar model, GaussianAvatar can render a realistic 2D portrait. After training, GaussianAvatar reconstructs a rigged FLAME-based 3D Gaussian Splatting avatar [ 5 ] defined as: \displaystyle\mathcal{F}_{\text{3DGS}}(\mathbf{p},\mathbf{M})\quad\textrm{where } \displaystyle\mathbf{p} \displaystyle:\textrm{3DGS primitives} (1) \displaystyle\mathbf{M} \displaystyle:\textrm{FLAME mesh} (2) Driven by the expression code output at Sect. 3.1 , the 3D facial FLAME mesh (right Fig. 3 ) morphed from one 3D face shape to another along with the Gaussian incorporated with the original FLAME mesh’s face elements. 3.2.2 Multi-view Display Calibration LGP collects 48 discrete views of a given 3D scene to project them across a 40-degree-wide view cone. The 48-view input requires a calibration function \Phi_{C} based on the display’s optical properties: \Phi_{C}(\mathbf{fov},\mathbf{d},\mathbf{\alpha}_{off},\mathbf{t}_{off},\mathbf{ar}) The \Phi_{C} ’s main objective is to extract the perspective view and projection matrix from situational parameters such as display device properties (aspect ration \mathbf{ar} , field-of-view \mathbf{fov} ) and user’s viewing position parameters (angle \mathbf{\alpha} , distance \mathbf{D} ) by calculating offset parameters (translation \mathbf{t}_{offset} and rotation \mathbf{\alpha}_{offset} , see Fig. 2 ) to correct the projection of neighbor views merged into each other at audience’s perspective. For simplification, we assume the discrete view is distributed horizontally on which the audience will observe the stereoscopic display. The detailed process was described in Alg. 1 where offset translation was calculated before being applied to the view matrix and projection matrix for each discrete view. The process is repeated with 48 views before sending to the LGD device via USB interface. 4. Evaluation We evaluate the current generative facial expression according to existing quantitative metrics regarding appropriateness, diversity, realism, and synchrony [ 7 ] , see Table 1 . The adopted method showed improvement across various metrics that could verify the quality of generated non-verbal reactions given multi-modal stimulation from a conversation. In the future, we plan to translate the current non-verbal facial reaction taking into account pain intensity for the pain assessment training as mentioned at the beginning. 5. Conclusion Integrating photo-realistic interactive avatars into medical training for pain assessment holds great promise for enhancing the skills of healthcare professionals. By focusing on technological enhancements, improving interactivity, conducting rigorous validation, and ensuring scalability, this research can significantly contribute to more effective and accessible medical education.