PCIE_Interaction Solution for Ego4D Social Interaction Challenge
Kanokphan Lertniphonphan, Feng Chen, Junda Xu, Fengbu Lan, Jun Xie, Tao Zhang, Zhepeng Wang · 2025
Read PCIE_Interaction Solution for Ego4D Social Interaction Challenge by Kanokphan Lertniphonphan, Feng Chen, Junda Xu, Fengbu Lan, Jun Xie, Tao Zhang, Zhepeng Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24404.txt.
Introduction
Kanokphan Lertniphonphan and co-authors report their entry to a social interaction challenge for first-person video. Their 2025 report covers the Ego4D challenge at a major vision conference. Two tasks define the contest: telling whether a person is looking at the camera wearer, and whether they are talking to them.
The looking task uses face crops alone, so the team enhanced face quality and blended several models. The talking task adds synced audio, and the team fused sound with sight, weighted by how clear the face looked. Their system scored 0.81 and 0.71 mean average precision on the two leaderboards. Code is public. The work shows how much social signal hides in a wearer's own view of a room.
Catalog record
| CREATOR | Kanokphan Lertniphonphan, Feng Chen, Junda Xu, Fengbu Lan, Jun Xie, Tao Zhang, Zhepeng Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 11 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-24404
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PCIE_Interaction Solution for Ego4D Social Interaction Challenge Kanokphan Lertniphonphan, Feng Chen, Junda Xu, Fengbu Lan, Jun Xie, Tao Zhang, Zhepeng Wang Abstract This report presents our team’s PCIE_Interaction solution for the Ego4D Social Interaction Challenge at CVPR 2025, addressing both Looking At Me (LAM) and Talking To Me (TTM) tasks. The challenge requires accurate detection of social interactions between subjects and the camera wearer, with LAM relying exclusively on face crop sequences and TTM combining speaker face crops with synchronized audio segments. In the LAM track, we employ face quality enhancement and ensemble methods. For the TTM task, we extend visual interaction analysis by fusing audio and visual cues, weighted by a visual quality score. Our approach achieved 0.81 and 0.71 mean average precision (mAP) on the LAM and TTM challenges leader board. Code is available at https://github.com/KanokphanL/PCIE_Ego4D_Social_Interaction 1 Introduction Egocentric video captured by wearable cameras has gained significant importance in computer vision and robotics. The Ego4D social interaction benchmark highlights interactions between the camera wearer and surrounding individuals through two key tasks: Looking At Me (LAM) and Talking To Me (TTM) et al. 2021 . The LAM task specifically detects whether a social partner’s eye gaze is directed toward the camera wearer, utilizing face bounding boxes with cross-frame identity consistency. In contrast, the TTM task identifies conversational engagement through synchronized audio-visual analysis of tracked individuals. While both tasks employ frame-level predictions, TTM’s labeling follows an utterance-level paradigm - even if the speaker momentarily looks away during a conversation segment, the interaction is still considered valid if the audio content indicates engagement with the camera wearer. Several approaches et al. 2021 ; Wei et al. leverage image sequences to detect eye gaze toward the camera wearer by analyzing visual cues from face crops. Wei et al. utilizes facial landmarks and head pose estimation to enhance spatial feature refinement. et al. 2021 incorporates both spatial and temporal dynamic refinement. More recently, Lertniphonphan et al. 2024 introduced a framework combining the InternVL image encoder Chen et al. 2023 with a Bi-LSTM network to jointly model spatial and temporal dependencies. The baseline model for TTM et al. 2021 extends the LAM architecture by incorporating an audio encoder to extract embeddings from synchronized audio segments. These audio embeddings are concatenated with visual features and jointly classified to predict TTM interactions. However, Lin et al. 2023 demonstrated that training separate models for audio and vision can gain a better results. Their method processes audio and visual modalities independently, then fuses the predictions using a face quality score to weight the TTM score of each modality. Our approach for social interaction analysis extends the LAM to the TTM task by employing separate models for visual and audio modalities. Based on the dataset characteristic, we introduce visual and audio filter before fusing the score from both models, yielding a robust final interaction score for TTM. 2 Method Figure 1 : Our proposed Social Interaction Detection framework Our proposed social interaction framework for the Ego4D challenges, illustrated in Figure 1 , addresses both Looking At Me (LAM) and Talking To Me (TTM) tasks through a designed multimodal architecture. The system processes visual and audio modalities through independent, task-optimized branches, with each stream passing through modality-specific filters prior to fusion. For LAM, we employ only the visual branch to detect gaze patterns. The TTM solution incorporates both modalities: visual predictions undergo max-score filtering to generate utterance-level representations, which are then fused with processed audio features through a quality-weighted fusion module. This dual-stream approach allows each modality to compensate for the other’s limitations while maintaining task-specific optimization. 2.1 Looking At Me The LAM dataset’s face crops exhibit significant motion blur due to relative movement between wearable cameras and subjects. Our analysis of GazeLSTM et al. 2021 demonstrated that augmenting the architecture with a transformer decoder yielded minimal improvement on validation data. Drawing from established methods, we developed an ensemble framework that integrates multiple model outputs while addressing image quality by applying deblurring techniques Chu et al. 2022 to enhance input face crops. Validation Test Method mAP ( \uparrow ) Acc ( \uparrow ) mAP ( \uparrow ) Acc ( \uparrow ) GazeLSTM et al. 2021 79.40% 92.78% 74.54% 87.04% GazeLSTM with Transformer head 76.81% 91.66% 73.29% 87.09% GazePose Wei et al. 81.89% 91.84% 76.95% 85.08% InternLSTM Lertniphonphan et al. 2024 84.23% 94.79% 76.57% 90.00% Table 1 : Looking at me experimental results 2.2 Talking To Me We adopt a dual-modality approach for the TTM task, implementing separate visual and audio models following Lin et al. 2023 . For audio processing, we employ the Whisper encoder Radford et al. 2022 , evaluating both small and large-v3 architectures. While the large model performs better on validation data (Table 2 ), this improvement does not generalize to the test set, suggesting potential overfitting to the validation distribution. The TTM annotation scheme presents unique challenges for visual modality integration. Since labels are assigned at the utterance level based on audio segments, speakers may intermittently look away while still engaged in conversation or might not look at the camera wearer at all. To address this mismatch between frame-level visual predictions and utterance-level labels, we implement a visual max-score filter. This filter aggregates visual predictions across each audio segment, retaining only the maximum visual score as the utterance-level representation. As shown in Table 2 , this filtering significantly improves visual-only performance. For multimodal fusion, we compare two approaches: (1) basic score averaging and (2) quality-weighted fusion Lin et al. 2023 , where weights are derived from frame-level face alignment scores Bulat and Tzimiropoulos 2017 . Final predictions undergo median filtering to smooth temporal inconsistencies before submission. Validation Test Method mAP ( \uparrow ) Acc ( \uparrow ) mAP ( \uparrow ) Acc ( \uparrow ) Audio modality Whisper_small 70.17% 63.91% 66.16% 60.80% Whisper_large_v3 70.51% 72.52% 66.17% 56.92% Visual modality GazeLSTM 63.75% 71.24% 54.86% 53.86% + Visual filter 63.27% 59.57% Audio and Visual modality Averaging 68.78% 62.32% Quality Score weight Fusion 69.81% 58.56% Table 2 : Talking to me experimental results 3 Experiments 3.1 Dataset and Evaluation metric We evaluate our approach on the Ego4D social benchmark et al. 2021 , which contains 572 video clips (5 minutes each), split into 389 training, 50 validation, and 133 test clips. The benchmark provides two distinct annotation schemes. For LAM, frame-level binary labels indicating whether each detected face is looking at the camera wearer. For TTM, utterance-level labels for audio-visual segments, marking whether the speaker is talking to the camera wearer. Mean Average Precision (mAP) and Top-1 accuracy are used for evaluation in both tasks. 3.2 Results Looking At Me We first reproduced the baseline approach from et al. 2021 , extending its architecture with a transformer decoder. For comprehensive comparison, we also implemented two state-of-the-art methods: GazePose Wei et al. and InternLSTM Lertniphonphan et al. 2024 (see Table 1 ). To enhance input quality, all test face crops were preprocessed using the image enhancement module to reduce motion blur and improve image quality. Our final submission combines predictions from multiple approaches through an ensemble of InternLSTM, GazeLSTM, and GazePose models with diverse parameter configurations. We applied a median filter to the ensemble outputs for temporal smoothing, achieving robust