EgoExoMem: Cross-View Memory Reasoning over Synchronized Egocentric and Exocentric Videos
Ruiping Liu, Junwei Zheng, Yufan Chen, Di Wen, Shaofang Quan, Chengzhi Wu, Jiaming Zhang, Kailun Yang, Kunyu Peng, Rainer Stiefelhagen · 2026
Read EgoExoMem: Cross-View Memory Reasoning over Synchronized Egocentric and Exocentric Videos by Ruiping Liu, Junwei Zheng, Yufan Chen, Di Wen, Shaofang Quan, Chengzhi Wu, Jiaming Zhang, Kailun Yang, Kunyu Peng, Rainer Stiefelhagen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18734.txt.
Introduction
Ruiping Liu and nine co-authors introduce EgoExoMem, a 2026 benchmark for cross view memory reasoning. It holds 2 point 6 thousand questions across eight question types. They propose E-Squared-Select, a training free frame picker for dual view retrieval. It combines budget allocation with per view k DPP sampling. Synced ego and exo views give complementary cues, yet existing models stay far from solving it. The best model reaches only about fifty-five percent, while the new method reaches about fifty-eight percent, beating memory baselines.
Catalog record
| CREATOR | Ruiping Liu, Junwei Zheng, Yufan Chen, Di Wen, Shaofang Quan, Chengzhi Wu, Jiaming Zhang, Kailun Yang, Kunyu Peng, Rainer Stiefelhagen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39519 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.18734
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EgoExoMem: Cross-View Memory Reasoning over Synchronized Egocentric and Exocentric Videos Ruiping Liu, Junwei Zheng, Yufan Chen, Di Wen, Shaofang Quan, Chengzhi Wu, Jiaming Zhang, Kailun Yang, Kunyu Peng, Rainer Stiefelhagen Abstract Egocentric memory is widely used in embodied intelligence, but it may be insufficient for comprehensive spatial-temporal reasoning. Inspired by human recall from both field and observer perspectives, we introduce EgoExoMem, the first benchmark for cross-view memory reasoning over synchronized egocentric and exocentric videos. EgoExoMem contains 2.6 K 2.6K high-quality MCQs across eight temporal, spatial, and cross-view QA types. To support dual-view retrieval, we propose E 2 -Select, a training-free frame selection method for synchronized ego-exo videos. It combines relevance-based budget allocation with per-view k-DPP sampling to handle view asymmetry and cross-view temporal consistency. Experiments show that ego and exo views provide complementary memory cues, while existing MLLMs remain far from solving the benchmark: the best model reaches only 55.3 % 55.3\% . E 2 -Select achieves state-of-the-art performance of 58.2 % 58.2\% over frame-selection and RAG-based memory baselines. Further analysis reveals systematic view-preference conflicts between question framing and answer grounding, underscoring the novelty and challenge of cross-view memory reasoning. The source code and dataset can be found at https://github.com/RuipingL/EgoExoMem . Figure 1 : EgoExoMem requires reasoning over synchronized ego-exo memory streams to answer questions unanswerable from either viewpoint alone. Blue indicates the temporal condition, green indicates keyframes or the correct answer, and red indicates distractors. 1 Introduction With the rapid advancement of multimodal large language models, embodied agents are increasingly expected to perceive, remember, and reason about complex real-world environments 54 . Central to this capability is memory: the ability to retain and retrieve past observations to support informed decision-making and long-horizon task execution 99 ; 18 . However, existing memory frameworks for embodied perception are predominantly built upon egocentric observations 24 ; 18 , capturing the world exclusively from the agent’s own first-person viewpoint. While egocentric memory has been widely studied for tasks such as episodic event recall 102 ; 103 ; 104 , scene understanding 101 ; 62 , and navigation 111 , it remains fundamentally limited by the agent’s physical vantage point, leaving out-of-view regions and their spatial relationships inconsistently observed. This raises a critical question: is egocentric memory alone sufficient for comprehensive spatial understanding and temporal reasoning in embodied environments? Real-world scenarios reveal fundamental limitations of purely egocentric memory. For holistic scene understanding, an agent’s egocentric stream captures its own interactions but offers limited visibility into other agents’ interactions within the shared space. While equipping all agents with smart glasses has been proposed 102 ; 41 , this is impractical due to device cost and wearable burden 52 . Egocentric recording is also ill-suited for capturing whole-body movements, particularly in healthcare contexts such as fall detection 100 . Given the growing prevalence of exocentric infrastructure, including hospital surveillance 81 ; 22 and smart home cameras 8 ; 78 , we propose complementing egocentric memory with exocentric observations to address these limitations. Humans naturally encode experiences from two complementary perspectives: field memory, which relives events through a first-person viewpoint, and observer memory, which recalls the same event from a third-person vantage point 65 . These perspectives mirror the brain’s parallel egocentric and allocentric reference frames 6 , and their interplay supports situation awareness 16 . Yet, existing ego-exo works focus on imitation learning or unidirectional knowledge transfer 108 , with methods treating the two streams independently 31 and benchmarks evaluating only local view-clip matching 27 or view-invariant temporal consistency 38 ; 77 , rather than reasoning over the full, synergistic memory formed by both views. To address this gap, we introduce EgoExoMem, a large-scale benchmark comprising 2.6K human-verified Multiple Choice Questions (MCQs) designed to evaluate memory-based reasoning over synchronized egocentric and exocentric videos sourced from EgoExo4D 25 and LEMMA 33 . We characterize memory-based reasoning along eight QA types covering temporal, spatial, and cross-view dimensions: Habitual Location, Instantaneous Position, Resulting Location, Egocentric Direction, Object State, Allocentric Relation, Third-Person Activity, and Temporal Ordering, with an example illustrated in Fig. 1 . A text-only check is applied to ensure the vision-dependency of all answers. Motivated by the view-asymmetric and temporally synchronized nature of EgoExoMem, we further propose E 2 -Select, a simple yet effective frame selection method with relevance-based budget allocation and k-DPP sampling, explicitly designed to handle view asymmetry and cross-view temporal consistency beyond single-view memory methods. Extensive experiments are conducted to validate the reasonableness of the EgoExoMem task, benchmark the dataset, demonstrate the effectiveness of E 2 -Select, and analyze failure cases. Proprietary 11 and open-source MLLMs 44 ; 91 ; 1 are evaluated on EgoExoMem as zero-shot baselines, confirming the task’s challenge. Memory mechanisms spanning frame selection 88 ; 53 , text retrieval 79 ; 40 , visually-grounded retrieval 60 ; 32 , and structured memory retrieval 41 ; 104 are used to benchmark EgoExoMem, while our E 2 -Select achieves state-of-the-art performance with an averaged MCQ accuracy of 58.2\% . Comprehensive ablation studies support the design choices of E 2 -Select. Failure case analysis reveals a view-preference conflict between question framing and answer grounding, highlighting the complementary potential of synergistic ego-exo views. 2 Related Work Table 1 : Comparison of related video QA benchmarks. “–” means not available. Benchmark Venue #QA #Videos #Tasks View Sync Memory QA Cross-view QA EgoSchema 63 NeurIPS’23 5.1k 5.1k 1 Ego – \times \times VSI-Bench 101 CVPR’25 5.0k 288 8 Ego – ✓ \times EgoMemoria 103 ICLR’25 7.0k 629 – Ego – ✓ \times EgoPlan-Bench 9 ECCV’24 4.9k – 1 Ego – \times \times EgoExoLearn 31 CVPR’24 2.2k 747 3 Ego+Exo Async \times ✓ EgoExoBench 27 NeurIPS’25 7.3k – 11 Ego+Exo Both \times ✓ MA-EgoQA 41 arXiv’26 1.7k – 5 Multi-ego – \times ✓ EgoExoMem (Ours) 2026 2.6k 390 8 Ego+Exo Sync ✓ ✓ Ego-Exo Video Understanding and Reasoning. Cross-view understanding spans view translation 59 ; 108 ; 61 ; 66 ; 82 , correspondence learning 29 ; 31 , cross-view grounding 21 ; 85 ; 58 , and audio-visual association 30 ; 35 . View-conditioned generation has also advanced rapidly, including ego-exo synthesis 97 ; 71 ; 69 , viewpoint-aware anticipation 83 , and analysis of active versus passive perspectives 56 . Beyond generation, view-invariant representation learning 70 and imitation error detection 49 further demonstrate the value of cross-view alignment for fine-grained activity understanding. Recent benchmarks have also expanded the ego-exo scope: EPFL-Smart-Kitchen 5 evaluates action and motion understanding in multi-modal ego-exo settings, and SAW-Bench 46 probes situated awareness in real-world environments. Closest to our setting, Ravi et al. 76 show that off-screen context can be recovered from egocentric observations, and EgoExoLearn 31 benchmarks skill assessment across paired viewpoints. Yet, existing methods treat the two streams as inputs to a shared representation rather than as asymmetric sources of complementary evidence, a distinction that prior cross-view benchmarks do not evaluate. Relevant benchmarks are summarized in Tab. 1 . Memory-Augmented Video Understanding. Episodic memory QA over egocentric video is well established 4 ; 13 ; 34 ; 36 ; 3 . Recent work pushes toward longer temporal horizons 102 ; 43 ; 17 and richer memory structures, including spatial mind palaces 23 , dynamic memory refinement 28 , and