S 3 Mem: Structured Spatiotemporal Scene-Event Memory for Long-Horizon Interactive Question Answering

Encheng Su 1 , Jinouwen Zhang 3 , Jianyu Wu 2 , Qiucheng Yu 4 , Chen Tang 5 , Pengze Li 6 , Lintao Wang 7 , Yizhou Wang 5,† , Xinzhu Ma 8 , Shixiang Tang 5 , Aoran Wang 3,† 1 University of Science and Technology of China 2 Shanghai Jiao Tong University 3 Shanghai AI Laboratory 4 City University of Hong Kong 5 The Chinese University of Hong Kong 6 Fudan University 7 The University of Sydney 8 Beihang University † Corresponding authors · 2026

Read S 3 Mem: Structured Spatiotemporal Scene-Event Memory for Long-Horizon Interactive Question Answering by Encheng Su 1 , Jinouwen Zhang 3 , Jianyu Wu 2 , Qiucheng Yu 4 , Chen Tang 5 , Pengze Li 6 , Lintao Wang 7 , Yizhou Wang 5,† , Xinzhu Ma 8 , Shixiang Tang 5 , Aoran Wang 3,† 1 University of Science and Technology of China 2 Shanghai Jiao Tong University 3 Shanghai AI Laboratory 4 City University of Hong Kong 5 The Chinese University of Hong Kong 6 Fudan University 7 The University of Sydney 8 Beihang University † Corresponding authors (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28831.txt.

Introduction

Encheng Su and eleven colleagues propose S3Mem, a scene-event memory for long-horizon QA. Long-horizon questions often need sparse evidence from mixed histories. Existing interfaces expand reader context or retrieve related chunks, but none select compact evidence for a reader. S3Mem writes histories into scene-event units, sending evidence packs to the reader. Its router rates units and links for single- and multi-hop chains without reader training. On LoCoMo it reaches F1 near one half and BLEU near two fifths with about 1,073 tokens per question.

Catalog record

CREATOREncheng Su 1 , Jinouwen Zhang 3 , Jianyu Wu 2 , Qiucheng Yu 4 , Chen Tang 5 , Pengze Li 6 , Lintao Wang 7 , Yizhou Wang 5,† , Xinzhu Ma 8 , Shixiang Tang 5 , Aoran Wang 3,† 1 University of Science and Technology of China 2 Shanghai Jiao Tong University 3 Shanghai AI Laboratory 4 City University of Hong Kong 5 The Chinese University of Hong Kong 6 Fudan University 7 The University of Sydney 8 Beihang University † Corresponding authors
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE97514
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.28831

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

S 3 Mem: Structured Spatiotemporal Scene-Event Memory for Long-Horizon Interactive Question Answering Encheng Su 1 , Jinouwen Zhang 3 , Jianyu Wu 2 , Qiucheng Yu 4 , Chen Tang 5 , Pengze Li 6 , Lintao Wang 7 , Yizhou Wang 5,† , Xinzhu Ma 8 , Shixiang Tang 5 , Aoran Wang 3,† 1 University of Science and Technology of China 2 Shanghai Jiao Tong University 3 Shanghai AI Laboratory 4 City University of Hong Kong 5 The Chinese University of Hong Kong 6 Fudan University 7 The University of Sydney 8 Beihang University † Corresponding authors Abstract Long-horizon interactive agents often accumulate large trajectory histories yet still fail to answer questions about earlier events reliably. We argue that the main bottleneck is not context length alone, but the trajectory-to-answer interface of long-term memory. When histories are stored as plain-text chunks and queried with standard retrieval-augmented generation (RAG), systems often retrieve locally relevant but chain-incomplete evidence, especially for spatial, temporal, repeated-event, and multi-hop state questions. We propose S 3 Mem , a structured scene-event episodic memory framework for long-horizon interactive question answering (QA). S 3 Mem writes trajectories into structured memory units, retrieves evidence through anchor-sensitive retrieval, and exposes a compact token-budget-aware evidence interface for answer-time inference. In this sense, S 3 Mem is a structured evidence harness that converts agent trajectories into query-aligned support. We evaluate S 3 Mem on two internal headline environments ( Crafter , Jericho ) and two out-of-family environments ( ScienceWorld , ALFWorld ). Under a shared frozen answer-time protocol, S 3 Mem consistently outperforms Vanilla RAG across all four environments, surpasses Graph-NoReader on Crafter , Jericho , and ALFWorld , and matches it on ScienceWorld while using dramatically fewer evidence tokens. Three adapted recent baselines—A-MEM-inspired, MemoryOS-adapted, and LightMem-adapted—improve over Vanilla RAG in several settings, but none matches S 3 Mem ’s overall accuracy–efficiency frontier. Overall, the evidence supports a bounded conclusion: under the current frozen answer-time protocol, structured write and anchor-sensitive evidence routing provide a stronger accuracy–efficiency frontier for long-horizon interactive QA than more generic memory interfaces. Keywords: episodic memory, long-horizon agents, interactive question answering, structured retrieval, evidence interface 1 Introduction LLM/VLM agents have made substantial progress in text-based environments [ 8 , 21 ] , visually grounded interaction [ 7 ] , and multi-step decision making [ 26 ] . Yet even strong agents remain brittle when asked about events that occurred much earlier in an episode. This weakness is especially visible in long-horizon interactive question answering , where a system must not only remember past observations, but also recover a usable evidence chain across objects, events, relations, state changes, and temporal order. A common practice is to externalize trajectory history as text chunks or short summaries, then retrieve top- k snippets with retrieval-augmented generation (RAG) [ 13 ] . This approach is simple and often effective in short contexts, but it breaks down when the question depends on repeated-event disambiguation, temporal offsets, spatial reasoning, or multi-hop state tracking. The problem is not merely that relevant evidence exists far away in the trajectory. Rather, generic memory interfaces weaken the trajectory-to-answer transfer in three recurrent ways. First, writing trajectories as flat text erases structure that later reasoning depends on. Second, retrieval based primarily on surface similarity often returns snippets that are locally relevant but globally chain-incomplete. Third, the answer stage receives long unstructured fragments rather than a compact, query-aligned evidence set, forcing the model to reconstruct the chain itself. We therefore argue that long-horizon interactive QA should be treated not merely as a text-retrieval problem, but as an episodic evidence-construction problem over agent-generated trajectories. A useful memory interface for this setting should satisfy three properties: (i) it should write trajectories into a structured representation aligned with the interaction process; (ii) it should retrieve evidence using question-conditioned anchors such as target steps, repeated occurrences, and state transitions; and (iii) it should expose a compact but sufficient evidence interface under a strict token budget. Our central claim is deliberately narrow: under the current frozen answer-time protocol, the key to long-horizon interactive QA is not exposing more history, but exposing the right structured evidence from history. Motivated by this view, we propose S 3 Mem , a structured scene-event episodic memory framework for long-horizon interactive QA. Instead of treating interaction history as generic text, S 3 Mem writes each interaction step as a structured memory unit containing scene, event, state, and temporal context. At query time, it performs anchor-sensitive retrieval that explicitly recovers target-step anchors, occurrence anchors, and state-transition anchors. It then constructs a compact, token-budget-aware evidence interface rather than exposing all retrieved evidence verbatim. In this sense, S 3 Mem should be understood as a structured evidence harness: its role is to convert long trajectories into the smallest query-aligned evidence chain that still supports answer-time inference. This framing also clarifies the limitation of competing baselines. The main weakness of plain-text RAG, graph-only retrieval, and recent generic memory neighbors is not that they lack memory altogether, but that their memory interfaces generalize weakly under long-horizon, budgeted QA. They may store large amounts of history or compress it efficiently, but they do not consistently preserve the decisive anchor-bearing step together with the minimal local chain required by the question. As a result, generic baselines face a tradeoff: exposing more history can improve accuracy only at very high token cost, whereas generic shrinking reduces tokens without recovering the same chain-complete evidence. Our objective is therefore not to build a general agent operating system, but to improve the trajectory-to-answer interface for episodic QA. We evaluate S 3 Mem on two internal headline environments, Crafter [ 7 ] and Jericho [ 8 ] , and two out-of-family environments, ScienceWorld [ 23 ] and ALFWorld [ 21 ] . Across all four environments, S 3 Mem consistently outperforms Vanilla RAG and surpasses the Graph-NoReader baseline on Crafter , Jericho , and ALFWorld , while matching it on ScienceWorld at dramatically lower evidence cost. The external evidence is intentionally interpreted more narrowly than the internal headline. On ScienceWorld , S 3 Mem mainly demonstrates efficiency generalization : it preserves near-ceiling accuracy while greatly reducing evidence tokens. On ALFWorld , it demonstrates a stronger accuracy and efficiency gain. At the same time, Full-History controls show that exposing more history can still improve accuracy when token cost is ignored, so the strongest supported claim of this paper is an accuracy–efficiency frontier claim rather than unconditional accuracy optimality. We further sharpen this interpretation through three complementary analyses. First, we compare against three recent neighboring memory systems—A-MEM, MemoryOS, and LightMem [ 25 , 12 , 4 ] —adapted to the same trajectory-grounded episodic-QA setting. Second, we test the strongest alternative explanation for the gain, namely that any sufficiently aggressive generic compressor might recover a similar frontier, through Full-History, Summarize-then-Answer, and RTK-style generic-compression controls. Third, we examine robustness under trajectory-source shift through a four-actor Crafter rollout study, and include a bonus out-of-family transfer test on ATM-Bench [ 27 ] . Our contributions are threefold: 1. Method contribution: We introduce S 3 Mem , a structured scene-event episodic memory interface that unifies structured writing, anchor-sensitive retrieval, and token-budget-aware evidence exposure for long-horizon interactive QA. 2. Empirical contribution:

Read the complete text