SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory
Huacan Chai, Yukai Wang, Yingxuan Yang, Dan Peng, Yuanyi Song, Zhihui Fu, Weiwen Liu, Jianghao Lin, Jun Wang, Weinan Zhang · 2026
Read SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory by Huacan Chai, Yukai Wang, Yingxuan Yang, Dan Peng, Yuanyi Song, Zhihui Fu, Weiwen Liu, Jianghao Lin, Jun Wang, Weinan Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15710.txt.
Introduction
Huacan Chai and nine colleagues launch SMMBench, a benchmark for source-distributed multimodal agent memory. Past benchmarks test memory inside pre-assembled contexts. They under-test whether agents can use proof scattered across sources that arose apart. SMMBench gauges whether agents can fetch, align, and compose proof across many sources, not one context. It tests cross-source reasoning, conflict resolution, preference reasoning, and action prediction. The set holds 1877 samples grounded in 264 sources. Tests on memory-style and retrieval-based baselines show current systems still struggle on these skills.
Catalog record
| CREATOR | Huacan Chai, Yukai Wang, Yingxuan Yang, Dan Peng, Yuanyi Song, Zhihui Fu, Weiwen Liu, Jianghao Lin, Jun Wang, Weinan Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 117601 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.15710
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory Huacan Chai, Yukai Wang, Yingxuan Yang, Dan Peng, Yuanyi Song, Zhihui Fu, Weiwen Liu, Jianghao Lin, Jun Wang, Weinan Zhang Abstract Existing benchmarks for multimodal memory reasoning largely evaluate systems within pre-assembled contexts, but under-evaluate whether agents can use evidence distributed across independently originated sources. We argue that source-distributed memory composition is an important and under-examined bottleneck in multimodal agent memory, especially when relevant evidence is fragmented across heterogeneous artifacts such as conversations, profiles, screenshots, tables, images, and documents. To address this gap, we introduce Source-distributed Multimodal Memory Benchmark (SMMBench), which measures whether agents can retrieve, align, and compose multimodal evidence scattered across multiple sources rather than reason within a single curated context. SMMBench evaluates four core capabilities: (1) cross-source multimodal reasoning; (2) conflict resolution; (3) preference reasoning; (4) memory-grounded action prediction. The benchmark contains 1,877 1,877 samples grounded in 264 264 sources. Experiments on representative memory-style and retrieval-based baselines show that current systems still struggle on these capabilities, positioning source-distributed multimodal memory as an important and still under-evaluated challenge for multimodal agents. Our data are available at https://huggingface.co/datasets/HuacanChai/SMMBench . 1 Introduction Multimodal agents are increasingly expected to act as persistent assistants in productivity, desktop, and enterprise settings [ 41 , 21 , 8 , 38 ] , where most real-world tasks are inherently cross-source : the information needed to answer questions or execute actions is typically accumulated over time across chats, tables, documents, and other artifacts, rather than packaged in a single context [ 16 , 35 , 1 ] . This setting reveals a challenge in agent memory: the difficulty is often not merely reading a long input, but using evidence that is distributed across independent sources created at different times and for different purposes, rather than reasoning over one pre-assembled context prepared for the final query [ 43 ] . We argue that source-distributed memory composition is an under-evaluated bottleneck in multimodal agent memory. Source-distributed means that the evidence is fragmented across multiple independently originated sources, such as separate group or private chats, profiles, tables, and documents, each with its own main purpose and local context. This brings challenges that are qualitatively different from reasoning over a single curated context. First, relevant evidence is distributed across multiple sources, and no single source is sufficient to determine the final answer on its own . For example, as illustrated in Figure 1 , an agent may need to connect evidence from a department chat, a meeting-location table, and a phone screenshot to infer that ‘John will fly to New York for Meeting A on Nov. 13’; no single source states this answer directly. Second, necessary evidence is often distributed across independently originated sources with different purposes and local contexts. Because these sources are created independently for different purposes rather than jointly organized for the query, their local contexts compartmentalize partial clues and make them harder to connect . This creates a distinct memory bottleneck: the agent must identify sources and bridge across their contextual boundaries to compose the answer. Third, information from different sources may conflict with one another, requiring the agent to update evidence and resolve conflicts by reasoning over their different authority levels or time states. Generally, the key challenge is not merely remembering isolated facts, but composing distributed evidence into answers or actions . Prior benchmarks have made important progress on multimodal long-context and memory settings, but most of them still evaluate reasoning within a single pre-assembled context. Multimodal long-context benchmarks such as MILEBench [ 26 ] and Mementos [ 31 ] evaluate whether MLLMs can retrieve, compare, and reason over long text-image contexts or visual streams. Mem-Gallery [ 2 ] further moves toward conversational memory over coherent multimodal interaction traces. However, these benchmarks primarily evaluate evidence use within a coherent context or unified retrieval corpus. They therefore leave under-evaluated whether agent memory systems can compose multimodal evidence distributed across independently originated sources. Figure 1: In real-world tasks, the necessary evidence is often distributed across multiple sources with distinct purposes and local contexts, while remaining potentially overlapping entities. Because no single source is sufficient, agents must retrieve and compose fragmented evidence across sources, making source-distributed memory a key bottleneck for memory-grounded responses and actions. To evaluate this gap, we introduce S ource-distributed M ultimodal M emory Bench ( SMMBench ), a benchmark for multimodal agent memory in which relevant evidence is intentionally distributed across multiple independently originated sources rather than provided as one pre-assembled context. The benchmark covers representative artifact types that arise in real-world persistent assistant scenarios, including conversations, profiles, tables, images, and documents, and organizes evaluation around four core capabilities: cross-source reasoning, conflict resolution, preference reasoning, and memory-grounded action prediction. It provides fine-grained evidence annotations together with both open-book and retrieval-based evaluation settings, enabling analysis of not only end-task accuracy but also how systems use distributed memory under different access conditions. Overall, the benchmark contains 1877 evaluation samples across 5 task types and 264 sources. Experimental results show that even the strongest evaluated systems still perform poorly in this setting, highlighting source-distributed memory as an important and still under-evaluated challenge for multimodal agents. Our contributions are as follows: • Problem Identification. We identify source-distributed memory composition as an under-evaluated bottleneck in multimodal agent memory, where the key challenge is composing evidence across independently originated sources rather than reasoning within one prepared context. • Challenge Characterization. We clarify how source-distributed memory differs from standard long-context reasoning by characterizing the distinct challenges introduced by independent sources, including source-level incompleteness, cross-source context bridging, and conflict resolution under different authority levels or time states. • Benchmark Construction. We introduce S ource-distributed M ultimodal M emory Bench , a benchmark that operationalizes this challenge through source objects such as conversations, profiles, screenshots, tables, images, and documents, and evaluates cross-source reasoning, conflict resolution, preference reasoning, and memory-grounded action prediction. • Empirical Findings. We provide fine-grained evidence annotations and evaluation experiments under open-book and retrieval-based settings. Experiments on 1877 samples, 5 task types, and 264 sources show that current representative methods remain far from effective. Table 1: Comparison with representative memory benchmarks. ✓ : Satisfies; ✗ : Does not satisfy. Benchmark MM Ev.M. Src. M. M.Ev. Indep. Src. X.S. C.R. P.R. A.P. LongMemEval [ 33 ] ✗ T C ✓ ✗ ✗ ✗ ✓ ✗ MemoryAgentBench [ 9 ] ✗ T C/D ✓ ✗ ✗ ✓ ✓ ✗ LoCCO [ 11 ] ✗ T C ✗ ✗ ✗ ✗ ✗ ✗ LoCoMo [ 19 ] ✓ T/I C ✓ ✗ ✓ ✗ ✓ ✗ Mementos [ 31 ] ✓ I I ✓ ✗ ✓ ✗ ✗ ✗ MMDU [ 18 ] ✓ T/I C ✗ ✗ ✓ ✗ ✗ ✗ MMRC [ 36 ] ✓ T/I C ✗ ✗ ✓ ✓ ✗ ✗ MultiHaystack [ 35 ] ✓ D/I/V D/I ✗ ✓ ✗ ✗ ✗ ✗ Mem-Gallery [ 2 ] ✓ T/I C ✓ ✗ ✓ ✓ ✗ ✗ SMMBench ✓ T/I/D/Tab. C/D/I ✓ ✓ ✓ ✓ ✓ ✓ MM : contains multimodal inputs; Ev.M. & Src.M. : modality types of evidence and sources, T for text, I for image, D for document, Tab. for table, C for conversation; M.E. : problems cannot