Selective QA over Conflicting Multi-Source Personal Memory: A Diagnostic Testbed and Method Comparison
Tiancheng Yang, Matthias Schonlau, Ilia Sucholutsky · 2026
Read Selective QA over Conflicting Multi-Source Personal Memory: A Diagnostic Testbed and Method Comparison by Tiancheng Yang, Matthias Schonlau, Ilia Sucholutsky (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30087.txt.
Introduction
Tiancheng Yang, Matthias Schonlau and Ilia Sucholutsky study question answering over conflicting memory sources, where a system must weigh sources or say nothing when proof falls short. Their test set spans 18 question forms, 8 reasoning types, 480 personas and over thirty-four thousand cases, with set distortions and fixed right answers. The best trained fusion method scores about four fifths right, ahead of the strongest prompt-only model at seven tenths. Allowed to abstain, it scores higher on answered questions at lower coverage, while different models shine on different reasoning types.
Catalog record
| CREATOR | Tiancheng Yang, Matthias Schonlau, Ilia Sucholutsky |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48498 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30087
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Selective QA over Conflicting Multi-Source Personal Memory: A Diagnostic Testbed and Method Comparison Tiancheng Yang, Matthias Schonlau, Ilia Sucholutsky Abstract Emerging personal AI agents are moving toward persistent, multi-source memory. This creates an evaluation problem: systems must decide how to use conflicting or incomplete evidence; they cannot just retrieve facts from one clean history. Existing benchmarks rarely show whether an error came from the evidence given to a method or from the method’s conflict-resolution step. We study this as selective QA over conflicting multi-source personal memory: systems answer based on conflicting, sometimes incomplete sources, or abstain when evidence is insufficient. We develop a benchmark containing 18 question templates across 8 reasoning types, 480 personas, 4 random seeds, and 34,560 instances, with controlled source distortions and deterministic ground truth. We evaluate the performance of baselines without access to any source, access to a single source, structured fusion methods, and frontier LLMs. The best trained fusion resolver reaches 80.3% accuracy, while the strongest prompt-only LLM baseline reaches 70.0%. With abstention, the same resolver reaches 85.3% selective accuracy at 78.3% coverage and the best LLM reaches 71.0% selective accuracy at 95.4% coverage. Different models have different strengths across reasoning types. We release the data, code, cached model outputs, and data-generating process for reuse. 1 Introduction Personal AI agents are assistants that use a user’s history, preferences, plans, and records to answer and act across sessions. Personal-agent systems and agent-memory architectures already point to a future setting in which this memory is layered: long-term profiles, recent logs, plans, self-reports, and device records [ 24 , 22 , 1 , 4 , 37 , 25 , 44 ] . As these systems mature, they will collect more memory from more places rather than maintain one clean history. Preferences can drift, plans can be optimistic, self-reports can carry topic-dependent bias, and sensors can be missing or noisy [ 16 , 3 , 27 , 2 , 8 ] . The exercise example in Figure 1 shows the resulting conflict at small scale: plans and self-reports can overstate activity, objective and device traces can be incomplete, and the resolver must decide whether to answer conservatively or abstain. This creates a deployment-relevant evaluation problem as well as an accuracy problem. If one black-box LLM both extracts memory facts and resolves conflicts, changing the model can also change the conflict policy. We use a resolver for the decision layer that maps the available memory representation, structured atoms (closed-class observations extracted from each source) or natural-language memory, to an answer or SKIP. Our benchmark separates extraction from resolution: an LLM can extract source atoms, while a structured resolver can learn source-to-ground-truth mappings from labels. Figure 1: A personal-memory agent sees five memory sources that disagree. Direct prompting can over-trust optimistic or vague sources, while a calibrated resolver can give a lower binned answer or abstain. To our knowledge, conflict resolution across heterogeneous personal-memory sources has not yet been formalized as an evaluation target. Among coherent chat-history memory benchmarks, LongMemEval [ 41 ] studies memory over a single user–assistant interaction history. Concurrent multi-source memory work such as LifeBench [ 7 ] studies long-horizon retrieval and integration from diverse digital traces. Our setting targets a different failure mode: multiple sources can report conflicting evidence about the same latent behavior, so the system must learn source reliability and abstain when support is insufficient. Prior conflict-QA work studies cases where contexts, passages, or evidence sources disagree [ 42 , 21 , 6 , 18 ] . Context-use studies ask whether QA systems rely on the provided context appropriately [ 31 ] . Our benchmark targets a different setting: named personal-memory sources, such as profiles, plans, logs, and self-reports, can be biased in different ways for different questions. Reliability-aware RAG is related, but it usually learns a source-level reliability weight rather than how each source type tends to distort particular facts [ 13 ] . Cross-modal conflict benchmarks probe perception rather than textual evidence fusion [ 38 , 28 ] . In multi-source personal memory, source reliability is part of the task. This lets us test whether a method can learn how each source maps to ground truth, instead of folding all conflict handling into a single prompted LLM. Existing QA benchmarks often force an answer, but personal-memory agents also need a calibrated option to abstain when sources contain irreconcilable conflict. Appendix A compares related benchmarks on multi-source evidence, directed bias, selective QA, resolver-versus-input decomposition, and synthetic data-generation support. The testbed follows this separation explicitly. A synthetic data-generating process (DGP) creates latent daily events, projects them into five biased memory streams, renders those streams as natural-language memory, and asks resolvers to answer closed-class questions or SKIP. Structured methods first use an LLM extractor to turn each stream into these atoms; LLM baselines may instead read the natural-language memory directly. This design lets us vary the resolver and the input representation separately. We make three contributions. 1. Task formalization. We define bias-aware selective QA over multi-source personal memory, where sources have directional distortions and resolvers may abstain under irreconcilable conflict (§ 2 ; Appendix A ). 2. Testbed. We release a synthetic diagnostic testbed of 18 question templates \times 480 personas \times 4 seeds (34,560 instances), with a documented DGP, label rules, code, and runnable extension examples for new questions and evidence streams (§ 2 ). 3. Diagnostic evaluation. We compare a suite of structured and LLM resolvers plus a direct-readout reference, showing that learned source models, calibrated abstention, and type-specific failure signatures reveal where conflict-resolution systems break (§ 4 ). 2 Benchmark Design We define selective QA over conflicting multi-source personal memory as follows. A persona p is associated with five evidence streams \mathcal{S}=\{s_{1},\ldots,s_{5}\} : long-term profile, planner, daily self-report, objective log, and device log. The streams describe the same 30-day window through different views: profile summaries can be stale, plans can be optimistic, self-reports can carry topic-dependent bias, objective logs can be noisy or incomplete, and device logs can have day-level and field-level missingness (full distortion model in Appendix B.1 ). Given a question q with discrete answer space \mathcal{V}_{q} ( |\mathcal{V}_{q}|\in\{3,4\} ), a method must produce either v\in\mathcal{V}_{q} or SKIP. Answers are scored against a deterministic ground truth (GT) label defined by the question-specific rule. SKIP is distinct from factual edge labels in \mathcal{V}_{q} , such as no_plans . We report both answer-only and selective variants in § 4 . The benchmark spans 18 question templates across eight reasoning types (Table 1 ). Table 1 groups templates by the evidence pattern a method must resolve. Seven types target different source-conflict patterns, while the control type checks calibration under minimal conflict. Topic coverage spans work, diet, social, sleep, and exercise; 15/18 questions use ordered answer scales and 3 use unordered categorical labels. Representative questions and the survey-design principles applied to all 18 are in Appendix B.2 . Table 1: Question families in the diagnostic testbed. The eight reasoning types cover evidence-conflict patterns that can arise across personal-memory sources; each row states the pattern tested by the corresponding templates. Type Question family What this tests A Source arbitration Several streams report the same quantity with different values; the method must learn which source is reliable