Selective QA over Conflicting Multi-Source Personal Memory: A Diagnostic Testbed and Method Comparison

Tiancheng Yang, Matthias Schonlau, Ilia Sucholutsky · 2026

Read Selective QA over Conflicting Multi-Source Personal Memory: A Diagnostic Testbed and Method Comparison by Tiancheng Yang, Matthias Schonlau, Ilia Sucholutsky (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30087.txt.

Introduction

Tiancheng Yang, Matthias Schonlau and Ilia Sucholutsky study question answering over conflicting memory sources, where a system must weigh sources or say nothing when proof falls short. Their test set spans 18 question forms, 8 reasoning types, 480 personas and over thirty-four thousand cases, with set distortions and fixed right answers. The best trained fusion method scores about four fifths right, ahead of the strongest prompt-only model at seven tenths. Allowed to abstain, it scores higher on answered questions at lower coverage, while different models shine on different reasoning types.

Catalog record

CREATORTiancheng Yang, Matthias Schonlau, Ilia Sucholutsky
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48498
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.30087

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Selective QA over Conflicting Multi-Source Personal Memory: A Diagnostic Testbed and Method Comparison Tiancheng Yang, Matthias Schonlau, Ilia Sucholutsky Abstract Emerging personal AI agents are moving toward persistent, multi-source memory. This creates an evaluation problem: systems must decide how to use conflicting or incomplete evidence; they cannot just retrieve facts from one clean history. Existing benchmarks rarely show whether an error came from the evidence given to a method or from the method’s conflict-resolution step. We study this as selective QA over conflicting multi-source personal memory: systems answer based on conflicting, sometimes incomplete sources, or abstain when evidence is insufficient. We develop a benchmark containing 18 question templates across 8 reasoning types, 480 personas, 4 random seeds, and 34,560 instances, with controlled source distortions and deterministic ground truth. We evaluate the performance of baselines without access to any source, access to a single source, structured fusion methods, and frontier LLMs. The best trained fusion resolver reaches 80.3% accuracy, while the strongest prompt-only LLM baseline reaches 70.0%. With abstention, the same resolver reaches 85.3% selective accuracy at 78.3% coverage and the best LLM reaches 71.0% selective accuracy at 95.4% coverage. Different models have different strengths across reasoning types. We release the data, code, cached model outputs, and data-generating process for reuse. 1 Introduction Personal AI agents are assistants that use a user’s history, preferences, plans, and records to answer and act across sessions. Personal-agent systems and agent-memory architectures already point to a future setting in which this memory is layered: long-term profiles, recent logs, plans, self-reports, and device records [ 24 , 22 , 1 , 4 , 37 , 25 , 44 ] . As these systems mature, they will collect more memory from more places rather than maintain one clean history. Preferences can drift, plans can be optimistic, self-reports can carry topic-dependent bias, and sensors can be missing or noisy [ 16 , 3 , 27 , 2 , 8 ] . The exercise example in Figure 1 shows the resulting conflict at small scale: plans and self-reports can overstate activity, objective and device traces can be incomplete, and the resolver must decide whether to answer conservatively or abstain. This creates a deployment-relevant evaluation problem as well as an accuracy problem. If one black-box LLM both extracts memory facts and resolves conflicts, changing the model can also change the conflict policy. We use a resolver for the decision layer that maps the available memory representation, structured atoms (closed-class observations extracted from each source) or natural-language memory, to an answer or SKIP. Our benchmark separates extraction from resolution: an LLM can extract source atoms, while a structured resolver can learn source-to-ground-truth mappings from labels. Figure 1: A personal-memory agent sees five memory sources that disagree. Direct prompting can over-trust optimistic or vague sources, while a calibrated resolver can give a lower binned answer or abstain. To our knowledge, conflict resolution across heterogeneous personal-memory sources has not yet been formalized as an evaluation target. Among coherent chat-history memory benchmarks, LongMemEval [ 41 ] studies memory over a single user–assistant interaction history. Concurrent multi-source memory work such as LifeBench [ 7 ] studies long-horizon retrieval and integration from diverse digital traces. Our setting targets a different failure mode: multiple sources can report conflicting evidence about the same latent behavior, so the system must learn source reliability and abstain when support is insufficient. Prior conflict-QA work studies cases where contexts, passages, or evidence sources disagree [ 42 , 21 , 6 , 18 ] . Context-use studies ask whether QA systems rely on the provided context appropriately [ 31 ] . Our benchmark targets a different setting: named personal-memory sources, such as profiles, plans, logs, and self-reports, can be biased in different ways for different questions. Reliability-aware RAG is related, but it usually learns a source-level reliability weight rather than how each source type tends to distort particular facts [ 13 ] . Cross-modal conflict benchmarks probe perception rather than textual evidence fusion [ 38 , 28 ] . In multi-source personal memory, source reliability is part of the task. This lets us test whether a method can learn how each source maps to ground truth, instead of folding all conflict handling into a single prompted LLM. Existing QA benchmarks often force an answer, but personal-memory agents also need a calibrated option to abstain when sources contain irreconcilable conflict. Appendix A compares related benchmarks on multi-source evidence, directed bias, selective QA, resolver-versus-input decomposition, and synthetic data-generation support. The testbed follows this separation explicitly. A synthetic data-generating process (DGP) creates latent daily events, projects them into five biased memory streams, renders those streams as natural-language memory, and asks resolvers to answer closed-class questions or SKIP. Structured methods first use an LLM extractor to turn each stream into these atoms; LLM baselines may instead read the natural-language memory directly. This design lets us vary the resolver and the input representation separately. We make three contributions. 1. Task formalization. We define bias-aware selective QA over multi-source personal memory, where sources have directional distortions and resolvers may abstain under irreconcilable conflict (§ 2 ; Appendix A ). 2. Testbed. We release a synthetic diagnostic testbed of 18 question templates \times 480 personas \times 4 seeds (34,560 instances), with a documented DGP, label rules, code, and runnable extension examples for new questions and evidence streams (§ 2 ). 3. Diagnostic evaluation. We compare a suite of structured and LLM resolvers plus a direct-readout reference, showing that learned source models, calibrated abstention, and type-specific failure signatures reveal where conflict-resolution systems break (§ 4 ). 2 Benchmark Design We define selective QA over conflicting multi-source personal memory as follows. A persona p is associated with five evidence streams \mathcal{S}=\{s_{1},\ldots,s_{5}\} : long-term profile, planner, daily self-report, objective log, and device log. The streams describe the same 30-day window through different views: profile summaries can be stale, plans can be optimistic, self-reports can carry topic-dependent bias, objective logs can be noisy or incomplete, and device logs can have day-level and field-level missingness (full distortion model in Appendix B.1 ). Given a question q with discrete answer space \mathcal{V}_{q} ( |\mathcal{V}_{q}|\in\{3,4\} ), a method must produce either v\in\mathcal{V}_{q} or SKIP. Answers are scored against a deterministic ground truth (GT) label defined by the question-specific rule. SKIP is distinct from factual edge labels in \mathcal{V}_{q} , such as no_plans . We report both answer-only and selective variants in § 4 . The benchmark spans 18 question templates across eight reasoning types (Table 1 ). Table 1 groups templates by the evidence pattern a method must resolve. Seven types target different source-conflict patterns, while the control type checks calibration under minimal conflict. Topic coverage spans work, diet, social, sleep, and exercise; 15/18 questions use ordered answer scales and 3 use unordered categorical labels. Representative questions and the survey-design principles applied to all 18 are in Appendix B.2 . Table 1: Question families in the diagnostic testbed. The eight reasoning types cover evidence-conflict patterns that can arise across personal-memory sources; each row states the pattern tested by the corresponding templates. Type Question family What this tests A Source arbitration Several streams report the same quantity with different values; the method must learn which source is reliable

Read the complete text