Same Ranking, Different Winner: How Scoring Targets Shape LLM Memory Benchmarks

Sugam Panthi, Rabab Abdelfattah · 2026

Read Same Ranking, Different Winner: How Scoring Targets Shape LLM Memory Benchmarks by Sugam Panthi, Rabab Abdelfattah (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24060.txt.

Introduction

Panthi and Abdelfattah show that scoring-target choice can silently flip conversational-memory benchmark verdicts. Their audit, TIAP, regrades saved ranked outputs without rerunning retrieval, crediting Raw, Source, or Canonical in turn. Switching only the credited target changed nDCG on 83.4 to 94.0 percent of shared queries, flipped target orderings on transfer runs, and reversed parser-density recommendations. A 1,902-case semantic audit found relaxed source-linked credit fully justified only 29.2 percent of the time. Memory papers, they urge, should define and report the scoring target explicitly.

Catalog record

CREATORSugam Panthi, Rabab Abdelfattah
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49903
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24060

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Same Ranking, Different Winner: How Scoring Targets Shape LLM Memory Benchmarks Sugam Panthi, Rabab Abdelfattah Abstract Conversational-memory systems increasingly transform dialogue history into facts, summaries, timelines, and other source-linked descendants, so a single source turn can coexist with several derived memories in the same retrieval index. This raises an under-specified evaluation question: which stored form should receive retrieval credit? We show that this scoring-target choice is often left implicit and can materially change benchmark conclusions. We present TIAP , a fixed-output audit that rescores saved ranked outputs under three targets—Raw, Source, and Canonical—without rerunning retrieval. On LoCoMo and LongMemEval-S, switching only the credited target changes nDCG on 83.4%–94.0% of shared queries, flips target orderings on Mem0 and MemoryOS transfer runs, and reverses parser-density recommendations. A 1,902-case semantic audit further shows that relaxed source-linked credit is fully justified only 29.2% of the time, despite high rubric reliability in a validation subset. These results reveal target noninvariance : conclusions about memory architectures can silently flip with a single benchmark-design choice. Conversational-memory papers should therefore define and report the scoring target explicitly. 1 Introduction Long term conversational AI systems are increasingly built around memory construction . Rather than passing raw dialogue history directly to the response model, modern chat assistants extract facts, consolidate updates, build timelines, populate profile memories, and often retain the original dialogue turn alongside its transformed descendants in the same retrieval index ( 6 ; 10 ; 20 ; 17 ; 24 ) . This storage pattern is now central to memory-augmented agents ( 11 ; 17 ; 29 ) , but it also creates an under-specified evaluation problem. A benchmark query is typically anchored to one source turn, while the evaluated store may contain several retrieval units derived from that same evidence. Which of these stored forms should receive retrieval credit? Figure 1: Overview of target noninvariance in conversational memory evaluation. A single source evidence turn can produce multiple stored memory forms in the same index. For a fixed ranked output, changing only the credited target can turn the same retrieval result into a miss or a hit. Figure 1 summarizes the core failure mode. A single source evidence turn may be stored as a raw memory, a source-linked derived memory, and a canonical serving memory. When the ranked output is fixed, the same retrieved item can be scored as a miss under a Raw target but as a hit under Source or Canonical targets. Thus, the benchmark conclusion can change even though retrieval behavior has not changed. The ambiguity has both semantic and numerical consequences. Semantically, a stored descendant may share the correct source anchor while omitting the answer, dropping an important condition, or preserving only weakly related context. In our 1,902 case audit of relaxed credit, only 29.2% of credited source-linked descendants fully support the query under a five-model majority vote. Thus, a more permissive target is not automatically a more correct one, and answer support can only be judged after the benchmark decides which stored evidence units are admissible ( 8 ; 22 ) . Numerically, target choice is not a bookkeeping artifact. If relaxing the target merely raised recall-like scores, the broadest target would dominate. Instead, target choice interacts with benchmark structure and memory architecture. In transfer runs, target orderings are non-monotonic for both Mem0 and MemoryOS, depending on the benchmark. Thus, the same fixed ranked outputs can recommend different memory-store designs depending only on which stored representation is credited. Because conversational-memory benchmarks increasingly guide retriever, storage-policy, and serving-representation choices, comparisons are only meaningful if the benchmark defines what counts as a correct retrieval. In transformed-memory stores that is exactly the contested object, yet it is typically left implicit. This is a benchmark-design problem rather than a system-tuning one: unlike hyperparameter sensitivity analysis, which varies a system knob while holding evaluation fixed, our audit varies the evaluation definition while holding system output fixed. It is also distinct from classical IR evaluation-sensitivity analysis, which varies relevance judgments or changes the retrieval unit globally rather than crediting system-generated descendants of a single source turn; we develop this contrast in Section 2 . We study this setting on LoCoMo and LongMemEval-S across four native retrievers. Switching only the scoring target changes Raw versus Canonical nDCG on 83.4% to 94.0% of shared queries. Transfer runs with Mem0 and MemoryOS remain target sensitive but show different target orderings, confirming that the effect is a benchmark validity problem on fixed ranked outputs rather than an artifact of target size alone. This paper makes three contributions. • We identify target noninvariance , a benchmark validity problem in transformed conversational memory: when ranked outputs are fixed, changing only the admissible target set can change scores, system rankings, and store design recommendations. • We show across two primary benchmarks, one supplementary stress test, and two external memory architectures that the effect is large, affecting 83.4% to 94.0% of shared queries, surviving transfer to Mem0 and MemoryOS, and not being removed by post-hoc aggregation. • We provide TIAP, a fixed-output audit procedure, and MTEL-Mem, a reproducibility layer that packages target mappings, source fixtures, and rescoring utilities for auditing saved traces. We do not argue that any one target is universally correct. Rather, when a benchmark can credit different stored forms of the same evidence, scoring target choice is part of the evaluation design and should be defined and reported explicitly. 2 Related Work Conversational-memory benchmarks such as LoCoMo, LongMemEval-S, and BEAM have established retrieval- and QA-based evaluation for long-horizon dialogue systems ( 15 ; 29 ; 25 ) . They differ in answer-level scoring: LoCoMo uses lexical metrics such as F1 and BLEU-1 alongside retrieval recall over annotated evidence turns, while LongMemEval-S uses a GPT-4o judge for flexible answer correctness and reports Recall and nDCG when retrieval traces expose answer-location labels. Our audit is upstream of these choices. It asks which stored memory IDs are eligible to satisfy the annotated evidence before either a lexical scorer or a judge evaluates the final answer. Memory-system papers ( 19 ; 31 ; 18 ; 6 ; 10 ; 20 ; 17 ; 24 ; 4 ; 12 ) focus on architecture, storage policy, and memory management. Many systems extract, summarize, or canonicalize conversational evidence before serving it. We study the evaluation target induced by these storage choices: when raw turns and derived memories are linked to the same evidence, which stored IDs should receive credit? The closest analogy is classical IR work on evaluation sensitivity ( 1 ; 30 ; 7 ; 28 ; 27 ; 16 ; 9 ) . Prior work shows that conclusions can be fragile under incomplete judgments ( 2 ; 32 ; 23 ; 3 ) , and document-versus-passage evaluation shows that the scored unit can matter even when the evidence is related ( 30 ; 7 ) . BEIR further shows that retrieval conclusions depend on benchmark design choices ( 26 ) . However, these settings usually vary judgments over fixed units or change the retrieval unit globally. In transformed memory stores, the system itself creates multiple source-linked descendants of the same evidence inside one evaluated index, and the benchmark must decide which descendants are admissible targets. RAG evaluation frameworks such

Read the complete text