HorizonBench : Long-Horizon Personalization with Evolving Preferences

Shuyue Stella Li, Bhargavi Paranjape, Kerem Oktar, Zhongyao Ma, Gelin Zhou, Lin Guan, Na Zhang, Sem Park, Lin Chen, Diyi Yang, Yulia Tsvetkov, Asli Celikyilmaz · 2026

Read HorizonBench : Long-Horizon Personalization with Evolving Preferences by Shuyue Stella Li, Bhargavi Paranjape, Kerem Oktar, Zhongyao Ma, Gelin Zhou, Lin Guan, Na Zhang, Sem Park, Lin Chen, Diyi Yang, Yulia Tsvetkov, Asli Celikyilmaz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17283.txt.

Introduction

Li and eleven colleagues introduce HorizonBench, a benchmark for long-horizon personalization built by a generator producing conversations from a mental state graph, with provenance for every preference change. It holds 4,245 items from 360 users with histories averaging about 4,300 turns and 163,000 tokens. Across 25 models, the best reaches 52.8 percent while most score at or below chance. Errors on evolved preferences pick the original value without tracking the update over a third of the time. This failure persists across context lengths and explicitness levels, marking state-tracking the bottleneck.

Catalog record

CREATORShuyue Stella Li, Bhargavi Paranjape, Kerem Oktar, Zhongyao Ma, Gelin Zhou, Lin Guan, Na Zhang, Sem Park, Lin Chen, Diyi Yang, Yulia Tsvetkov, Asli Celikyilmaz
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59209
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.17283

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HorizonBench : Long-Horizon Personalization with Evolving Preferences Shuyue Stella Li, Bhargavi Paranjape, Kerem Oktar, Zhongyao Ma, Gelin Zhou, Lin Guan, Na Zhang, Sem Park, Lin Chen, Diyi Yang, Yulia Tsvetkov, Asli Celikyilmaz Abstract User preferences evolve across months of interaction, and tracking them requires inferring when a stated preference has been changed by a subsequent life event. We define this problem as long-horizon personalization and observe that progress on it is limited by data availability and measurement, with no existing resource providing both naturalistic long-horizon interactions and the ground-truth provenance needed to diagnose why models fail. We introduce a data generator that produces conversations from a structured mental state graph, yielding ground-truth provenance for every preference change across 6-month timelines, and from it construct HorizonBench , a benchmark of 4,245 items from 360 simulated users with 6-month conversation histories averaging ∼ {\sim} 4,300 turns and ∼ {\sim} 163K tokens. HorizonBench provides a testbed for long-context modeling, memory-augmented architectures, theory-of-mind reasoning, and user modeling. Across 25 frontier models, the best model reaches 52.8% and most score at or below the 20% chance baseline. When these models err on evolved preferences, over a third of the time they select the user’s originally stated value without tracking the updated user state. This belief-update failure persists across context lengths and expression explicitness levels, identifying state-tracking capability as the primary bottleneck for long-horizon personalization. Figure 1: HorizonBench pipeline overview. (1) The generator produces conversations from a structured mental state graph, so every conversation turn is traceable to the otherwise unobservable mental state that produced it. (2) Preferences are established in conversations, then (3) implicitly shifted by evolution events without being restated. (4) Conversation episodes are generated from the updated mental state instance conditioning on event and intent. (5) Pre-evolution preference values serve as hard-negative distractors in the resulting HorizonBench benchmark. See Figure 2 for a complete benchmark item. 1 Introduction Millions of users already interact with AI companions over months of sustained conversation ( 23 ) , and their preferences change throughout these interactions. Consider the user in Figure 1 , who initially prefers Socratic philosophical engagement but shifts to a direct framework style after becoming CEO, without restating the preference. When she later asks whether it is better to be respected or liked, an assistant that retrieves her original Socratic preference has performed retrieval correctly but failed at belief update. Recent work shows that most variance in user behavior is state-driven rather than trait-driven ( 4 ) , yet models and benchmarks alike model only fixed traits. We define the problem of tracking such evolving preferences as long-horizon personalization , the task of identifying a user’s current preference value from a conversation history in which stated preferences have been changed by subsequent life events. Progress in long-horizon personalization is limited by gaps in both data availability and measurement. On the data side, the user’s underlying mental state is never directly observable in real interaction data, so ground truth about when and why preferences changed cannot be known with certainty, and existing synthetic approaches lack causal models of preference evolution ( 18 ; 2 ) . On the measurement side, personalization benchmarks test static preferences ( 24 ; 10 ) or, when they include changing information ( 6 ; 15 ; 11 ; 19 ) , do not provide the experimental controls needed to determine why models fail. When a model answers incorrectly on a dynamic preference, existing evaluations cannot distinguish whether the model never retrieved the preference (retrieval failure) or retrieved the original value but failed to revise it after a life event (belief-update failure). These two failures require different solutions, yet no existing benchmark can tell them apart. Table 1 compares existing work along these dimensions. We address both gaps. The data generator produces conversations from a structured mental state graph where life events drive preference changes through typed dependency edges, making the mapping between preferences and surface behavior known by construction across 6-month interaction histories (Figure 1 ). HorizonBench derives evaluation items from these timelines, yielding 4,245 items from 360 simulated users with 6-month conversation histories averaging {\sim} 4,300 turns and {\sim} 163K tokens. Pre-evolution preference values serve as hard-negative distractors in 5-option questions, a 5-LLM consensus filter retains only items unanswerable without the full conversation history, and three independently controlled dimensions (evolution status, expression explicitness, context length) enable targeted analysis of failure modes. HorizonBench provides a testbed for long-context modeling, memory-augmented architectures, theory-of-mind reasoning, and user modeling. Across the 4,245 items, the best of 25 frontier models (Claude-opus-4.5) achieves 52.8%, and most score at or below the 20% chance baseline. When models err on evolved preferences, all 25 select the pre-evolution distractor at rates significantly above chance ( p<0.001 ) retrieving the user’s originally stated value without considering the life events that changed it. Controlled experiments confirm this pattern persists across context lengths and expression explicitness levels, identifying state tracking as the primary bottleneck for long-horizon personalization. We release the generator, benchmark, and the underlying mental state graph publicly. Our contributions include: 1. A state-first data generator that constructs conversations from a structured mental state graph, inverting the standard approach of inferring mental state from conversations and providing ground-truth provenance for every preference change across 6-month timelines (§ 4 ). 2. HorizonBench , a diagnostic benchmark of 4,245 items from 360 simulated users with 6-month conversation histories averaging {\sim} 4,300 turns and {\sim} 163K tokens, featuring pre-evolution hard-negative distractors, 5-LLM consensus history filtering, and three independently controlled dimensions (§ 5 ). 3. We identify belief-update failure as a specific, targetable deficit that persists across model families, context lengths, and expression explicitness levels (§ 6 ). The controlled dimensions (evolution status, expression explicitness, context length) allow methods and models to be evaluated separately on memory retrieval and belief update. 2 Related Work Personalization benchmarks have established that LLMs struggle with preference retrieval and long-context recall ( 24 ; 5 ; 6 ; 10 ; 11 ; 15 ) , and several treat implicit vs. explicit expression as a primary difficulty axis ( 6 ; 10 ; 3 ) . Recent work reports that dynamic preferences are harder than static ones ( 6 ) , consistent with our findings, but without pre-evolution distractors or controlled experimental dimensions, existing evaluations cannot determine whether these errors reflect retrieval failure or the belief-update anchoring we identify in § 6.2 . No existing benchmark independently varies the mechanisms driving preference change or provides the controls to isolate retrieval from belief update. Long-term memory benchmarks ( 15 ; 11 ; 19 ) evaluate knowledge updating alongside information extraction but do not model why tracked information changes or provide controlled experiments isolating difficulty sources. Our belief-update failure finding connects to the knowledge conflict literature ( 20 ) , where models anchor on prior evidence even when newer context supersedes it. HorizonBench provides the first controlled evidence that this anchoring extends to personalization over long interaction horizons. The data generator builds on LLM-based agent simulation ( 12 ) , extending prior approaches to life-long personalization ( 18 ; 2 ) with a structured mental state graph that enforces preference consistency across 6-month timelines through typed dependency edges and full evolution

Read the complete text