Less is More: Benchmarking LLM Based Recommendation Agents

Kargi Chauhan, Mahalakshmi Venkateswarlu · 2026

Read Less is More: Benchmarking LLM Based Recommendation Agents by Kargi Chauhan, Mahalakshmi Venkateswarlu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20316.txt.

Introduction

Kargi Chauhan and Mahalakshmi Venkateswarlu challenge a common belief about recommendation agents built on large language models: that longer purchase histories give better predictions. They benchmark four leading models, GPT-4o-mini, DeepSeek-V3, Qwen2.5-72B, and Gemini, on product recommendation.

Their systematic test asks how much history length actually helps. The title gives their headline: less is more. If short histories predict as well as long ones, services can store less personal data and answer faster. The result also warns builders that piling up user history can hurt rather than help. The test is plain, and the point is sharp.

Catalog record

CREATORKargi Chauhan, Mahalakshmi Venkateswarlu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-20316

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Less is More: Benchmarking LLM Based Recommendation Agents Kargi Chauhan, Mahalakshmi Venkateswarlu Abstract . Large Language Models (LLMs) are increasingly deployed for personalized product recommendations, with practitioners commonly assuming that longer user purchase histories lead to better predictions. We challenge this assumption through a systematic benchmark of four state of the art LLMs GPT-4o-mini, DeepSeek-V3, Qwen2.5-72B, and Gemini 2.5 Flash across context lengths ranging from 5 to 50 items using the REGEN dataset. Surprisingly, our experiments with 50 users in a within subject design reveal no significant quality improvement with increased context length. Quality scores remain flat across all conditions (0.17–0.23). Our findings have significant practical implications: practitioners can reduce inference costs by approximately 88% by using context (5–10 items) instead of longer histories (50 items), without sacrificing recommendation quality. We also analyze latency patterns across providers and find model specific behaviors that inform deployment decisions. This work challenges the existing “more context is better” paradigm and provides actionable guidelines for cost effective LLM based recommendation systems. 1. Introduction Large Language Models (LLMs) have emerged as powerful tools for personalized recommendations, capable of understanding user preferences from purchase histories and generating contextually relevant product suggestions ( fan2023recommender ; wu2024survey ) . Unlike traditional collaborative filtering approaches that rely on learned embeddings, LLM-based recommenders can directly process rich textual descriptions of user history, item metadata, and contextual information. As recommendation systems evolve into agentic ecosystems capable of planning, reasoning, and multi-turn interaction, the ability to efficiently manage context becomes foundational for scalable and responsive personalization ( wang2023recmind ; zhang2024agent4rec ) . A common assumption among practitioners is that providing more user history longer context leads to better recommendation quality. This belief drives the use of extended context windows, often including dozens or even hundreds of past interactions. However, this assumption remains largely untested in recommendation scenarios. While LLMs have demonstrated impressive capabilities in utilizing long contexts for tasks like document summarization and question answering, their ability to leverage extended user histories for product recommendations is unclear ( liu2024lost ) . Moreover, longer contexts come with significant costs: increased API expenses, higher latency, and greater computational requirements. This question has critical implications for deploying recommendation agents. More context may help the LLM better understand user preferences, but excessive history could introduce noise or cause the model to lose focus on recent, relevant interactions ( quadrana2018sequence ) . Longer prompts require more computation, directly impacting response times in real time recommendation scenarios. API based LLM services typically charge per token, making context length a direct cost driver for large scale deployments. Additionally, LLM context windows are a finite resource; identifying the minimal sufficient history enables autonomous agents to preserve capacity for multi turn reasoning and long horizon planning ( wang2023recmind ) . In this paper, we systematically investigate the relationship between context length and recommendation quality across four state of the art LLMs. Our research question is straightforward yet critical: Does providing more user purchase history improve LLM based product recommendation quality? To answer this question, we conduct controlled experiments using the REGEN dataset ( su2025regen ) , testing context lengths of 5, 10, 15, 25, and 50 items across GPT-4o-mini, DeepSeek-V3, Qwen2.5-72B, and Gemini 2.5 Flash. We employ a within subject design where the same 50 users are evaluated at all context lengths, ensuring fair comparisons. Our key contributions and findings are: (1) Systematic Benchmark : We design and execute experiments measuring recommendation quality and latency across multiple context lengths using four production LLMs from different providers. (2) Flat Quality Curves : All four models show no significant quality improvement as context length increases from 5 to 50 items. Quality scores remain in the 0.17–0.23 range with overlapping confidence intervals. (3) Universal Pattern : This finding holds across models from four different providers (OpenAI, Google, Alibaba, DeepSeek), suggesting a fundamental limitation rather than model-specific behavior. (4) Cost Implications : Token usage increases approximately 8 \times when moving from 5 to 50 items, while quality remains unchanged. Practitioners can save up to 88% on inference costs by using minimal context. (5) Latency Analysis : We identify model specific latency patterns that inform deployment decisions for real time recommendation scenarios. These findings challenge the “more context is better” paradigm and provide evidence based guidelines for deploying cost effective LLM based recommendation systems. As LLM based agents become central to recommendation ecosystems, understanding their computational efficiency is critical for scalable deployment. Our results show that LLM-based recommendation systems work just as well with 5–10 items of user history as with 50 items, enabling significant cost savings and faster response times for large scale deployments. 2. Related Work 2.1. LLMs for Recommendations The integration of LLMs into recommendation systems has gained significant attention in recent years. Fan et al. ( fan2023recommender ) provide a comprehensive survey of LLM based recommender systems, highlighting their potential for understanding user intent and generating explanations. P5 ( geng2022p5 ) unified multiple recommendation tasks through a text to text framework, demonstrating that a single language model can handle rating prediction, sequential recommendation, and explanation generation. TALLRec ( bao2023tallrec ) showed that LLMs can be effectively tuned for recommendation with limited training data, achieving competitive performance against specialized models. Hou et al. ( hou2024large ) demonstrate that LLMs can achieve competitive performance on sequential recommendation tasks when properly prompted. More recently, agent based approaches have emerged. RecMind ( wang2023recmind ) frames recommendation as a planning problem where an LLM agent reasons over user preferences. Agent4Rec ( zhang2024agent4rec ) simulates user recommender interactions to evaluate recommendation strategies. Recent work has also explored conversational recommendation systems where LLMs engage in multi-turn dialogues to elicit user preferences ( friedman2023leveraging ) . InstructRec ( zhang2023instructrec ) demonstrated that instruction tuned LLMs can follow natural language preference descriptions to generate recommendations. Additionally, Chat-Rec ( gao2023chatrec ) proposed augmenting LLMs with traditional recommender systems to combine the strengths of both paradigms. However, most existing work focuses on demonstrating LLM capabilities rather than systematically studying the factors that influence recommendation quality. Our work fills this gap by specifically investigating the role of context length. 2.2. Why Doesn’t More Context Help? Our findings challenge the intuitive assumption that more user history should lead to better recommendations. Several factors may explain this pattern. First, the Lost in the Middle phenomenon identified by Liu et al. ( liu2024lost ) suggests that LLMs struggle to effectively utilize information positioned in the middle of long contexts. When context grows from 5 to 50 items, most information falls into this underutilized region. Second, recency bias may render older purchases less predictive of future behavior. The most recent 5 items likely capture a user’s current interests as effectively as 50 items, consistent with patterns observed in sequential recommendation research ( quadrana2018sequence ) . Third, signal saturation implies diminishing returns to additional context. The first few items may sufficiently establish user preferences, with additional items contributing noise rather than signal. Finally, task difficulty imposes a ceiling on performance. Predicting the exact next product purchase is inherently challenging; even perfect context utilization may not yield substantial quality improvements. 2.3. Sequential Recommendation Traditional sequential recommendation methods like SASRec ( kang2018sasrec ) and BERT4Rec ( sun2019bert4rec ) use self-attention to

Read the complete text