RetailBench: Evaluating Long-Horizon Autonomous Decision-Making and Strategy Stability of LLM Agents in Realistic Retail Environments
Linghua Zhang, Jun Wang, Jingtong Wu, Zhisong Zhang · 2026
Read RetailBench: Evaluating Long-Horizon Autonomous Decision-Making and Strategy Stability of LLM Agents in Realistic Retail Environments by Linghua Zhang, Jun Wang, Jingtong Wu, Zhisong Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16453.txt.
Introduction
Zhang and co-authors test language model agents as long-run store managers. Short tasks hide whether agents can hold a steady strategy. RetailBench simulates one supermarket over very long horizons. Agents set prices, restock, pick suppliers, and watch cash flow. Seven models ran for 180 days under agent frameworks. Only a few survived the full run. Even the best trailed a privileged oracle policy by far. Weak evidence gathering explains much of the gap.
Catalog record
| CREATOR | Linghua Zhang, Jun Wang, Jingtong Wu, Zhisong Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70539 |
| LICENSE | http://creativecommons.org/publicdomain/zero/1.0/ |
| RIGHTS | CC0 |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.16453
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RetailBench: Evaluating Long-Horizon Autonomous Decision-Making and Strategy Stability of LLM Agents in Realistic Retail Environments Linghua Zhang, Jun Wang, Jingtong Wu, Zhisong Zhang Abstract Large Language Model (LLM)-based agents have achieved notable success on short-horizon and highly structured tasks, yet their ability to maintain coherent decision-making over long horizons in realistic and dynamic environments remains an open challenge. We introduce RetailBench , a high-fidelity benchmark designed to evaluate long-horizon autonomous decision-making in realistic commercial scenarios, where agents must operate under stochastic demand and evolving external conditions. We further propose the Evolving Strategy & Execution framework, which separates high-level strategic reasoning from low-level action execution, enabling adaptive and interpretable strategy evolution over time. This design is crucial for long-horizon tasks, where non-stationary environments and error accumulation require strategies to be revised at a different temporal scale than action execution. Experiments on eight state-of-the-art LLMs across progressively challenging environments show that our framework improves operational stability and efficiency compared to other baselines. However, performance degrades substantially as task complexity increases, revealing fundamental limitations in current LLMs for long-horizon, multi-factor decision-making. 1 Introduction Recent large language models (LLMs), particularly when augmented with reasoning and tool-use capabilities, have demonstrated strong performance on a variety of cognitively demanding tasks, including code editing, mathematical problem solving, and complex information retrieval ( 14 ; 25 ; 11 ) . However, accumulating empirical evidence indicates that such capabilities fail to generalize into robust, domain-agnostic autonomy, especially in realistic settings requiring long-horizon planning, persistent objective alignment, and stable behavioral consistency—core prerequisites for participation in real-world economic systems ( 1 ; 16 ; 20 ) . Correspondingly, existing agent benchmarks—spanning web interaction ( 21 ; 35 ; 40 ; 7 ; 14 ; 31 ) primarily focus on short-horizon or highly structured tasks, limiting their ability to evaluate sustained interaction with complex, evolving environments. Recent studies on long-horizon autonomy consistently demonstrate that even state-of-the-art agents struggle to maintain coherent strategies over extended time spans ( 22 ; 2 ; 5 ) . Figure 1: Overview of the hierarchical supermarket environment, illustrating intra-day agent–environment interactions and end-of-day state transition dynamics. To systematically study this challenge, we introduce RetailBench , a new benchmark grounded in real-world commercial data and informed by established economic modeling principles.RetailBench centers on a supermarket operation scenario that demands long-horizon decision-making, sustained interaction with a dynamic environment, and the integration of heterogeneous historical information. The benchmark evaluates whether LLM-based agents can autonomously sustain realistic business operations under complex, multi-factor conditions. We evaluate eight state-of-the-art LLMs in this environment. Our results show that current models struggle to maintain stable decision quality as the decision space expands and often fail to incorporate all relevant information. Moreover, hallucinations and economically irrational behaviors frequently emerge during long-horizon execution, leading to environment collapse and preventing sustained autonomous operation. Our contributions are summarized as follows: • We introduce RetailBench , a high-fidelity benchmark for evaluating long-horizon autonomous decision-making in realistic retail environments. • We propose the Evolving Strategy & Execution agent framework, which improves operational stability compared to a Reflection-based baseline. • Through extensive experiments, we identify systematic failure modes of current LLM-based agents in long-horizon, multi-factor decision-making settings. 2 Environment Construction 2.1 Problem Formulation and Overview We model supermarket operations as a Markov Decision Process (MDP), in which an autonomous agent manages a single retail store over a finite horizon of days. At each day t , the agent makes a sequence of operational decisions that jointly determine the store’s daily outcomes. Formally, the MDP is defined as (\mathcal{S},\mathcal{A},\mathcal{T},R,\gamma) , where \mathcal{S} denotes the state space, \mathcal{A} the action space, \mathcal{T} the stochastic transition dynamics, R the reward function, and \gamma\in(0,1] the discount factor. At the beginning of each day t , the agent observes the initial state S_{t,0} and executes a sequence of intra-day actions indexed by k . Each action a_{t,k}\in\mathcal{A} induces a transition to the next intra-day state S_{t,k+1} . When the agent issues the end-today action, the environment transitions to the initial state of the next day, S_{t+1,0} , according to the transition dynamics \mathcal{T} . Figure 1 illustrates the overall interaction process. The environment supports long-horizon operation over more than one thousand simulated days, with episodes terminating if the store fails to pay rent for five consecutive days. 2.2 State Space The intra-day state S_{t,k} summarizes the complete operational context of the store at step k of day t and is composed of multiple interdependent components: S_{t,k}=\big(S_{t,k}^{\text{prod}},S_{t,k}^{\text{inv}},S_{t,k}^{\text{sup}},S_{t,k}^{\text{dem}},S_{t,k}^{\text{ext}},S_{t,k}^{\text{fin}}\big) . • S_{t,k}^{\text{prod}} and S_{t,k}^{\text{inv}} encode product-level attributes and on-hand inventory status, including prices, shelf life, and historical sales records. Product demand is grounded in real-world retail data derived from the Dominick’s dataset ( 15 ) . • S_{t,k}^{\text{sup}} represents the supply chain state, including supplier prices, quality levels, and delivery lead times, constructed to reflect empirically observed price–quality relationships ( 13 ) . • S_{t,k}^{\text{dem}} captures demand-side signals such as recent customer traffic and aggregated review statistics, which influence consumer purchasing behavior ( 10 ) . • S_{t,k}^{\text{ext}} represents external contextual information, including active news events with market-, category-, or product-level impact scope, synthesized to resemble real-world financial news ( 4 ) . • S_{t,k}^{\text{fin}} denotes the financial state of the store, including available cash and estimated net worth, accounting for inventory depreciation over product shelf life. Together, these components define a richly structured yet partially stochastic state space. Further details of the environment design are deferred to Appendix A . 2.3 Action Space At each time step t , the agent selects an action a_{t}\in\mathcal{A} , defined as a tuple of decision components: a_{t}=\big(a_{t}^{\text{price}},a_{t}^{\text{repl}},a_{t}^{\text{info}},a_{t}^{\text{mem}},a_{t}^{\text{end}}\big) . Each component corresponds to a distinct class of operational decisions: • a_{t}^{\text{price}} denotes pricing decisions for individual SKUs; • a_{t}^{\text{repl}} denotes inventory replenishment decisions, including supplier selection and order quantities; • a_{t}^{\text{info}} denotes information acquisition actions, such as querying sales history, customer reviews, supplier conditions, or current news; • a_{t}^{\text{mem}} denotes memory operations that allow the agent to write and retrieve persistent notes across days; • a_{t}^{\text{end}} denotes a day-termination action that concludes the current decision phase. Within a single day, the agent may execute multiple information queries and operational adjustments before issuing the a_{t}^{\text{end}} action to advance the environment to the next day. 2.4 Day Transition Dynamics After the agent issues the day-termination action, the environment transitions to the next day according to S_{t,k+1}\sim\mathcal{T}(\cdot\mid S_{t,k},a_{t}) , where the transition function \mathcal{T} factorizes into a sequence of stochastic updates that jointly model daily supermarket operations. Specifically, each day-level transition consists of the following steps: 1. Customer traffic sampling. The total customer traffic for the day is sampled according to stochastic demand dynamics. 2. Sales realization. Given the sampled customer traffic and exogenous factors (e.g., promotions, seasonal effects, and news signals), the realized sales volume of each product is determined. 3. Reviews and returns generation. Customer reviews and product return events are generated conditional on sales outcomes, product quality, and external influences. 4. Inventory update. Sold units are deducted from on-hand inventory, and replenishment orders scheduled to arrive on the current day are added to stock. 5. Financial and exogenous state update. The agent’s financial state is updated based on realized revenues and costs, after which new exogenous information (e.g., news or market signals) for