ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents

Chinmay Savadikar, Mingyu Zhao, Yuanzheng Zhu, Han Li, Shuang Xie, Alberto Castelo, Tianfu Wu, Lingyun Wang, [0.5em] North Carolina State University · 2026

Read ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents by Chinmay Savadikar, Mingyu Zhao, Yuanzheng Zhu, Han Li, Shuang Xie, Alberto Castelo, Tianfu Wu, Lingyun Wang, [0.5em] North Carolina State University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16116.txt.

Introduction

Savadikar and seven colleagues introduce ShopGym, a testbed for e-commerce web agents that is both lifelike and repeatable. Live shops are lifelike but unstable and hard to rerun, while hand-built sandboxes cover too little ground. ShopArena turns live seed shops into self-run sandbox shops, and ShopGuru builds benchmark tasks across seven skill groups. Together they yield shops that reset, can be inspected, and stay stable, while keeping shopping-task structure.

Catalog record

CREATORChinmay Savadikar, Mingyu Zhao, Yuanzheng Zhu, Han Li, Shuang Xie, Alberto Castelo, Tianfu Wu, Lingyun Wang, [0.5em] North Carolina State University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE93093
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.16116

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents Chinmay Savadikar, Mingyu Zhao, Yuanzheng Zhu, Han Li, Shuang Xie, Alberto Castelo, Tianfu Wu, Lingyun Wang, [0.5em] North Carolina State University Abstract Developing and evaluating e-commerce web agents requires environments that preserve meaningful task structure while enabling controllable, reproducible, and scalable scientific comparison. Existing methodologies force a tradeoff: live storefronts provide realism but are non-stationary, difficult to inspect, and irreproducible, while hand-built sandbox benchmarks provide control but cover only a narrow range of layouts, catalogs, policies, and interaction patterns. We argue that the core bottleneck is methodological: the field lacks a scalable way to construct evaluation settings that are simultaneously realistic, diverse, controllable, inspectable, and reproducible. We introduce ShopGym, an integrated framework for realistic simulation and scalable benchmarking of e-commerce web agents. ShopGym is a framework for constructing e-commerce simulation environments and grounded benchmark tasks. Its simulation layer, ShopArena, converts live seed storefronts into self-contained sandbox shops through anonymized shop specifications and a staged, validated generation process. On top of these simulated storefronts, ShopGuru synthesizes benchmark tasks across seven skill categories, grounding each task in the shop’s catalog, navigation structure, policies, and interaction affordances. Together, ShopArena and ShopGuru produce self-contained, resettable, inspectable, and stable evaluation artifacts that preserve structural properties and agent-evaluation signals relevant to shopping tasks. We validate the framework through graph-based structural analysis and agent-based behavioral evaluation with 224 generated tasks across six sandbox shops: three constructed with synthetic data and three with real data. Our results show that the synthetic shops preserve key structural properties of live storefronts, with agent performance on synthetic shops positively correlated with performance on live storefronts. 1 Introduction As web agents become more capable, the field increasingly needs stable environments that support long-horizon interaction in realistic, user-facing settings. E-commerce is a particularly demanding domain for this purpose: shopping tasks require agents to navigate multi-page interfaces, interpret product information, satisfy user constraints, compare alternatives, and execute actions such as selecting variants or adding items to cart. These tasks combine perception, reasoning, and sequential decision-making in a grounded setting, making storefronts a compelling testbed for web-agent evaluation. The importance of this domain is reflected in growing interest in shopping assistants [ 23 , 25 , 17 ] , offline A/B testing [ 2 , 15 ] , and user behavior simulation for e-commerce applications [ 10 , 23 , 26 , 20 ] . Existing environments for e-commerce agents face a fundamental tradeoff between realism and experimental control. Live-websites [ 21 , 7 , 11 ] offer strong realism: agents are run and evaluated against realistic layouts, and naturally occurring variation and interaction patterns. But live storefronts are inherently non-stationary. Catalogs, layouts, and webpage design on live storefronts can change, collections can be reorganized, and storefronts participating in live A/B testing can serve separate variants for different requests. This non-stationary behavior makes reproducible training and evaluation of web agents challenging. The evaluation performance can reflect not only agent capability but also incidental variation from website drift, location-specific behavior, and operational noise, making results difficult to reproduce and agents difficult to compare fairly [ 21 ] , and training runs difficult to reproduce. Sandbox environments [ 28 , 8 , 22 , 16 , 18 , 12 ] , in contrast, are controllable and repeatable by design, but at the cost of realism and diversity of live websites. Manually constructing realistic storefront environments is expensive and covers only a narrow range of layouts, catalog structures, and interaction patterns. Existing work proposing environments for e-commerce agents has largely explored the two methodologies, with works like DeepShop [ 11 ] using live websites, and WebShop [ 22 ] , ShoppingBench [ 16 ] , ShopSimulator [ 18 ] , and WebMall [ 12 ] manually building sandbox environments. We argue that the deeper bottleneck is methodological : the field lacks a scalable way to construct reliable simulated environments that are realistic, controllable, inspectable, and reproducible . To address this gap, we propose ShopGym , a framework for constructing realistic e-commerce simulation environments and grounded web-agent benchmarks. ShopGym comprises two complementary components in a synergistic workflow: (a) the simulation environment layer, ShopArena , transforms one or more live seed storefronts into a self-contained sandbox shop; and (b) ShopGuru synthesizes benchmark tasks grounded in the sandbox shops. Figure 1 : ShopGym comprises two components. ShopArena provides a simulation environment populated with synthetic sandbox shops, along with a scalable pipeline that generates new sandbox shops from one or more live seed storefronts through specification synthesis followed by data and code generation. ShopGuru then consumes the resulting catalog, collections, pages, and shop statistics to generate both short-horizon tasks covering primitive skills and long-horizon shopping journeys that combine these skills. ShopArena: validated simulation environment. To generate the simulation environment grounded in real storefronts, ShopArena separates storefront understanding (exploration) from sandbox synthesis (generation) through an intermediate specification document that serves as the sole interface between exploration and generation. Given one or more seed storefronts, ShopArena autonomously explores the sites and writes an anonymized specification capturing their navigation structure, user-facing affordances, high-level page organization, and catalog statistics. When multiple seeds are provided, ShopArena composes their structural and behavioral signals into a single specification, so one sandbox shop can span diversity that no individual source storefront would cover. A staged generation pipeline then synthesizes a runnable sandbox shop from the specification, iteratively refining the implementation through build checks, type checking, and multimodal visual validation. ShopArena does not aim to clone a particular storefront; it produces behaviorally aligned approximations – synthetic shops grounded in real storefront structure and interaction patterns, but with synthetic brand identity, products, descriptions, and imagery. The goal is not visual or brand-level fidelity but structural, behavioral, and statistical alignment sufficient for controlled evaluation. Because the specification is human-readable, benchmark creators can also edit generated environments in controlled ways without re-exploring the source storefronts . ShopGuru: grounded task generation. ShopGuru consumes the sandbox shop’s catalog, navigation graph, and store policies and synthesizes tasks that are valid against the environment they will be executed in. This design separates live-web exploration from benchmark execution. The generated shops are stable and resettable, while the intermediate specification provides an inspectable control surface that benchmark creators can edit without re-exploring the source storefronts. We evaluate ShopGym by generating synthetic sandbox shops and anonymized twin shops, then validating them using structural metrics and agent-based behavioral comparisons. Across paired tasks defined over source and generated shops, agent success rates on sandbox environments positively correlate with success rates on live storefronts, indicating that generated storefronts retain meaningful evaluative signal from live-web settings while substantially improving control and reproducibility. While it does not completely replace live-web evaluation, ShopGym provides a stronger methodology for standardized benchmarking and analysis. To summarize, our contributions are: • We identify the realism-control tradeoff in e-commerce environments and agent evaluation as a methodological bottleneck, and argue that the missing capability is scalable construction of realistic, controllable, and reproducible environments and evaluation benchmarks. • We bridge this gap with ShopGym , a framework for constructing realistic, controlable, and reproducible e-commerce agent evaluation environments. • We present ShopArena , a pipeline that converts live seed storefronts

Read the complete text