OxyEcomBench : Benchmarking Multimodal Foundation Models across E-Commerce Ecosystems
Yong Liu, Ximan Liu, Guoqing Yang, Bing Bai, Xiaoqiang Xu, Zhen Chen, Ke Zhang, Yan Li · 2026
Read OxyEcomBench : Benchmarking Multimodal Foundation Models across E-Commerce Ecosystems by Yong Liu, Ximan Liu, Guoqing Yang, Bing Bai, Xiaoqiang Xu, Zhen Chen, Ke Zhang, Yan Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13173.txt.
Introduction
Liu and seven co-authors introduce OxyEcomBench, a multimodal benchmark of about 6,300 instances for bilingual Chinese-English e-commerce. Existing e-commerce benchmarks take one stakeholder view or a narrow task set, missing the full pipeline. OxyEcomBench covers operators, merchants and customers across 6 capability aspects and 29 tasks, with text-only and mixed inputs. Data come from authentic platforms, verified by domain experts, with a four-level difficulty rubric. Even leading models score modestly on it, and gaps between models narrow.
Catalog record
| CREATOR | Yong Liu, Ximan Liu, Guoqing Yang, Bing Bai, Xiaoqiang Xu, Zhen Chen, Ke Zhang, Yan Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46358 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.13173
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OxyEcomBench : Benchmarking Multimodal Foundation Models across E-Commerce Ecosystems Yong Liu, Ximan Liu, Guoqing Yang, Bing Bai, Xiaoqiang Xu, Zhen Chen, Ke Zhang, Yan Li Abstract Large language models (LLMs) and multimodal large language models (MLLMs) have become indispensable tools across a wide range of applications. E-commerce, however, poses distinctive challenges—including intricate domain knowledge, long-tail product evidence, heterogeneous visual data, and the interplay among multiple stakeholder roles—that diverge substantially from the general world knowledge these models are primarily trained on, often causing a notable gap between their open-domain and e-commerce performance. To systematically quantify this gap, we introduce OxyEcomBench, a unified multimodal benchmark comprising approximately 6,300 high-quality instances for real-world bilingual Chinese–English e-commerce. Although several e-commerce benchmarks have been proposed, they typically adopt a single stakeholder perspective, target a narrow set of tasks, or address isolated challenges (e.g., visual salience or multi-turn dialogues independently), making it difficult to holistically assess models’ understanding of the full e-commerce pipeline. OxyEcomBench addresses these limitations by jointly covering platform operators, merchants, and customers across 6 capability aspects and 29 tasks, supporting text-only and mixed-modality inputs with single-image, multi-image, single-turn, and multi-turn configurations. All data is sourced from authentic e-commerce platforms and verified by domain experts. The benchmark further adopts a difficulty-aware design with a four-level P0–P3 rubric applied to all 29 tasks whose difficulty admits stable expert consensus, and rigorously prioritizes visually salient multimodal cases in which key evidence resides in images rather than text alone. Evaluations on 20 mainstream LLMs and MLLMs show that even the leading models attain modest performance and that performance gaps narrow on OxyEcomBench, suggesting that insufficient e-commerce-specific knowledge infusion mutes the advantages of advanced general-purpose models in this domain. 1 Introduction Large language models (LLMs) ( 1 ) and multimodal large language models (MLLMs) ( 19 ; 3 ) have demonstrated remarkable capabilities across a wide range of tasks. However, deploying them in highly specialized and complex real-world scenarios, such as e-commerce, presents distinct challenges. As a data-rich and economically significant domain, e-commerce is characterized by platform-specific policies, long-tail product knowledge, and heterogeneous visual evidence. Critically, the e-commerce ecosystem involves three primary stakeholder roles with fundamentally different needs: platform operators who manage marketplace infrastructure, policies, and user experience; merchants who list products, handle orders, and manage post-sale operations; and customers who search, evaluate, and purchase products. This intricate multi-role interplay, combined with the requirement for deep domain expertise, diverges substantially from the general world knowledge these models are primarily trained on, often causing a notable performance gap in real business workflows. Figure 1 : Taxonomy of OxyEcomBench . The inner ring represents the three stakeholder roles—platform, merchant, and customer—as independent evaluation perspectives. The middle ring groups the benchmark into 6 capability aspects, and the outer ring enumerates 29 distinct e-commerce tasks, with each task inheriting the color of its corresponding capability aspect. Recent efforts have made commendable progress in benchmarking LLMs and MLLMs for e-commerce, yet they typically focus on isolated dimensions of the domain. For instance, several benchmarks predominantly adopt a single stakeholder perspective, such as consumer-oriented tasks ( 15 ) or customer support scenarios ( 31 ; 33 ) . Concurrently, other pioneering works have begun to explore specific technical challenges, such as multi-turn conversational interactions ( 21 ) or the necessity of visual salience in multimodal tasks. While these targeted evaluations provide valuable insights, they remain largely fragmented. Real-world e-commerce workflows rarely occur in isolation; they frequently involve complex, multi-turn dialogues coupled with visually salient evidence, spanning the diverse and intertwined needs of platform operators, merchants, and customers. Consequently, the community still lacks a holistic evaluation framework that captures the full complexity of the e-commerce ecosystem. To the best of our knowledge, existing benchmarks largely fall short of simultaneously addressing the distinct evaluation requirements of multiple stakeholder roles within a unified framework. Furthermore, many existing datasets lack fine-grained, expert-calibrated difficulty annotations, making it challenging to determine whether a model’s success stems from genuine multi-step reasoning and cross-modal integration, or merely from solving simplistic, surface-level queries. To systematically quantify the gap between open-domain capabilities and specialized e-commerce requirements, we present OxyEcomBench . Unlike previous datasets that target isolated challenges, OxyEcomBench is designed to holistically assess models’ understanding of the full e-commerce pipeline. As summarized in Figure 1 , the benchmark organizes 3 stakeholder roles, 6 capability aspects, and 29 tasks into a unified taxonomy for real-world e-commerce evaluation. It comprises approximately 6,300 high-quality, bilingual Chinese–English instances spanning these capability aspects and tasks, with each instance manually inspected by e-commerce experts. The dataset rigorously prioritizes visually salient multimodal cases and authentic multi-turn interactions, with all data sourced from real-world e-commerce platforms to ensure ecological validity. Our main contributions are as follows: • Comprehensive Multi-Role Evaluation: We introduce OxyEcomBench , the first benchmark to holistically integrate platform, merchant, and customer perspectives, enabling a complete assessment of foundation models across the entire e-commerce ecosystem. • Rigorous Multimodal & Interactive Assessment: By curating visually salient tasks and authentic multi-turn dialogues, the benchmark effectively prevents text-only shortcuts and evaluates models on complex, real-world multimodal reasoning. In addition, we establish a reliable four-level (P0–P3) difficulty rubric based on expert consensus, providing a nuanced tool to differentiate between surface-level pattern matching and deep cognitive reasoning. • Domain-Specific Empirical Insights: Extensive evaluations on mainstream LLMs and MLLMs show that even leading models attain modest performance and that performance gaps narrow on OxyEcomBench , indicating that insufficient e-commerce-specific knowledge infusion can mute the advantages of advanced general-purpose models. 2 Related Work 2.1 General Multimodal Benchmarks The rapid progress of multimodal large language models (MLLMs) has been mirrored by a rich landscape of evaluation suites, ranging from early perception and single-hop reasoning benchmarks (VQA ( 2 ) , COCO Captions ( 7 ) , OK-VQA ( 24 ) , TextVQA ( 28 ) ) and comprehensive multi-task suites (MME ( 8 ) , MMBench ( 20 ) , MM-Vet ( 37 ) , TouchStone ( 4 ) , LVLM-eHub ( 34 ) , LAMM ( 36 ) ), to expert-level reasoning tests (ScienceQA ( 23 ) , MathVista ( 22 ) , MMMU ( 38 ) , CMMU ( 13 ) , ChartQA ( 25 ) ), multi-image and interleaved suites (BLINK ( 10 ) , Mantis ( 14 ) , MUIRBench ( 30 ) ), and reliability-oriented diagnostics targeting hallucination and visual salience (POPE ( 17 ) , HallusionBench ( 12 ) , SEED-Bench ( 16 ) , MMStar ( 6 ) , MMHal-Bench ( 29 ) , AMBER ( 32 ) ). While these general-domain benchmarks offer broad coverage of world knowledge, multi-image reasoning, and visual dependency, they lack the specialized domain expertise—platform-specific policies, long-tail product attributes, and multi-role interactions—required for industrial e-commerce, so strong general-benchmark performance does not necessarily translate into reliability on real-world e-commerce scenarios. 2.2 E-commerce Specific Benchmarks Recognizing the limits of general-domain evaluations, the community has introduced benchmarks tailored to the e-commerce domain. However, these pioneering efforts typically focus on isolated dimensions of the ecosystem, leaving significant gaps in holistic evaluation. Concept Understanding and Text-centric Interactions. Early e-commerce benchmarks predominantly focused on text-based concept comprehension and single-turn interactions. For instance, Shopping MMLU ( 15 ) introduced a comprehensive suite for