Market-Bench: Benchmarking Large Language Models on Economic and Trade Competition

Yushuo Zheng, Huiyu Duan, Zicheng Zhang, Yucheng Zhu, Xiongkuo Min, Guangtao Zhai · 2026

Read Market-Bench: Benchmarking Large Language Models on Economic and Trade Competition by Yushuo Zheng, Huiyu Duan, Zicheng Zhang, Yucheng Zhu, Xiongkuo Min, Guangtao Zhai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05523.txt.

Introduction

Zheng and five colleagues benchmark language models as competing retailers in a simulated supply chain, bidding for scarce stock and setting prices and slogans. Testing 20 models reveals wide gaps and a winner-take-most pattern: few retailers steadily grow capital while most hover near break-even despite similar semantic scores. The benchmark logs full bid, price, and balance-sheet trails for economic study.

Catalog record

CREATORYushuo Zheng, Huiyu Duan, Zicheng Zhang, Yucheng Zhu, Xiongkuo Min, Guangtao Zhai
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38726
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.05523

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Market-Bench: Benchmarking Large Language Models on Economic and Trade Competition Yushuo Zheng, Huiyu Duan, Zicheng Zhang, Yucheng Zhu, Xiongkuo Min, Guangtao Zhai Abstract The ability of large language models (LLMs) to manage and acquire economic resources remains unclear. In this paper, we introduce Market-Bench , a comprehensive benchmark that evaluates the capabilities of LLMs in economically-relevant tasks through economic and trade competition. Specifically, we construct a configurable multi-agent supply chain economic model where LLMs act as retailer agents responsible for procuring and retailing merchandise. In the procurement stage, LLMs bid for limited inventory in budget-constrained auctions. In the retail stage, LLMs set retail prices, generate marketing slogans, and provide them to buyers through a role-based attention mechanism for purchase. Market-Bench logs complete trajectories of bids, prices, slogans, sales, and balance-sheet states, enabling automatic evaluation with economic, operational, and semantic metrics. Benchmarking on 20 open- and closed-source LLM agents reveals significant performance disparities and winner-take-most phenomenon, i.e. , only a small subset of LLM retailers can consistently achieve capital appreciation, while many hover around the break-even point despite similar semantic matching scores. Market-Bench provides a reproducible testbed for studying how LLMs interact in competitive markets. 1 Introduction Figure 1: Existing LLM benchmarks focus on either semantic complexity or quantitative competition, but rarely both simultaneously under economic scarcity. Thus, we propose Market-Bench, coupling marketing slogans and operations to jointly evaluate mathematical optimization and language comprehension. The advancement of large language models (LLMs) has driven the application of AI in retail from passive analytics to active supply chain management and personalized marketing 9 ; 33 , requiring both the computational precision to manage scarce resources and the semantic flexibility to construct persuasive narratives. Marketing language is a decisive economic variable that stimulates purchase intentions by arousing consumers’ latent motivations 25 ; 26 , and recent work has begun to formalize the aesthetic evaluation of commercial imagery ( 19 ) . However, current LLM benchmarks largely evaluate these capabilities in isolation. As shown in Figure 1 , existing frameworks typically focus either on abstract strategic rationality (“Homo Economicus”) or open-ended social role-play (“Homo Loquens”), rarely demanding both simultaneously. Social simulations 30 ; 31 model free-form dialogue but impose no hard economic constraints such as scarcity or bankruptcy, while economic simulators 15 ; 28 test rigorous quantitative logic but abstract away linguistic complexity. Although LLM evaluation has expanded rapidly across visual quality assessment ( 12 ; 14 ; 13 ; 43 ; 46 ; 24 ; 40 ) , multimodal reasoning ( 45 ; 44 ; 42 ; 27 ; 47 ; 48 ) , and domain-specific fields such as medicine ( 20 ; 23 ; 21 ; 22 ; 41 ) , no existing benchmark unifies multi-turn buyer–seller interactions across both numeric and semantic dimensions under strict constraintsthat is, a testbed for dual-process reasoning within a closed-loop economy remains absent. This raises a fundamental question: Can contemporary LLMs effectively perform quantitative reasoning and semantic adaptation to survive in markets characterized by scarcity and competition? The market domain is uniquely challenging because optimal pricing and procurement fail without persuasive marketing, while strong marketing fails if budgeting or inventory is mismanaged,a strict co-dependency absent from benchmarks that test either capability alone. Moreover, agents must infer hidden buyer preferences from limited numerical signals under multi-turn competitive dynamics, where errors compound into bankruptcy rather than minor accuracy drops. We introduce Market-Bench , a configurable multi-agent supply-chain economy formulated as a Partially Observable Markov Game with financial and physical constraints. At each step, agents receive structured text observations,private funds and inventory, auction parameters, and public market history,and output strictly formatted JSON combining discrete mathematical actions (procurement bids, retail prices) with a free-form marketing slogan. Crucially, we propose Persona-Gated Attention (PGA), a mechanism grounded in consumer “consideration set” theory 25 that operationalizes free-form language as a computable, economically consequential variable. Unlike subjective LLM-as-judge heuristics, PGA provides a strict mathematical gatekeeper: it dynamically computes semantic alignment between generated slogans and hidden buyer personas to determine market visibility, directly linking language generation to financial outcomes. Our contributions are as follows: • We define Market-Bench , a closed-loop economic environment that enforces hard scarcity (finite funds, bankruptcy) and competitive exclusion. • We propose Persona-Gated Attention , a mechanism that operationalizes the economic value of language, requiring agents to optimize semantic similarity to latent buyer personas to secure market access. • We provide a reproducible benchmark implementation and report results across 20 LLM agents using automatically computed economic, supply-chain, and semantic metrics. Figure 2: Overview of the Market-Bench environment. Agents operate in a competitive supply chain economy, making procurement, pricing, and marketing decisions under scarcity conditions. The Persona-Gated Attention mechanism requires agents to align their marketing slogans with latent buyer personas to convert demand into sales. Performance is evaluated across economic, supply chain, and cognitive metrics. 2 Related Work Work Competitive Interaction Hard Scarcity Closed-Loop Economy Free-Form Language Text \to Payoff Agent Societies 30 ✗ ✗ ✗ ✓ ✗ 50 ✗ ✗ ✗ ✓ ✗ 31 ✗ ✗ ✗ ✓ ✗ Strategic Games 51 ✓ ✗ ✗ ✓ ✗ 8 ✓ ✗ ✗ ✓ ✗ 28 ✓ ✓ ✗ ✓ ✗ 11 ✓ ✗ ✗ ✓ ✓ Economic Simulators 15 ✗ ✗ ✓ ✗ ✗ Market-Bench ✓ ✓ ✓ ✓ ✓ Table 1: Comparison of representative LLM-agent benchmarks and economic simulators. Market-Bench is the only benchmark providing both a closed-loop multi-agent market under hard scarcity and free-form language that directly affects payoff via Persona-Gated Attention. 2.1 LLM-Based Agent Societies Generative agents have shifted evaluation from static QA to dynamic sandboxes. 30 showed LLMs can simulate believable social behavior, and recent frameworks like AgentSociety 31 and MultiAgentBench 51 have scaled these simulations to urban environments and diverse tasks. However, these benchmarks primarily assess social believability or cooperative completion, rarely imposing the “scarcity conditions” (finite funds, bankruptcy) characteristic of real economies. Market-Bench bridges this gap by enforcing strict financial survival ( Funds_{i}<0\Rightarrow Bankruptcy ) within a competitive oligopoly. 2.2 Strategic Reasoning and Game Theory A distinct line of research evaluates LLMs as “Homo Economicus”, testing their rationality in classic game-theoretic setups. 8 and 28 subject agents to matrix games like Prisoner’s Dilemma 4 or Trust Game 6 to measure cooperation, deception, and Nash equilibrium convergence. Similarly, 11 explores bilateral bargaining, focusing on whether LLMs can reach Pareto-optimal deals. 49 further demonstrate the effectiveness of competitive game settings for revealing capability gaps among multimodal models. While these benchmarks test strategic logic, they often abstract away the linguistic complexity of commerce, reducing interaction to selecting a move (Cooper/Defect) or outputting a number. Market-Bench addresses this limitation by integrating free-form language as a first-class action, where agents must generate persuasive slogans to unlock market access through our novel Persona-Gated Attention mechanism. 2.3 Supply Chain and Business Optimization LLMs are increasingly deployed as decision-support tools for supply chain management 32 ; 15 . 15 introduce ABIDES-Economist for macroeconomic simulation, and Anthropic’s Project Vend 3 deployed Claude to autonomously run a retail shop in a single-agent setting. These systems generally position the LLM as a backend optimizer rather than an autonomous agent surviving competitive pressure. Market-Bench unifies procurement, pricing, and marketing into a single scalable loop under active multi-agent competition. 3

Read the complete text