Market-based Architectures in RL and Beyond Conference: Proc. of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025); May 19 – 23, 2025; Detroit, Michigan, USA bs034 CCS: Theory of computation Algorithmic game theory and mechanism design CCS: Computing methodol
Abhimanyu Pallavi Sudhir, Long Tran-Thanh · 2025
Read Market-based Architectures in RL and Beyond Conference: Proc. of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025); May 19 – 23, 2025; Detroit, Michigan, USA bs034 CCS: Theory of computation Algorithmic game theory and mechanism design CCS: Computing methodol by Abhimanyu Pallavi Sudhir, Long Tran-Thanh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05828.txt.
Introduction
Abhimanyu Pallavi Sudhir and Long Tran-Thanh proposed market-based learning agents in a 2025 paper. In such agents, a crowd of sub-agents bids in an inner market to pick the whole agent's next act. Markets can split a hard task into parts that run side by side, each with its own focus.
Their new form splits the state itself into axes, called goods, that sub-agents can trade over. That split buys more focus and more parallel work than past market designs. The authors argue the market frame could ease core AI pains such as search and scaling. They also sketch links to large language models for near-term use.
Catalog record
| CREATOR | Abhimanyu Pallavi Sudhir, Long Tran-Thanh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-05828
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Market-based Architectures in RL and Beyond Conference: Proc. of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025); May 19 – 23, 2025; Detroit, Michigan, USA bs034 CCS: Theory of computation Algorithmic game theory and mechanism design CCS: Computing methodologies Reinforcement learning CCS: Computing methodologies Multi-agent systems Abhimanyu Pallavi Sudhir, Long Tran-Thanh Abstract . Market-based agents refer to reinforcement learning agents which determine their actions based on an internal market of sub-agents. We introduce a new type of market-based algorithm where the state itself is factored into several axes called “goods”, which allows for greater specialization and parallelism than existing market-based RL algorithms. Furthermore, we argue that market-based algorithms have the potential to address many current challenges in AI, such as search , dynamic scaling and complete feedback , and demonstrate that they may be seen to generalize neural networks; finally, we list some novel ways that market algorithms may be applied in conjunction with Large Language Models for immediate practical applicability. Y. Vorobeychik, S. Das, A. Nowé (eds.) Blue Sky Ideas Track 1. Introduction Before neural networks won the mandate of heaven, an AI research paradigm that had shown considerable promise was that of market-based architectures , i.e. AI agents that determine their output based on some internal market-based mechanism ( Kwee et al. 2001 ; Chang et al. 2020 ) . There are general intuitive arguments that motivate such a line of research. Philosophers and psychologists have long pondered multi-agent models of the mind ( Minsky 1988 ) ; moreover, one may imagine that “any” machine learning task could in principle be solved by a market of agents solving sub-tasks with their individual reward set by the sale value of their output. There is work suggesting that markets can capture some notion of bounded rationality , e.g. the Boundedly Rational Inductive Agent (BRIA) ( Oesterheld et al. 2023 ) and Algorithmic Bayesian Epistemology ( Neyman 2024 ) . In some sense, markets ‘‘aggregate’’ the intelligence or capacities of their individual participants. 1 1 1 See e.g. Hayek on the role of markets in aggregating information ( von Hayek 1937 ) – or the famous parable “I, Pencil” ( Leonard E Read 1958 ) : “… no one person, no matter how smart, could create from scratch a small, everyday pencil [yet the market makes over a billion of them each year] …”. There is also some empirical work on emergent intelligent behaviour in markets comprised of zero-intelligence traders ( Gode and Sunder 1993 ; Schwartz 2008 ; Jamal et al. 2015 ) “Market-based architectures” can be made concrete in the case of reinforcement learning (RL), where the majority of work in this area lies (see 1.1 for a brief summary). For example in the Hayek machine ( Baum 1999 ) and its derivatives, the setting is a Markov Decision Problem (MDP) and there is a single resource, the “right to act and collect reward”, that is traded between sub-agents. At each time step, this resource is sold to the highest-bidding sub-agent, who performs some action that modifies the state and collects reward. In this paper, we argue that market-based agents represent an underexplored and promising niche, especially in context of recent advancements in language models (LLMs), and have potential to address a range of present challenges in contemporary AI research. Specifically, we make the following claims and contributions: Theoretical framework for market-based agents. We present two general frameworks for market-based RL agents: (1) the “deep market” (Def 1 ), where a single good, the state , is passed through a sequence of transacting agents, and (2) the “wide market” (Def 2 ), in which the state space itself is partitioned into factors called goods . The deep framework is not much of a departure from existing algorithms, and can be applied to any Partially Observed Markov Decision Process (POMDP); to our knowledge the wide framework is original to us, and is a generalization of the deep framework which better mirrors the success of real-world markets allowing for greater specialization and parallelism. A Python library for creating market-based algorithms will be released upon publication. Markets, neural networks and backpropagation. We demonstrate that these market-based agents can in principle be applied even to basic supervised learning tasks such as classification, and that neural networks (though not backpropagation or gradient descent) emerge as a special case of them. Furthermore we generalize the result in ( Wentworth 2018 ) to wide markets, demonstrating a suggestive relationship between backpropagation and markets at equilibrium. Search, complete feedback and alignment. We claim that markets can address several present problems in AI research, specifically: they are a natural framework for search , their scale or depth can be dynamic rather than fixed, and they allow complete feedback ( Demski 2024 ) , a property widely regarded as valuable in AI alignment. Markets and LLMs. We present novel ways in which market algorithms might be applied in conjunction with LLMs to address their limitations: they can be used for developing reasoning models like o1 , and LLMs can facilitate “information markets” that can in turn improve human feedback mechanisms in AI training. 1.1. Related Work Market-based RL. The majority of early work in this area has focused on market-based reinforcement learning (RL) algorithms. The pioneering work in this domain consists of Holland’s Learning Classifier Systems or “bucket brigade” ( Holland 1985 ) in rule-based systems, where condition-action agents (“classifiers”) bid to post messages (which can be actions described in some language) onto a global message board. Improvements to this paradigm were made by Schmidhuber ( Schmidhuber 1987 ; Schmidhuber 1989 ) who allowed agents to determine their own bids and imposed credit conservation, and by Baum ( Baum 1999 ) who further strictly enforced property rights, resulting in a much more familiar set-up called the “Hayek Machine”. The Hayek machine, which can be applied to any Markov Decision Process (MDP), was subsequently extended to POMDPs by ( Kwee et al. 2001 ) by adding external memory, and more recently ( Chang et al. 2020 ) modified the framework to use Vickrey auctions and prove that a Nash equilibrium of the market produces a globally optimal policy. Bounded rationality and markets. Other, more recent work in this area includes: Logical Induction ( Garrabrant et al. 2020 ) , an algorithm that assigns probabilities to mathematical sentences based on their prices in a prediction market that (roughly speaking) pays off when a sentence is proven; and the Boundedly Rational Inductive Agent (BRIA) ( Oesterheld et al. 2023 ) which solves finite decision problems by assigning it to the highest-bidding trader, similar to in market-based RL. The key insight of these works is that markets are useful for modeling boundedly rational agents . The main results of each work – the fact that the logical inductor cannot by dominated by any polynomial-time trader, and the “boundedly rational inductive agent criterion” in the latter paper – are specific and precise formulations of the Efficient Market Hypothesis ( Neyman 2024 ) . Markets and neural networks. A specific equivalence between classifier systems and neural networks has been studied in the Neural bucket brigade ( Davis 1988 ; Schmidhuber 1989