Convergence to collusion in algorithmic pricing
Kevin Michael Frick · 2026
Read Convergence to collusion in algorithmic pricing by Kevin Michael Frick (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15825.txt.
Introduction
Kevin Michael Frick asks how fast pricing algorithms can learn to collude, testing average-reward soft actor-critic algorithms in a repeated Bertrand duopoly with two symmetric firms, logit demand, and fixed marginal costs. The algorithms reach collusive prices within 50,000 periods, and in 36 percent of runs, learned punishments make one-period cheating unprofitable, so high prices become mutual best responses. Where punishments fail, profitable deviations are typically only hundredths of a percent.
Catalog record
| CREATOR | Kevin Michael Frick |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71837 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.15825
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Convergence to collusion in algorithmic pricing Thanks: I sincerely wish to thank Emilio Calvano, Giacomo Calzolari, Vincenzo Denicolò, Massimiliano Furlan, Santiago Ortiz, Xavier Lambin, Sergio Pastorello, Violetta van Veen, two anonymous referees, and Editor Özlem Bedre Defolie for precious comments and suggestions concerning all drafts of this article. Any remaining errors are the author’s sole responsibility. Replication code, based on a framework by 22 , is available under the GNU Affero General Public License v3.0 at https://github.com/kmfrick/price_simulator . This paper supersedes previous drafts titled “Algorithmic pricing using policy gradient reinforcement learning” and “Convergence rates and collusive outcomes of pricing algorithms”. The author declares no conflicts of interest. Kevin Michael Frick Abstract Artificial intelligence algorithms are increasingly used by firms to set prices. Previous research shows that they can exhibit collusive behaviour, but how quickly they can do so has so far remained an open question. I show that a modern deep reinforcement learning model deployed to price goods in a repeated oligopolistic competition game with continuous prices converges to a collusive outcome in an amount of time that matches empirical observations, under reasonable assumptions on the length of a time step. This model shows cooperative behaviour supported by reward-punishment schemes that discourage deviations. Keywords : algorithmic pricing, optimal control, artificial intelligence, collusion, reinforcement learning JEL : D21, D43, D83, L12, L13 1 Introduction In recent years, legislative attention highlights a serious threat to the antitrust policy framework: the potential for artificial intelligence to undermine market competition ( 40 ; 10 ; 50 ) . A prominent body of literature in economics and computer science has thus emerged to investigate whether pricing algorithms can indeed autonomously learn collusive behaviour. The possibility of algorithmic collusion was demonstrated by 11 and 29 using pricing algorithms based on Q-learning. Their findings have been replicated and expanded ( 4 ; 6 ; 25 ; 12 ; 1 ) . Moreover, empirical evidence of algorithmic collusion has emerged. An important contribution by 5 analysed the impact of widespread adoption of algorithmic pricing in the German retail gasoline market. The study found that nearby competitors’ adoption of pricing algorithms increased their mean margins by almost a third compared to pre-adoption levels, whereas sole adoption did not lead to significant margin changes. The empirical analysis in 5 aligns with 11 by demonstrating a gradual increase in margins over time, suggesting that algorithms have a learning curve. However, the time scales differ noticeably. 5 revealed a collusion timescale of a couple of years, consistent with the findings of 9 in the case of human collusion. In contrast, 11 and subsequent experimental literature suggest that convergence to collusive outcomes takes a large number of periods, in the order of hundreds of thousands. The actual length of a period may vary across industries, but it must be long enough for new prices’ demand effects to materialise and observable profits to emerge. In 5 , prices change on average every hour, making the one million time steps required for convergence in 11 equivalent to dozens of years of real-life time. Such discrepancies may challenge the applicability of experimental and theoretical results and the solidity of the empirical evidence ( 46 ; 17 ) . Acknowledging the problem of the slow learning time scale, 11 attribute it to the nature of Q-learning algorithms, which learn slowly by design. They speculate that more sophisticated algorithms might learn to collude more quickly. The primary contribution of this article is to verify this conjecture. I analyse a class of algorithms more advanced than Q-learning and demonstrate that these algorithms can autonomously learn to collude much faster than 11 , achieving a reduction of two orders of magnitude. The slow learning rate of Q-learning algorithms is due to their estimation of a Q-matrix, containing the value of each possible action (typically, a firm’s price) in each possible state of the dynamical system (typically, a vector of prices in past periods). This estimation starts from arbitrary initial values and updates the matrix based on observations of the firm’s profits for each possible action-state combination. This approach has several implications. First, the action space, which would normally be continuous, must be discretised. Second, learning requires visiting all cells of the matrix, necessitating random exploration instead of always choosing the action deemed optimal based on acquired knowledge. Third, the updating must be gradual to prevent rapid forgetting of past learning, requiring each cell to be visited multiple times. For instance, in the baseline analysis of 11 , the Q-matrix comprises over 3,000 cells 1 1 1 There are 225 spaces, given by 15 actions per agent, and thus 225\times 15=3375 state-action pairs. , requiring more than 60,000 periods per cell to visit each cell 20 times, even under permanent exploration. However, collusion cannot be learned during exploration mode because the best response to uniform random pricing by a rival cannot be a collusive strategy, as there is no scope for punishment for deviation from collusion. Consequently, the algorithms must be in exploitation mode frequently enough, leading to the need for approximately one million periods for convergence. To overcome this issue, this article employs average-reward soft actor-critic algorithms ( 20 ; 3 ) , the current state of the art in the reinforcement learning literature. An actor-critic algorithm learns to estimate a continuous action-value function rather than a discrete matrix, and at the same time learns to estimate a continuous strategy profile that takes as arguments the estimated action values. These algorithms exploit the topological structure of the action and strategy spaces, which Q-learning disregards, such that after observing the profit for a particular combination of current and past prices, the algorithms update the value not only for that specific point but also for nearby prices. Another innovation concerns the direction of exploration. Soft actor-critic algorithms optimise exploration by imposing a constraint on the Kullback-Leibler divergence 2 2 2 Kullback-Leibler divergence measures non-commutatively how different two probability distributions are from each other, and is defined for two continuous densities f(\cdot),g(\cdot) over support \Omega as \int_{\Omega}f(x)\log[f(x)/g(x)]dx . between the policy and a uniform distribution, that is, they learn the policy that maximises long-run profit while staying as random as possible ( 38 ) . Exploration is thus optimised instead of uniform, so that the algorithms more intensively explore the vicinity of what they believe to be the optimal policy, that is, the optimal reaction function to past prices. I show that the combined effect of these natural and sensible changes to the learning strategy of Q-learning algorithms is two-fold. On the one hand, compared to Q-learning, convergence to a Nash equilibrium is not guaranteed. This is to be expected: deep reinforcement learning algorithms are notoriously unstable, with around two-thirds of trials routinely failing to learn any sensible control rule in continuous control tasks ( 21 ) . On the other hand, two or more soft actor-critic algorithms competing à la Bertrand can maintain an equilibrium supra-competitive price that is robust to deviations in around 50,000 periods under logit, non-stochastic demand and constant marginal costs, no capacity constraints, no entry or exit, and no informational asymmetry. If a period is one hour, this would be about five years, a timescale comparable to empirical