LLM Biases
Jinhui Han, Ming Hu, Xilin Zhang · 2026
Read LLM Biases by Jinhui Han, Ming Hu, Xilin Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26960.txt.
Introduction
Han, Hu and Zhang ask whether transformer agents' workings can also bias them in patterned ways. Studying recommenders that generate the next user action from history, they identify four bias channels. Positional bias pushes influence toward recent history, aiding speed but hurting stability and diversity. Popularity amplification magnifies frequency gaps into lopsided exposure, feeding echo chambers. Latent driver bias overweights a few past events when choice drivers go unobserved. Synthetic data bias arises as platforms retrain on model-shaped logs, concentrating outputs and erasing long-tail options.
Catalog record
| CREATOR | Jinhui Han, Ming Hu, Xilin Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 133209 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.26960
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLM Biases Jinhui Han, Ming Hu, Xilin Zhang Abstract Transformer-based agentic AI is rapidly being deployed on major platforms to help users shop, watch, and navigate content with less effort. While these systems can deliver impressive performance, a key concern is whether they may be less reliable than they appear. We ask a simple but fundamental question: whether the mechanisms that make transformer-based agents effective can also induce systematic biases or distortions? We study this question through a theoretical analysis of transformer-based generative recommenders, in which the next user interaction is generated sequentially from a user’s history. Focusing on how the model allocates attention across historical evidence, we identify four bias channels: (i) Positional bias: stronger positional encoding shifts influence toward recent history, improving responsiveness but potentially reducing stability and long-term diversity; (ii) Popularity amplification: small frequency differences in data can be magnified into disproportionate exposure, contributing to Matthew effects and echo chambers; (iii) Latent driver bias: when important drivers of user choices are not directly observed, the model can place overly concentrated weight on a small subset of past events, creating overconfident attributions. (iv) Synthetic data bias: when users increasingly follow AI suggestions and platforms retrain on model-shaped synthetic logs, outputs can concentrate over time, and long-tail alternatives can disappear first. Our analysis highlights mechanism-level reliability risks that may not be visible in offline performance metrics. The four bias channels indicate that large-scale deployment may systematically distort exposure and choice. For managers, the immediate implication is to treat these as operational risk factors and to monitor concentration and drift over time, rather than assuming that performance gains alone guarantee reliability. † † runningtitle: LLM Biases † † authors: Guanghua School of Management, Peking University, Beijing 100871, China, [email protected] Rotman School of Management, University of Toronto, Toronto, Ontario, Canada M5S 3E6, [email protected] Rotman School of Management, University of Toronto, Toronto, Ontario, Canada M5S 3E6, [email protected] † † affiliation: † † affiliation: † † affiliation: † † affiliation: † † affiliation: † † affiliation: 1 Introduction With the rapid advances and broad deployment of large language models (LLMs), firms are increasingly adopting transformer-based models and LLM-style agents to tackle large-scale, complex decision problems. In Operations Management (OM) and Operations Research (OR), this progress has accelerated a shift from using AI primarily for prediction (e.g., demand forecasting or scoring) to using AI as a decision policy that maps rich histories into actions. A growing body of OM research and emerging business practice now explores transformer and agentic architectures for core decisions in assortment planning ( 50 , 39 ) , inventory management ( 17 , 64 , 6 ) , and supply chain planning and management ( 45 , 61 ) . This development is particularly timely because customers and markets are increasingly delegating decisions to AI in real-world commerce journeys, reshaping how demand is formed and observed ( 3 , 5 ) . Against this broader backdrop, LLM-style and transformer-based AI agents are reshaping consumer platforms as shopping assistants, streaming copilots, and content navigators. Each time a platform decides what to show next or where to route a user, through a ranked list, a feed slot, an item page, or a bundle, it is effectively allocating a scarce exposure budget across items, sellers, and categories. In this sense, agentic recommendation is no longer just a back-end ranking tweak, but it has become a primary mechanism through which platforms convert attention into demand. Reflecting this shift, major platforms are investing heavily in transformer-based recommendation capabilities. YouTube, for example, states that recommendations operate primarily on the homepage and the “Up Next” panel ( 19 ) and develops industrial-scale generative recommendation methods ( 21 ) . In music streaming, Spotify positions its AI DJ as a personalized guide that selects what to play next ( 47 ) , and Spotify Research discusses semantic identifiers to support generative search and recommendation ( 40 ) . Beyond streaming, Pinterest reports a transformer-based real-time user action model deployed on its largest engagement surface, Homefeed ( 59 ) . Amazon describes Rufus as a shopping assistant that helps customers discover products and supports intent-based shopping flows ( 4 ) . OpenAI has rolled out shopping assistants in ChatGPT, including Pulse and “Buy it in ChatGPT”, which are explicitly designed to support product research and recommend relevant options ( 38 ) . Moreover, Meta describes GEM as an LLM-inspired “foundation model” designed for ads recommendation ( 28 ) . What makes LLM-style and transformer-based AI agents powerful, relative to many traditional generative models, is the attention mechanism , which enables the model to sift through a long history (past interactions, context, and instructions) and focus on the most relevant elements for the next prediction. Despite their impressive performance in reducing search and decision frictions, a central question for large-scale adoption remains: do transformer-based/LLM-style agents introduce systematic biases in recommendations? We take a step toward answering this question by focusing on transformer-based generative recommenders (rather than fully dialogue-driven assistants), which preserves analytical tractability while maintaining practical relevance. In this framework, given a user’s interaction history, the model generates a short token code that maps to the next recommended item. The code can be learned from rich item information, thereby reflecting text, images, and structured attributes rather than a simple item ID. Importantly, generative recommenders can be deployed as the decision engine behind shopping assistants, streaming copilots, and content agents, which continuously produce personalized actions rather than static rankings. Building on this setup, we establish four practically meaningful biases endogenously induced by the transformer mechanism: (i) Positional bias . Transformers almost always include information about where each past interaction appears in the sequence, either its absolute position or, more often, relative position, indicating how far it is from the current step. We show that relative positional encodings can systematically put more weight on certain parts of a user’s history, so that recommendations tilt toward older versus newer behavior depending on the effective strength of the positional signal. (ii) Popularity bias . Transformers can yield super-linear exposure amplification, with two distinct manifestations: (a) a platform-level Matthew effect in which already popular items receive disproportionate exposure, increasing sales/play/traffic concentration; and (b) a user-level echo-chamber effect that reinforces within-user frequency patterns. (iii) Latent driver bias . Generative recommenders learn from clicks, views, and purchases. Yet many choices are decided by latent drivers not recorded in the data (e.g., situational context, user mood). We capture this mismatch as stochastic perturbations in representations and show that, as uncertainty increases, attention can become brittle and unreliable. (iv) Synthetic data bias . As users increasingly delegate decisions to AI, interaction logs become endogenously shaped by the agent rather than generated by independent users. When platforms repeatedly retrain on these synthetic logs, recommendations can drift toward a narrower portion of the product catalog. We emphasize that the four biases we have identified are not arbitrarily chosen. Each aligns with recognized concerns in recommender systems and with emerging issues in rapidly expanding agentic AI decision environments. Positional bias and popularity bias, for example, are widely discussed as model-intrinsic sources of bias in recommendation ( 11 , 15 ) . On the other hand, misspecification of latent uncertainty and synthetic-data bias