LLM Agents Display Human Biases but Exhibit Distinct Learning Patterns
Idan Horowitz ([email protected]), Ori Plonsky ([email protected]) · 2025
Read LLM Agents Display Human Biases but Exhibit Distinct Learning Patterns by Idan Horowitz ([email protected]), Ori Plonsky ([email protected]) (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10248.txt.
Introduction
Idan Horowitz and Ori Plonsky compare how large language models and humans learn from repeated choices with feedback. In such tasks, people show known quirks, such as giving rare events too little weight. The authors ran models and humans through the same choice tasks to see whether the same surface patterns hide the same inner process.
On average, the models did look human-like, showing the same broad biases. A closer look broke that picture. The models leaned hard on the most recent outcomes, a strong recency bias that humans did not show. Human tricks, like switching after a surprise, were absent in the models. Similar averages, the authors warn, can hide very different minds at work.
Catalog record
| CREATOR | Idan Horowitz ([email protected]), Ori Plonsky ([email protected]) |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-10248
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLM Agents Display Human Biases but Exhibit Distinct Learning Patterns Idan Horowitz ([email protected]), Ori Plonsky ([email protected]) Abstract We investigate the choice patterns of Large Language Models (LLMs) in the context of Decisions from Experience tasks that involve repeated choice and learning from feedback, and compare their behavior to human participants. We find that on the aggregate, LLMs appear to display behavioral biases similar to humans: both exhibit underweighting rare events and correlation effects. However, more nuanced analyses of the choice patterns reveal that this happens for very different reasons. LLMs exhibit strong recency biases, unlike humans, who appear to respond in more sophisticated ways. While these different processes may lead to similar behavior on average, choice patterns contingent on recent events differ vastly between the two groups. Specifically, phenomena such as “surprise triggers change” and the “wavy recency effect of rare events” are robustly observed in humans, but entirely absent in LLMs. Our findings provide insights into the limitations of using LLMs to simulate and predict humans in learning environments and highlight the need for refined analyses of their behavior when investigating whether they replicate human decision making tendencies. Keywords: language models, LLMs, decisions from experience, recency bias, decision making, surprise, learning Introduction Large Language Models (LLMs), particularly in the form of AI chatbots, assistants, or agents, have become a dominant focus over the last few years, with enormous potential to aid and improve global efficiency, while perhaps also presenting a threat in their incredibly human-seeming conversational abilities. As such, the volume of research around them and their possible applications has grown enormously. As probabilistic models, LLMs may not be susceptible to the same behavioral tendencies that humans commonly exhibit, making them difficult to understand, study and use in a predictable manner. Additionally, the rapid advancements in LLMs call for continuous re-evaluation of these findings. In this study, we evaluate LLMs in a domain previously unexplored in that respect: Decisions From Experience (DFE) tasks. In DFE tasks, participants face a repeated choice between options that are each associated with some payoff distribution and get feedback on their choices. Usually, participants do not have a-priori knowledge of the the payoff distribution, and should thus learn it from the feedback. These studies allow exploration of the basic learning processes that emerge when people repeatedly engage with an environment and learn it from experience. Studies of DFE reveal several highly robust decision-making phenomena Erev et al. (2023) , and in this work, we investigate whether LLMs also exhibit similar behaviors. Decisions from Experience Phenomena Our focus is on four distinct behavioral phenomena that emerge in DFE studies. One of the most robust of these phenomena is underweighting of rare events Barron & Erev (2003) ; Hertwig et al. (2004) . When making decisions based on experience with the choice task, people tend to behave as if they give rare events lower weight than they objectively deserve. Underweighting of rare events sharply contrasts the pattern predicted by Prospect Theory Kahneman & Tversky (1979) (that pertains to one-shot decisions based on a description of the task). The difference between the way people treat rare events when basing their decisions on experience vs. a description of the choice task is known as the “description-experience gap” Hertwig & Erev (2009) . A second behavioral phenomenon that emerges in DFE studies is surprise triggers change Nevo & Erev (2012) , according to which a surprising event in one trial increases the chances that a decision maker will switch their selection in the next trial. This occurs even when the payoffs in that previous trial were reinforcing of their most recent choice. A third observation from DFE studies is the wavy recency effect of rare events Plonsky et al. (2015) , which refers to the sequential dependencies that people exhibit in consecutive trials. Individuals appear to respond to rare events in a wavy pattern: in the trials that immediately follow a rare event, they choose as if their sensitivity to the rare event (a) decreases for a few trials, usually reaching a trough 3 trials after the rare event, and then, somewhat oddly, (b) increases to a peak approximately 12-15 trials after the rare event (i.e., they behave as if the rare event becomes gradually more likely to re-occur), and then diminishes back. Together, these changing sensitivities make a wave-like pattern of choice. Finally, when facing decision tasks in which the realized outcomes of the choice options are correlated, subjects exhibit a “correlation effect” Diederich & Busemeyer (1999) suggesting that they quickly learn to choose the best option when the (usually positive) correlation implies that option consistently outperforms the alternatives, but learn slowly when the (usually negative) correlation implies that the best option (which is better on average) is not consistently better than the alternatives. Our Contribution A thorough investigation of Decisions from Experience (DFE) studies, and particularly the phenomena mentioned above, has yet to be conducted on LLM agents. The goal of this research is to explore the behavior of LLM agents in common DFE tasks, and compare their decisions in these tasks to those of humans. We also explore how the manner by which the history of decisions is provided to the agent, and how the LLM’s temperature, the model’s degree of variability, influence the results, if at all. Finally, we try to infer the manner by which the LLM agents learn by comparing their behavior to several basic choice strategies. A better understanding of these issues has the potential to aid in the use of LLMs to generate human-like data and to improve on the current methods used in modeling and predicting human behavior. Existing Work Numerous studies have attempted to assess the behavior of LLMs through the lens of cognitive psychology (e.g. Binz & Schulz (2023) ; Coda-Forno et al. (2024) ; Jones & Steinhardt (2022) ), with results inconclusive. While language models display many human-like intuitions Hagendorff et al. (2023) and behavioral similarities, including abstract reasoning Yehudai et al. (2024) , communication Liu, Sumers et al. (2024) ; Liu et al. (2023) and decision making patterns (including in the errors and biases that arise from such tasks) Mei et al. (2024) ; Rathje et al. (2024) ; Lampinen et al. (2024) , there are also areas in which the irrationality of LLMs does not align with that of humans Macmillan-Scott & Musolesi (2024) . They also tend to exhibit certain distortions Aher et al. (2023) ; Liu, Geng et al. (2024) , have low social intelligence Wang et al. (2024) and perform poorly in social scenarios with information asymmetry Zhou et al. (2024) . For example, Nobandegani et al. (2024) Nobandegani et al. (2024) found that while LLMs behave rationally in response to the Allais Paradox, they fail to do so for the Ellsberg Paradox. Nonetheless, the similarities to humans in the way LLMs behave have led to successes in using LLMs to recover findings in human experiments and therefore simulate human behavior Aher et al. (2023) ; Argyle et al. (2023) ; Coletta et al. (2024) ; Horton (2023) ; Shapira et al. (2024) . These works lay the foundation to integrate LLMs in the prediction of human decision-making Aher et al. (2023)