Your AI, Not Your View: The Bias of LLMs in Investment Analysis CCS: Computing methodologies Natural language processing
Hoyoung Lee, Junhyuk Seo, Suhwan Park, Junhyeong Lee, Wonbin Ahn, Chanyeol Choi, Alejandro Lopez-Lira, Yongjae Lee · 2025
Read Your AI, Not Your View: The Bias of LLMs in Investment Analysis CCS: Computing methodologies Natural language processing by Hoyoung Lee, Junhyuk Seo, Suhwan Park, Junhyeong Lee, Wonbin Ahn, Chanyeol Choi, Alejandro Lopez-Lira, Yongjae Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-20957.txt.
Introduction
Hoyoung Lee and co-authors expose built-in biases in how language models judge investments in a 2025 paper. In finance, a model's stored knowledge can clash with live market data. If the model's own leanings differ from its client's goals, advice can go wrong, yet these leanings were little measured.
Using made-up scenarios with balanced and one-sided arguments, the team extracted each model's latent preferences. Across most models, technology stocks, large companies, and contrarian bets were favored. Worse, first impressions hardened into confirmation bias as counter-evidence piled up. The authors publish a public leaderboard scoring bias across many models, arguing bias should be a standard check for financial AI.
Catalog record
| CREATOR | Hoyoung Lee, Junhyuk Seo, Suhwan Park, Junhyeong Lee, Wonbin Ahn, Chanyeol Choi, Alejandro Lopez-Lira, Yongjae Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-20957
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Your AI, Not Your View: The Bias of LLMs in Investment Analysis CCS: Computing methodologies Natural language processing Hoyoung Lee, Junhyuk Seo, Suhwan Park, Junhyeong Lee, Wonbin Ahn, Chanyeol Choi, Alejandro Lopez-Lira, Yongjae Lee Abstract . In finance, Large Language Models (LLMs) face frequent knowledge conflicts due to discrepancies between pre-trained parametric knowledge and real-time market data. These conflicts become particularly problematic when LLMs are deployed in real-world investment services, where misalignment between a model’s embedded preferences and those of the financial institution can lead to unreliable recommendations. Yet little research has examined what investment views LLMs actually hold. We propose an experimental framework to investigate such conflicts, offering the first quantitative analysis of confirmation bias in LLM-based investment analysis. Using hypothetical scenarios with balanced and imbalanced arguments, we extract the latent preferences of models and measure their persistence. Focusing on sector, size, and momentum, our analysis reveals distinct, model-specific tendencies. In particular, we observe a consistent preference for large-cap stocks and contrarian strategies across most models. These preferences often harden into confirmation bias, with models clinging to initial judgments despite counter-evidence. 1. Introduction The rapid advancement of LLMs has spurred a surge of innovation within the financial sector, where they are particularly adept at processing qualitative and unstructured information. Research is now actively exploring their use across a range of applications, including forecasting stock price movements from news sentiment ( Lo and Ross 2024 ) , extracting nuanced insights from complex analyst reports ( Kim et al. 2023 ) , and aiding in the construction and optimization of portfolios ( Hwang et al. 2025 ; Lee et al. 2025 ; Ko and Lee 2024 ) . This trend is now evolving towards even greater autonomy through the development of sophisticated LLM-based agents. These systems, which may function as a single powerful agent or as collaborative multi-agent teams, are designed to execute complex, dynamic tasks like active trading and automated portfolio management ( Zhang et al. 2024 ; Yu et al. 2024 ; Luo et al. 2025 ; Xiao et al. 2024 ) . Figure 1. A conceptual illustration of knowledge conflict in LLM-based financial services. Even when a firm targets a specific investment theme (e.g., Energy), the LLM’s inherent preferences (e.g., Technology) may override user intent, producing biased and inconsistent recommendations. A critical but underexplored issue in financial applications is knowledge conflict . In a domain as fluid and time-sensitive as finance, conflicts between an LLM’s parametric knowledge and real-time market data are frequent. This conflict becomes particularly revealing when the model is presented with a mix of information. Crucially, studies show that when an LLM encounters both supporting evidence (aligning with its ingrained beliefs) and counter evidence simultaneously, it exhibits a strong confirmation bias ( Xie et al. 2023 ) . Instead of weighing the arguments objectively, the model stubbornly adheres to the evidence that confirms its internal knowledge while disregarding the counter evidence. This tendency to reinforce internal biases over objective reasoning poses a major risk to LLM-based financial services. For instance, as illustrated in Figure 1 , even if a financial institution wants to target a specific sector (e.g., Energy), the LLM may override this with its own preference (e.g., Technology). Consequently, this creates a dilemma: the service reflects the model’s bias, not the user’s intent, leading to distorted, unpredictable decisions that ultimately erode client trust. To address this core problem, we must first systematically uncover these hidden biases. We therefore seek to answer the following research questions: RQ 1: What intrinsic preferences do LLMs exhibit towards key financial factors like sector, size, and momentum? RQ 2: How do these intrinsic preferences lead to biases when LLMs are forced to make decisions under contradictory evidence? To answer these questions, this study introduces a three-stage experimental framework designed to systematically elicit and verify LLM biases in investment analysis. In the first stage, we construct targeted arguments for each stock, such as positive vs. negative sentiment or momentum vs. contrarian perspectives, to represent competing investment views. In the second stage, we present these arguments in a balanced manner to induce a knowledge conflict and reveal the model’s latent preferences. In the third stage, we introduce progressively stronger counter evidence to examine the resilience of these preferences, observing how they evolve into rigid, confirmation-biased judgments. The main contributions of this paper are twofold. First, we propose a systematic methodology to identify and quantify latent biases in LLMs for financial applications. Second, we provide the first quantitative analysis of confirmation bias exhibited by LLMs in investment analysis, demonstrating a clear link between a model’s inherent preferences and its stubbornness against contradictory facts. By systematically uncovering these hidden risks, our work lays a critical foundation for developing more transparent and trustworthy financial AI. Our code, experimental results, and model details are available online. 1 1 1 https://github.com/ihobbang250/bias 2. Background This section provides background on two distinct research areas: knowledge conflict and financial biases in LLMs. We first review the challenges of knowledge conflict, then the issue of financial biases, providing the necessary context for our investigation into their intersection. 2.1. Knowledge Conflict A critical vulnerability in LLMs is knowledge conflict, which arises when external, contextual information clashes with the model’s internal, parametric knowledge ( Sun et al. 2025 ; Jin et al. 2024 ) . A significant body of research demonstrates that when faced with such conflicts, LLMs exhibit a strong confirmation bias. Foundational work by ( Xie et al. 2023 ) revealed that LLMs behave as "stubborn sloths," clinging to any piece of evidence that supports their internal knowledge, even against a majority of contradictory facts. This over-reliance on internal memory is further evidenced by findings that LLMs struggle to suppress their parametric knowledge even when instructed to ( Sun et al. 2025 ) and can exhibit a Dunning-Kruger-like effect, confidently trusting their own faulty beliefs over correct external information ( Jin et al. 2024 ) . This tendency toward knowledge-based stubbornness is part of a broader pattern. Given their training on vast amounts of human data, LLMs have been shown to inherit and functionally replicate human cognitive biases ( Echterhoff et al. 2024 ) . A key example is the choice-supportive bias, where the mere act of making an initial choice significantly boosts the model’s confidence in that choice, making it highly resistant to change ( Kumaran et al. 2025 ; Zhuang et al. 2025 ) . This phenomenon is part of a wider landscape of biases identified in LLMs when they act as evaluators. For instance, models exhibit familiarity bias (preferring text they find easier to process), are susceptible to anchoring effects ( Stureborg et al. 2024 ) , and can be biased towards their own generated contexts over externally retrieved information, even when their own generated text is incorrect ( Tan et al. 2024 ) . 2.2. Financial Biases in LLMs The presence of these cognitive biases is particularly concerning in the economic and financial domains. Initial research has begun to map their characteristics, with frameworks applying utility theory demonstrating that LLMs are neither perfectly rational nor consistently human-like ( Ross et al. 2024 ) . Other studies note that even specialized financial LLMs can exhibit strong