Ideological Bias in LLMs’ Economic Causal Reasoning

Donggyu Lee, Hyeok Yun, Jungwon Kim, Junsik Min, Sungwon Park, Sangyoon Park, Jihee Kim, [0.6em] Graduate School of Data Science, KAIST, Daejeon, South Korea, [email protected], [email protected], [email protected] · 2026

Read Ideological Bias in LLMs’ Economic Causal Reasoning by Donggyu Lee, Hyeok Yun, Jungwon Kim, Junsik Min, Sungwon Park, Sangyoon Park, Jihee Kim, [0.6em] Graduate School of Data Science, KAIST, Daejeon, South Korea, [email protected], [email protected], [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21334.txt.

Introduction

Donggyu Lee and six colleagues test whether language models show ideological bias when judging economic cause and effect. From 10,490 causal triplets drawn from top economics and finance journals, they isolate 1,056 ideology-contested cases where intervention-oriented and market-oriented views predict opposite signs. Contested items prove harder across the board, and in 17 of 20 models accuracy runs higher when the verified sign matches intervention-oriented expectations. Errors also lean intervention-oriented, and one-shot prompting does not erase the skew.

Catalog record

CREATORDonggyu Lee, Hyeok Yun, Jungwon Kim, Junsik Min, Sungwon Park, Sangyoon Park, Jihee Kim, [0.6em] Graduate School of Data Science, KAIST, Daejeon, South Korea, [email protected], [email protected], [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49885
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.21334

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Ideological Bias in LLMs’ Economic Causal Reasoning Donggyu Lee, Hyeok Yun, Jungwon Kim, Junsik Min, Sungwon Park, Sangyoon Park, Jihee Kim, [0.6em] Graduate School of Data Science, KAIST, Daejeon, South Korea, [email protected], [email protected], [email protected] Abstract Do large language models (LLMs) exhibit systematic ideological bias when reasoning about economic causal effects? As LLMs are increasingly used in policy analysis and economic reporting, where directionally correct causal judgments are essential, this question has direct practical stakes. We present a systematic evaluation by extending the EconCausal benchmark with ideology-contested cases—instances where intervention-oriented (pro-government) and market-oriented (pro-market) perspectives predict divergent causal signs. From 10,490 causal triplets (treatment–outcome pairs with empirically verified effect directions) derived from top-tier economics and finance journals, we identify 1,056 ideology-contested instances and evaluate 20 state-of-the-art LLMs on their ability to predict empirically supported causal directions. We find that ideology-contested items are consistently harder than non-contested ones, and that across 18 of 20 models, accuracy is systematically higher when the empirically verified causal sign aligns with intervention-oriented expectations than with market-oriented ones. Moreover, when models err, their incorrect predictions disproportionately lean intervention-oriented—and this directional skew is not eliminated by one-shot in-context prompting. These results highlight that LLMs are not only less accurate on ideologically contested economic questions, but systematically less reliable in one ideological direction than the other, underscoring the need for direction-aware evaluation in high-stakes economic and policy settings. 1 Introduction Do Large Language Models exhibit systematic ideological bias when reasoning about economic causal effects? This question is pressing because LLMs are increasingly deployed in high-stakes analytical workflows, such as economic reporting, policy evaluation, and corporate decision support ( 15 ; 18 ; 11 ) , where predicting causal directions correctly is essential. Yet in many such settings, that direction is genuinely contested: a single intervention can trigger competing mechanisms whose relative magnitudes are debated along ideological lines. A minimum wage increase, for instance, is expected to reduce hiring under a market-oriented framework that prioritizes labor cost effects, but to sustain employment under an intervention-oriented framework that emphasizes demand or job-matching gains. When LLMs predict causal directions in such settings, their outputs may implicitly privilege one set of priors over another, producing an invisible tilt in downstream policy advice. Existing approaches to detecting such effects fall short in complementary ways. Work on LLM bias typically relies on survey-style instruments that probe expressed opinions or preferences, often in social or political domains, and does not assess how models reason about causal relationships grounded in empirical evidence. In contrast, benchmarks for causal reasoning focus on formal logic or context-dependent inference, but assume a single correct expectation for each causal effect and do not account for settings where competing, internally consistent frameworks imply different expected directions. As a result, we lack a framework for asking whether LLMs reason about economic causation more reliably in one ideological direction than another, the setting where bias would have the most direct policy consequences. In this paper, we present the first systematic analysis of ideological bias in LLMs’ economic causal reasoning. Leveraging the EconCausal benchmark dataset—a large-scale corpus of 10,490 context-annotated causal triplets derived from top-tier economics and finance journals—we identify 1,056 ideologically contested triplets (10.1%) where market-oriented and intervention-oriented frameworks predict divergent causal signs. We evaluate LLM accuracy separately across contested and non-contested settings, and define a directional bias measure that captures systematic asymmetries in model predictions across competing frameworks. Through an extensive evaluation of 20 state-of-the-art LLMs, we document a striking asymmetry in reasoning ability. Models exhibit lower accuracy on ideologically contested triplets and perform significantly better when the empirical truth aligns with intervention-oriented expectations, showing an average accuracy gap of +9.7 to +15.1 percentage points relative to market-oriented cases. Errors are not random but systematically directional, and this pattern is robust across model families and difficulty levels. These findings highlight substantial reliability risks when LLMs serve as intermediaries in policy analysis, where implicit ideological anchoring can lead to systematically skewed conclusions. Our contributions are as follows. First, we introduce the notion of ideologically contested causal relationships as a new lens for studying LLM causal reasoning beyond context dependence. Second, we construct a labeled subset of EconCausal distinguishing contested from non-contested triplets, enabling controlled evaluation across these settings. Third, we provide a systematic empirical analysis showing lower accuracy and asymmetric performance in contested cases. Fourth, we define a directional bias measure and document consistent asymmetry patterns across a range of models. Together, our results highlight the importance of evaluating causal reasoning not only across contexts, but also across settings where disagreement over causal expectations is intrinsic. Figure 1: Overview of ideological divergence in LLM economic causal reasoning 2 Related Work 2.1 Political and Ideological Bias in LLMs Whether LLMs exhibit systematic political biases has attracted considerable attention. A common approach applies political orientation instruments to LLMs and positions their responses on ideological scales, and the emerging consensus is that instruction-tuned LLMs lean left on social and cultural dimensions ( 26 ; 25 ; 12 ; 19 ) . More recent efforts confirm a leftward tendency but show that its magnitude was overstated by earlier instruments, and that small changes in question format can substantially shift measured positions ( 24 ; 6 ) . These leanings have been traced to multiple sources. 8 show that political biases in pretraining corpora propagate into downstream task predictions. 9 find that optimizing for truthfulness amplifies left-leaning tendencies in larger models, and 5 document a monotonic relationship between parameter count and alignment with left-leaning parties. However, these studies focus on expressed preferences in response to opinion-like questions. They reveal what positions LLMs appear to endorse, but not whether ideological priors systematically distort reasoning about verifiable factual claims. This distinction may be especially important in economic settings, where users can easily interpret a model’s causal prediction as reflecting an empirical regularity rather than a prior absorbed during training. Whether ideological leanings also shape how LLMs reason about economic causal relationships—where ground-truth effect signs are available from peer-reviewed empirical research—remains unexplored. 2.2 LLMs in Economic and Causal Reasoning Work on causal reasoning suggests that LLMs often rely on priors rather than principled inference, which may help explain why directional bias can arise in economic prediction. Benchmarks such as CLadder and CORR2CAUSE consistently find that LLMs struggle with genuine causal reasoning across different levels of formalism ( 13 ; 14 ; 28 ; 3 ; 30 ) . A common explanation is that their causal judgments rely heavily on priors absorbed during pretraining rather than principled inference: LLMs have been described as “causal parrots,” shown to exploit textual metadata rather than observational evidence, to adopt rigid rule-like strategies, and to inadequately revise beliefs when confronted with contradictory evidence ( 31 ; 16 ; 29 ; 4 ) . Related work on LLMs in economic and policy settings points to similar limitations. LLMs have already been applied to policy-adjacent tasks such as classifying central bank communications ( 27 ) , while existing benchmarks show persistent weaknesses in economic and causal reasoning ( 23 ; 22 ; 21 ; 10 ) . Yet this literature evaluates performance mainly in terms of accuracy and robustness, not whether prediction errors are systematically directional along ideological lines. Our

Read the complete text