Are LLMs Biased Like Humans? Causal Reasoning as a Function of Prior Knowledge, Irrelevant Information, and Reasoning Budget.

Hanna M. Dettki, Charley M. Wu, Bob Rehder · 2026

Read Are LLMs Biased Like Humans? Causal Reasoning as a Function of Prior Knowledge, Irrelevant Information, and Reasoning Budget. by Hanna M. Dettki, Charley M. Wu, Bob Rehder (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02983.txt.

Introduction

Do language models reason about causes the way humans do? Hanna M. Dettki and 2 co-authors benchmark that question in a 2026 paper. They compare more than twenty models against matched human judgments. Eleven tasks share a collider structure, where two separate causes produce one shared effect, a classic test of causal reasoning.

A small interpretable model summarized the models' judgments well. Most models reasoned in a more rule-like way than humans. Humans seemed to account for unmentioned hidden factors, while models largely did not mirror human collider biases. Extra tests added abstract wording or irrelevant text. Chain-of-thought prompting improved robustness for many models. The authors suggest model reasoning can complement human judgment, but may falter when uncertainty is intrinsic.

Catalog record

CREATORHanna M. Dettki, Charley M. Wu, Bob Rehder
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-02983

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Are LLMs Biased Like Humans? Causal Reasoning as a Function of Prior Knowledge, Irrelevant Information, and Reasoning Budget. Hanna M. Dettki, Charley M. Wu, Bob Rehder Abstract Large language models (LLMs) are increasingly used in domains where causal reasoning matters, yet it remains unclear whether their judgments reflect normative causal computation, human-like shortcuts, or brittle pattern matching. We benchmark 20+ LLMs against a matched human baseline on 11 causal judgment tasks formalized by a collider structure ( C 1 → E ← C 2 C_{1}\!\rightarrow\!E\!\leftarrow\!C_{2} ). We find that a small interpretable model compresses LLMs’ causal judgments well and that most LLMs exhibit more rule-like reasoning strategies than humans who seem to account for unmentioned latent factors in their probability judgments. Furthermore, most LLMs do not mirror the characteristic human collider biases of weak explaining away and Markov violations. We probe LLMs’ causal judgment robustness under (i) semantic abstraction and (ii) prompt overloading (injecting irrelevant text), and find that chain-of-thought (CoT) increases robustness for many LLMs. Together, this divergence suggests LLMs can complement humans when known biases are undesirable, but their rule-like reasoning may break down when uncertainty is intrinsic—highlighting the need to characterize LLM reasoning strategies for safe, effective deployment. Keywords: Large Language Models; Causal Inference; Human and Machine Reasoning; Bayesian Modeling Introduction Human causal judgment is powerful but brittle: people routinely deviate from normative inference under limited attention, fatigue, and contextual framing 6 ; 23 ; 2 ; 27 . This motivates AI-assisted decision-making in high-stakes settings where consistent causal analysis could complement human limitations. This is especially important since large language models (LLMs) are increasingly deployed in domains where causal understanding matters (e.g., legal and medical decision support) 5 ; 20 . Successful deployment therefore depends not only upon whether LLMs can generate correct output in known contexts via pattern matching / associative reasoning , but crucially on whether they can apply causal reasoning strategies allowing them to generalize to new contexts. A core challenge is that in the real world, many causal judgment settings are ill-defined (e.g., no exact base rates of events are known) and hence humans rely on heuristics and prior assumptions, often introducing biases. Before deploying LLMs in high-stakes settings, it is hence paramount to not only compare LLM judgments to humans and investigate whether they reproduce certain human biases, provided that LLMs have been trained on largely human generated data, but also whether LLMs provide sensible causal judgments (e.g., that the probability of a variable increases as more evidence arises). Related Work. Recent work evaluates LLMs on causal benchmarks with fully specified generative models, such as CLADDER 11 , where queries have a single numeric ground truth given explicit base rates and link strengths. Such settings primarily test whether models can apply statistical rules, but they are less informative about an agent’s implicit world model (e.g., assumed priors or causal strengths) when these quantities are underspecified, as is often the case in real-world settings. We therefore adopt a classic paradigm from human causal cognition 18 ; 19 ; 3 ; 24 that deliberately leaves causal tasks underspecified, enabling a direct probe of the assumptions agents bring to the task; it also avoids requiring formal training in causal statistics, making human baselines more straightforward to collect. Our study connects to the small body of work directly comparing humans and LLMs on the same reasoning tasks 9 ; 13 ; 12 . It also relates to evidence that LLM judgments can shift under irrelevant context or superficial content changes 22 ; 15 , and that Chain-of-Thought (CoT) can sometimes mitigate such fragility—addressing concerns that apparent causal reasoning may reflect learned correlations rather than causal understanding 26 . Similarly, 28 study probabilistic coherence in LLM judgments via repeated elicitation and violations of probability identities, using Bayesian cognitive models to interpret systematic deviations; our work instead examines the coherence and robustness of qualitative reasoning signatures across different causal queries and content manipulations. Within collider structures, humans reliably exhibit systematic departures from normative qualitative patterns—including too little explaining away 8 ; 21 and Markov violations in which beliefs about one cause depend on information about an alternative cause 1 ; 14 ; 16 ; 17 . Whether and when LLMs reproduce these human-like biases, particularly given training on human-generated text, remains unclear. Finally, while 7 tested against the same human baseline but evaluated only four LLMs, substantially broader evaluations and systematic robustness analyses remain limited, motivating the present work. Contributions. We pursue the question of how LLMs’ causal reasoning compares to that of humans by evaluating 20\!+\! LLMs on a causal inference benchmark derived from classic collider-graph judgments with a human baseline 17 . We are interested in three core questions: (Human-comparison) do models reproduce human-like response patterns and biases? (Mechanism) can their responses be compactly explained by a normative causal Bayesian network (CBN) model? (Robustness) do these behaviors persist under content manipulations that reduce reliance on memorized world knowledge as well as in the face of inserted irrelevant information? We test the above both with direct prompting where we ask for a single probability judgment as a response and with chain-of-thought (CoT) 25 where we ask LLMs to first reason step-by-step. Together, this yields a cognitively-grounded account of how different LLMs reason causally—often more rule-based than humans and mostly not replicating human biases, with systematic shifts under CoT prompting. We additionally release an LLM-friendly version of the causal inference benchmark (including the human baseline) and causAIign , software package that supports structure-matched custom prompts and content manipulations. Methods Benchmark tasks and human baseline. We build on causal inference tasks from 17 (RW17; Experiment 1, Model-Only condition; N\!{=}\!48 NYU undergraduates), which elicit probability judgments in a collider graph C_{1}\!\rightarrow\!E\!\leftarrow\!C_{2} . Variables are binary, causes are asserted as independent, and participants report probabilities (scale 0–100) for a target event (“off”/0 or “on”/1) given observed values of the other variables. 17 test 11 inference tasks (I–XI) corresponding to distinct conditional-probability queries on this collider—e.g., predictive queries such as p(E\!{=}\!1\!\mid\!C_{1}\!{=}\!0,\!C_{2}\!{=}\!1) in Figures 1(a) and 3(a) —embedded in three different cover stories (sociology, weather, economy). Critically, because priors over variables and probabilities such as p(E\!\mid\!C_{1},\!C_{2}) are not specified, there is no ground-truth “correct” probability; the benchmark is therefore designed to assess reasoning strategies of causal judgment (e.g., how consistent agents reason across tasks, or how rule-following they are) and qualitative patterns , e.g., assessing the level of explaining away and Markov-(non) compliance and compare them across agents and to the human baseline. LLM prompting and experimental conditions. For LLMs, we convert the RW17 training and test-phase information given to humans into a single textual prompt that includes (i) a domain introduction, (ii) causal mechanism statements for both causes, (iii) the observed evidence, and (iv) an inference question requesting a single numeric probability judgment \!\in\![0,100] . We evaluate two prompting strategies: direct prompting (single numeric answer) and chain-of-thought (CoT) prompting (“think step by step” before the numeric answer). We set temperature to 0.0 to minimize sampling variance; for models that expose an explicit reasoning-budget control, we use it as provided by the API. We test robustness via two content manipulations crossed with prompting, yielding 2\!\times\!2\!\times\!2\!=\!8 experimental conditions: (i) Prior knowledge : original

Read the complete text