LLMs Struggle to Perform Counterfactual Reasoning with Parametric Knowledge
Khurram Yamin, Gaurav Ghosal, Bryan Wilder · 2025
Read LLMs Struggle to Perform Counterfactual Reasoning with Parametric Knowledge by Khurram Yamin, Gaurav Ghosal, Bryan Wilder (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15732.txt.
Introduction
Khurram Yamin, Gaurav Ghosal, and Bryan Wilder ask in 2025 whether large language models can reason counterfactually by blending given facts with stored knowledge. Note that the head title differs from the catalog title, and the head governs here. Models carry vast built-in knowledge, but new settings often supply facts that clash with it, and sound reasoning must respect the stated premises.
Across synthetic and real multi-step problems, models mostly ignored the given facts and answered from memory. Simple fine-tuning afterward also struggled to install the skill. Worse, it often degraded the stored knowledge itself. The results expose a real limit in how models repurpose what they know.
Catalog record
| CREATOR | Khurram Yamin, Gaurav Ghosal, Bryan Wilder |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-15732
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLMs Struggle to Perform Counterfactual Reasoning with Parametric Knowledge Khurram Yamin, Gaurav Ghosal, Bryan Wilder Abstract Large Language Models have been shown to contain extensive world knowledge in their parameters, enabling impressive performance on many knowledge intensive tasks. However, when deployed in novel settings, LLMs often encounter situations where they must integrate parametric knowledge with new or unfamiliar information. In this work, we explore whether LLMs can combine knowledge in-context with their parametric knowledge through the lens of counterfactual reasoning . Through synthetic and real experiments in multi-hop reasoning problems, we show that LLMs generally struggle with counterfactual reasoning, often resorting to exclusively using their parametric knowledge. Moreover, we show that simple post-hoc finetuning can struggle to instill counterfactual reasoning ability – often leading to degradation in stored parametric knowledge. Ultimately, our work reveals important limitations of current LLM’s abilities to re-purpose parametric knowledge in novel settings. † † affiliationnotice: Equal contribution 1 Introduction Large Language Models (LLMs) internalize extensive world knowledge during pretraining, powering tasks like open-domain QA, fact retrieval, and knowledge-base completion ( Petroni et al. 2019 ; Liu et al. 2019 ; Roberts et al. 2020 ) . Benchmarks such as NaturalQuestions and HotpotQA focus on recall and multi-hop composition but do not require integrating novel premises at inference time ( Yang et al. 2018 ; Kwiatkowski et al. 2019 ) . For example: “If Paris were located in Italy, in which country would the Eiffel Tower stand?” Answering this demands two capabilities: Contextual Override —suppressing “Paris is in France”—and Selective Retrieval —retaining the Paris–Eiffel Tower link. Standard benchmarks do not probe this dual requirement. Prior work on retrieval-augmented generation illustrates how external documents can both aid and confuse LLMs when facts disagree ( Lewis et al. 2021 ) , and studies of grokking transformers show multi-hop chaining without dynamic premise integration ( Wang et al. 2024a ) . However, none systematically evaluate counterfactual multi-hop reasoning against a pretrained knowledge graph. We ask: Can modern LLMs combine stored knowledge with in-context counterfactual premises to answer multi-hop queries correctly? To address this, we introduce synthetic graph-based and real-world causal benchmarks that isolate reinforcing, additive, contradictory, and irrelevant contexts; empirically analyze GPT-4o to identify context-ignoring and context-overfitting failure modes under standard, chain-of-thought, and fine-tuned prompts; demonstrate that simple post-hoc fine-tuning yields only marginal gains on counterfactual tasks and can degrade performance on factual benchmarks; and discuss implications for interactive systems, retrieval-augmented pipelines, and safety-critical applications requiring accurate conditional reasoning. Our results reveal a fundamental limitation: despite powerful fact memorization and retrieval, current LLMs lack mechanisms to dynamically modify their internal knowledge graph in response to new or conflicting information. Addressing this gap will require novel modeling and training paradigms. 2 Related Works Multi-Hop QA Multi-hop QA benchmarks measure a model’s ability to chain stored facts but do not test integration of new premises. Yang et al. 2018 introduce a dataset for contextual multi-hop QA, and Wang et al. 2024a show that transformers can chain relations in a synthetic “grokking” regime. Real-world analyses compare reasoning over different relation types ( Yang et al. 2024 ) and investigate how intermediate entities emerge in transformer layers ( Biran et al. 2024 ) . In contrast, we examine how LLMs combine parametric memory with in-context information. Knowledge Conflicts LLMs face knowledge conflicts when external context contradicts internal facts. Techniques like contextual decoding constraints ( Yuan et al. 2024 ) and attention pruning ( Li et al. 2025 ) enforce premises, while closed-book QA relies solely on retrieved parameters ( Petroni et al. 2019 ; Roberts et al. 2020 ) . Retrieval-augmented methods (REALM ( Guu et al. 2020 ) , RAG ( Lewis et al. 2021 ) , DPR ( Karpukhin et al. 2020 ) , FiD ( Izacard & Grave 2020 ) ) merge external evidence, and finer-grained approaches like AdaCAD ( Wang et al. 2024b ) and CD2 ( Jin et al. 2024 ) balance parametric and contextual sources. However, none target selective retention of stored knowledge under counterfactual premises. Causal Reasoning and Counterfactuals in NLP Counterfactual reasoning with LLMs remains challenging. Benchmarks such as CLadder ( Jin et al. 2023 ) , QRData ( Liu et al. 2024 ) , and CounterBench ( Chen et al. 2025 ) reveal poor performance on formal causal and counterfactual tasks. Critics argue LLMs often “parrot” causal patterns from data rather than infer causally ( Zečević et al. 2023 ) , and Yamin et al. 2024 detail failures like over-reliance on parametric knowledge. While methods exist for generating faithful explanations ( Gat et al. 2024 ) or counterfactual tokens ( Chatzi et al. 2024 ) , a systematic study of multi-hop counterfactual integration is still lacking. 3 Experiments on Real-World LLMs (a) (b) (c) (d) Figure 1 : Causal Counterfactual Plots comparing standard GPT-4o, GPT-4o CoT and GPT-4o Fine tuned Results. (a) Counterfactual Reinforces Prior, (b) Counterfactual Adds new Information, (c) Counterfactual Conflicts with Prior, (d) Counterfactual is Irrelevant to Prior and Query. 95 \% CI is shown. 3.1 Problem Formulation for a Causal Case We inject contextual information at test time. For example, if the model’s parametric knowledge includes Y_{1}\!\rightarrow\!Y_{2} and prompt asserts Y_{2}\!\rightarrow\!X_{0} , can it deduce Y_{1}\!\rightarrow\!X_{0} ? Results shown in main paper are for GPT 4o ( OpenAI et al. 2024 ) , and LLama 3.1 8B ( Grattafiori et al. 2024 ) results can be found in the Appendix. We categorize contextual edits into four partitions (all prompts and code are in the appendix/supplement): 1. Scenario 1 (Reinforcing Prior Knowledge): Prompting the LLM with a relationship already present in its prior knowledge graph, thereby reinforcing an existing edge. Example: Given excessive rain causes flooding, query whether excessive rain causes infrastructure damage. 2. Scenario 2 (Adding New Information): Prompting the LLM with scenario-specific information necessary to answer the query, but absent from its parametric knowledge graph, akin to adding an edge. Example: Informing the LLM that excessive rain causes Timmy to eat vegetables, and querying whether excessive rain improves Timmy’s health. 3. Scenario 3 (Contradicting Prior Knowledge): Prompting the LLM with information that strongly contradicts its existing parametric knowledge, equivalent to replacing an edge in its prior knowledge graph. Example: Informing the LLM that excessive rain causes desert expansion and querying whether excessive rain promotes cactus growth. 4. Scenario 4 (Irrelevant Information): Prompting the LLM with unrelated information, akin to providing an edge from a disconnected knowledge graph. Example: Informing the LLM that fatty food consumption causes heart attacks, and querying whether excessive rain leads to flooding. 3.2 Prompting Methods We compare three strategies: Standard —direct causal query; CoT —chain-of-thought prompting; FT —fine-tuning on counterfactual examples with CoT explanations (160 examples, hyperparameters in Appendix). Models are queried over either 1 or two counterfactual hops for simplicity. 3.3 Results We ask a binary cause/non-cause question (random baseline 50 \% ). Figure 1 reports GPT-4o accuracy across scenarios. 3.3.1 Success when Context Does Not Oppose Prior We see that in Scenario 1 where we reinforce prior knowledge, Figure 1(a) shows that standard prompting achieves around 85 \% accuracy, with CoT accuracy rises above 95 \% , and FT further closes errors to near 100 \% with low variance across trials. These results