Out-of-Context Abduction: LLMs Make Inferences About Procedural Data Leveraging Declarative Facts in Earlier Training Data
Sohaib Imran [email protected], Rob Lamb, Peter M. Atkinson [email protected] · 2025
Read Out-of-Context Abduction: LLMs Make Inferences About Procedural Data Leveraging Declarative Facts in Earlier Training Data by Sohaib Imran [email protected], Rob Lamb, Peter M. Atkinson [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00741.txt.
Introduction
Sohaib Imran, Rob Lamb, and Peter Atkinson test whether language models can reason about facts buried in their training data. Their 2025 paper calls this skill out-of-context abduction: inferring the best explanation for an observation from facts learned earlier, without being shown those facts again.
The team trained models on names and behavior notes for made-up chatbots, but never on sample chats with them. After seeing example replies in a chatbot's style, one leading model could name at least one of the chatbots correctly. Extra training on the behavior notes also made the model act more like the chatbot described. The authors link the result to questions of situational awareness and AI safety.
Catalog record
| CREATOR | Sohaib Imran [email protected], Rob Lamb, Peter M. Atkinson [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-00741
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Out-of-Context Abduction: LLMs Make Inferences About Procedural Data Leveraging Declarative Facts in Earlier Training Data Sohaib Imran [email protected], Rob Lamb, Peter M. Atkinson [email protected] Abstract Large language models (LLMs) are trained on large corpora, yet it is unclear whether they can reason about the information present within their training data. We design experiments to study out-of-context abduction in LLMs, the ability to infer the most plausible explanations for observations using relevant facts present in training data. We train treatment LLMs on names and behavior descriptions of fictitious chatbots, but not on examples of dialogue with the chatbots. We find that OpenAI’s GPT 4o LLM can correctly infer at least one chatbot’s name after observing example responses characteristic of that chatbot. We also find that previously training GPT 4o on descriptions of a chatbot’s behavior allows it to display behaviors more characteristic of the chatbot when iteratively trained to display such behaviors. Our results have implications for situational awareness in LLMs and, therefore, for AI safety. 1 1 footnotetext: Lancaster Environment Centre, Lancaster University, Lancaster LA1 4YQ, UK 2 2 footnotetext: School of Computing and Communications, Lancaster University, Lancaster LA1 4WA, UK 3 3 footnotetext: JBA Trust, 1 Broughton Park, Skipton BD23 3FD, UK 4 4 footnotetext: Geography and Environmental Science, University of Southampton, Highfield, Southampton SO17 1BJ, UK 5 5 footnotetext: College of Surveying and Geo-Informatics, Tongji University, No.1239, Siping Road, Shanghai, PR China, 200092 1 Introduction With the recent popularity of large language models (LLMs), much research has been dedicated to evaluate their reasoning capabilities ( Huang & Chang 2023 ; Webb et al. 2023 ; Lee et al. 2025 ) . However, research on the subject has yet to has yet to show conclusively that LLMs can reason, partly due to inconsistent findings and varied interpretations of similar results ( Mirzadeh et al. 2024 ; Wu et al. 2024 ) . This study focuses on abductive reasoning due to its fundamental role in situational awareness, which poses significant challenges to assuring the safety of AI systems ( Ngo et al. 2023 ) . Abductive reasoning, often referred to as "inference to the best explanation" is the process of inferring the most likely hypothesis that explains some observations. The functional form of abduction is \displaystyle A \displaystyle\rightarrow B \displaystyle B \displaystyle A where A is one of the hypotheses that explain the observation B . Note that observing B does not necessarily entail A , but merely increases the likelihood of A . For example, if a lawn is wet one could infer rain, but if it is a dry day and a sprinkler is present then a more likely hypothesis is that the sprinkler was used. Abduction has two interpretations. The weak interpretation views it only as a mechanism for generating plausible hypotheses for observations without assessing the likelihood of the hypotheses. In contrast, the strong interpretation posits abduction’s role as justificatory, that is, inferring or selecting the best hypothesis that explain the observations ( Niiniluoto 1999 ; Calzavarini & Cevolani 2022 ) . LLMs are capable of weak abductive inference, i.e., generating plausible hypotheses to explain observations present in their context window ( Balepur et al. 2024 ; Shi et al. 2024 ; Zhao et al. 2024 ) . Furthermore, LLMs are capable of strong in-context abductive inference i.e. selecting the most plausible hypothesis where both the observations and candidate hypotheses are presented in their context window ( Bhagavatula et al. 2019 ; Bang et al. 2023 ) . However, it is difficult to delineate whether these capabilities result from reasoning or simply pattern matching to imitate human reasoning ( Shanahan et al. 2023 ; McCoy et al. 2024 ; Wu et al. 2024 ) . We investigate the ability of LLMs to perform strong out-of-context abduction—inferring the most plausible explanations for observations by leveraging relevant facts learned during training. Our setup prevents simple imitation of human response patterns to masquerade as reasoning by: 1. Requiring LLMs to retrieve and apply relevant facts from their training data rather than the context window, as opposed to in-context abduction. 2. Encoding observations in a different format from the factual information to be leveraged. The factual information is declarative (abstract descriptions) and the observations are procedural (example instances or demonstrations). 2 Problem Definition Let \mathcal{C} denote a set of classes. For each class c\in\mathcal{C} , let \mathcal{D}_{c} be the set of all abstract descriptions characterizing c . Each description d_{c}\in\mathcal{D}_{c} defines semantic, structural, or behavioral properties of outputs that belong to c . \tilde{\mathcal{X}}_{c} is the fuzzy set of all realizations under c . \tilde{\mathcal{X}}_{c} is defined by a membership function \mu_{\tilde{\mathcal{X}}_{c}}:\mathcal{X}\to[0,1] (1) where \mathcal{X} is the set of all natural language sequences. The value \mu_{\tilde{\mathcal{X}}_{c}}(x) represents the degree of membership of x\in\mathcal{X} in \tilde{\mathcal{X}}_{c} . Therefore, a natural language sequence x_{1}\in\mathcal{X} is more characteristic of c than x_{2}\in\mathcal{X} if \mu_{\tilde{\mathcal{X}}_{c}}(x_{1})>\mu_{\tilde{\mathcal{X}}_{c}}(x_{2}) . We implement the membership function with a scoring function for each class c : s_{c}(x)=\mu_{\tilde{\mathcal{X}}_{c}}(x) (2) Crisp sets with varying degrees of membership in \tilde{\mathcal{X}}_{c} can be generated by partitioning the space \tilde{\mathcal{X}}_{c} into subsets of elements based on thresholds: \mathcal{X}_{c}^{\tau_{1},\tau_{2}}=\{x\in\mathcal{X}\mid\tau_{1}\leq s_{c}(x)<\tau_{2}\} (3) where \tau\in[0,1] is a threshold parameter. The shorthand notation \mathcal{X}_{c}^{<\tau} and \mathcal{X}_{c}^{\geq\tau} can be used to denote sets whose membership scores are, respectively, below or at least as large as a given threshold \tau . Let f represent an LLM which maps from an input space \mathcal{Q}\subseteq\mathcal{X} to an output space \mathcal{A}\subseteq\mathcal{X} , and can also accept an optional class description d_{c}\in\mathcal{D}_{c}\subseteq\mathcal{X} . We write: f:\mathcal{Q}\times(\mathcal{D}_{c}\cup\{\bot\})\to\mathcal{A} (4) where \{\bot\} represents the null set. d_{c} biases f toward outputs with higher membership in \tilde{\mathcal{X}}_{c} : \mathbb{E}\bigl[s_{c}(f(q,d_{c}))\bigr]>\mathbb{E}\bigl[s_{c}(f(q))\bigr]. (5) 2.1 Abductive Inference We are interested in a function g that infers the underlying class c from any set of realizations \hat{\mathcal{X}}_{c}\subset\mathcal{X} where \mathbb{E}\bigl[s_{c}(\hat{\mathcal{X}}_{c})\bigr]\gg\mathbb{E}\bigl[s_{c}(\mathcal{X})\bigr] : g(\hat{\mathcal{X}}_{c})=c (6) In practice, a subset of the LLM responses \mathcal{A}_{c}^{\tau_{1},\tau_{2}}\subseteq\mathcal{X}_{c}^{\tau_{1},\tau_{2}} is used to derive \hat{\mathcal{X}}_{c} with \tau_{1},\tau_{2} such that \mathbb{E}\bigl[s_{c}(\mathcal{A}_{c}^{\tau_{1},\tau_{2}})\bigr]\gg\mathbb{E}\bigl[s_{c}(\mathcal{A})\bigr] under the assumption \mathbb{E}\bigl[s_{c}(\mathcal{A})\bigr]\approx\mathbb{E}\bigl[s_{c}(\mathcal{X})\bigr] . Since \mathcal{A}_{c}^{\tau_{1},\tau_{2}} is a filtered subset of \mathcal{A} , this only requires \tau_{1} to lie above a point x in the support of s_{c}(\mathcal{A}) below which a significant probability mass of s_{c}(\mathcal{A}) lies. In practice, we use thresholds larger than the average score of LLM responses for the class: \tau_{1}>\mathbb{E}\bigl[s_{c}(\mathcal{A})\bigr] (7) 2.2 Out-of-Context Abduction To implement g , an LLM f is trained on declarative factual information of the form c\leftrightarrow\mathcal{D}_{c}^{train} i.e. statements associating the class c with its abstract descriptions \mathcal{D}_{c}^{train}\subseteq\mathcal{D}_{c} for multiple classes c_{1},c_{2},... . No descriptions d_{c}\in\mathcal{D}_{c} appear in the same context window in which the class c is to be inferred from realizations (equation 6 ), making this an out-of-context abduction problem. This is our treatment LLM f_{treat}=f_{c_{1}\leftrightarrow\mathcal{D}_{c_{1}}^{train},c_{2}\leftrightarrow\mathcal{D}_{c_{2}}^{train},...}=g . Therefore equation 6 becomes: f_{treat}(\hat{\mathcal{X}}_{c},\hat{q})=c (8) where \hat{q}\in\hat{\mathcal{Q}}\subset\mathcal{Q} is a question about which class generated \hat{\mathcal{X}}_{c} . Experiment 1: To test for out-of-context abduction in LLMs, experiment 1 (section 4.1 ) measures if observing realizations more characteristic of a class c than a different class c^{\prime} leads the treatment LLM trained on statements associating multiple classes and their behavior descriptions f_{treat} to conclude that the observations were generated by c rather than c^{\prime} : P(f_{treat}(\hat{\mathcal{X}}_{c},\hat{q})=c)>P(f_{treat}(\hat{\mathcal{X}}_{c},\hat{q})=c^{\prime}) (9) where \mathbb{E}\bigl[s_{c}(\hat{\mathcal{X}}_{c})\bigr]\gg\mathbb{E}\bigl[s_{c^{\prime}}(\hat{\mathcal{X}}_{c})\bigr] . Furthermore, this conclusion must be informed