Counterfactual-Consistency Prompting for Relative Temporal Understanding in Large Language Models
Jongho Kim, Seung-won Hwang · 2025
Read Counterfactual-Consistency Prompting for Relative Temporal Understanding in Large Language Models by Jongho Kim, Seung-won Hwang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11425.txt.
Introduction
Jongho Kim and Seung-won Hwang present a 2025 paper on time reasoning in large language models. Such models often mix up simple links, such as which event came before or after another. They can also give replies that clash with each other. The pair aim to fix that clash, not just single answers.
Their method poses extra what-if questions based on the first one. It then forces the set of answers to stay in line with each other. Tests on several data sets showed clear gains. The method helped with event order for both plain and hidden time links. It also helped with common-sense time tasks. The work targets a known weak spot in model reasoning.
Catalog record
| CREATOR | Jongho Kim, Seung-won Hwang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-11425
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Counterfactual-Consistency Prompting for Relative Temporal Understanding in Large Language Models Jongho Kim, Seung-won Hwang Abstract Despite the advanced capabilities of large language models (LLMs), their temporal reasoning ability remains underdeveloped. Prior works have highlighted this limitation, particularly in maintaining temporal consistency when understanding events. For example, models often confuse mutually exclusive temporal relations like “before” and “after” between events and make inconsistent predictions. In this work, we tackle the issue of temporal inconsistency in LLMs by proposing a novel counterfactual prompting approach. Our method generates counterfactual questions and enforces collective constraints, enhancing the model’s consistency. We evaluate our method on multiple datasets, demonstrating significant improvements in event ordering for explicit and implicit events and temporal commonsense understanding by effectively addressing temporal inconsistencies. 1 Introduction Despite the impressive capabilities of LLMs, a line of research Jain et al. 2023 ; Chu et al. 2023 has highlighted that these models often lack temporal reasoning abilities. This is especially true for relative event understanding, where the goal is to infer temporal relationships between events or properties within an event in the passage, without depending on absolute time indicators (e.g., specific dates). Figure 1: Example of leveraging counterfactual questions to resolve temporal inconsistency in LLMs. The primary challenge is that LLMs lack temporal consistency in their responses Qiu et al. 2024 ; Chen et al. 2024 . Temporal consistency is defined as the model’s ability to ensure that conflicting timelines do not co-exist. For instance in Figure 1 -(a), if the model is temporally inconsistent, mutually exclusive temporal relations like “before” and “after” are sometimes confused when ordering events, leading to contradictory predictions—such as stating that Event A happens both before and after Event B in the same context. While events with time indicators are often addressed with mathematical reasoning Zhu et al. 2023 ; Su et al. 2024 , no existing work has successfully tackled the challenge of temporal inconsistency in the events’ relative understanding without requiring explicit time markers. Chain-of-thought (CoT) reasoning Wei et al. 2022 , which primarily aids mathematical and symbolic reasoning Sprague et al. 2024 , is also reported to fail to solve such inconsistency Qiu et al. 2024 . Considering temporal consistency is fundamental in temporal reasoning, its absence in LLM can undermine key tasks like planning Sakaguchi et al. 2021 ; Zhang et al. 2024 . These observations highlight the need for alternative reasoning skills to achieve temporal consistency. This study answers the following research question: Can we prompt LLMs to elicit the ability to mitigate temporal inconsistency? Inspired by counterfactual augmentation, where models are exposed with lexically similar, but typically label-flipping pairs in training Kaushik et al. 2020 , we extend it to LLMs to generate temporally counterfactual questions : We introduce lexically small interventions to the original input (e.g. before to after, years to centuries) that drastically affect its temporal semantics. By providing these questions and self-generated answers alongside the original input, the model would rely less on lexical similarities and better understand the semantics. To this end, we propose a novel counterfactual-consistency prompting (CCP), designed to enhance the temporal consistency of LLMs, as described in Figure 1 -(b). CCP first generates temporal counterfactual exemplars and then applies the insights gained to address the original temporal question. This method is particularly effective in relative event understanding because the counterfactual exemplars not only encourage the model to understand different temporal semantics but also directly impose temporal constraints. For instance, if the model states that “Event A happens after Event B” and also recognizes that “Event A happens before Event B”, the conflict forces the model to collectively re-weight the validity of these two statements. We show performance gain of CCP across multiple relative event understanding tasks. Our effectiveness in mitigating temporal inconsistencies is further demonstrated by our inconsistency metric. Dataset Relation Revision Type Example TempEvalBi-QA & TRACIE r_{1}(e_{1},e_{2}) Original They got married after they moved to Maine. r_{2}(e_{1},e_{2}) r_{1}\rightarrow r_{2} They got married before they moved to Maine. MCTACO: Ordering r_{1}(e_{1},e_{2}) Original They went to the store after they were put in jail. r_{1}(e_{1},e_{3}) e_{2}\rightarrow e_{3} They repented after they were put in jail. MCTACO: Duration, r_{1}(e_{1}) Original It has existed for 1 year . Frequency, Typical Time r_{2}(e_{1}) r_{1}\rightarrow r_{2} It has existed for centuries . MCTACO: Stationarity r_{1}(e_{1}) Original She is still in Ranchipur. \neg r_{1}(e_{1}) r_{1}\rightarrow\neg r_{2} She is not in Ranchipur. Table 1: Types of counterfactuals targeted for generation. The examples illustrate how counterfactual questions modify the semantics regarding temporal aspects ( r_{1} , r_{2} ) including relations and properties, for events ( e_{1} , e_{2} , e_{3} ). 2 Method Our idea is to make the model approximate the temporal constraints using counterfactuals. Because temporal reasoning imposes unique interdependencies, where one temporal aspect affects another Han et al. 2019 ; Kim et al. 2024 , counterfactuals enable us to capture these constraints. For example, if the model establishes from a counterfactual exemplar that “Event e_{1} happens [ r_{2} : before] Event e_{2} ”, it is constrained to predict the original question that “Event e_{1} cannot happen [ r_{1} : after] Event e_{2} ”: r_{2}(e_{1},e_{2})\in\mathcal{V}\implies r_{1}(e_{1},e_{2})\not\in\mathcal{V} (1) where r(e_{a},e_{b}) represents the temporal relation r between events e_{a} and e_{b} , and \mathcal{V} represents the set of coherent temporal relations with the context. 2.1 Generating Temporally Counterfactual Questions Given a context C , our task is to provide an accurate answer to question Q while maintaining temporal consistency. We start by creating temporally counterfactual questions, Q^{c_{1}}...Q^{c_{i}} . Counterfactual augmentation conventionally aims to generate instances with lexically minimal edits while keeping others unchanged Huang et al. 2019 ; Kaushik et al. 2020 ; Wang and Culotta 2020 . This strategy discourages models from relying too much on superficial similarity. However, previous works Kaushik et al. 2020 require arbitrary label-flipping edits, which are unsuitable for LLM inference with unknown test labels. In contrast, we focus on temporally counterfactual questions that specifically edit the temporal semantics of the original sentence. Our approach ensures that the model adheres to the “temporal constraints”, yet retains the effect of label-flipping that emphasizes temporal cues over superficial similarity. Specifically, we set the types of counterfactual questions based on the temporal semantics each dataset aims to capture. The types of temporal counterfactuals are listed in Table 1 . For sentences representing temporal relation between two events ( r_{1}(e_{1},e_{2}) ), we substitute the relation r_{1} to r_{2} , or event e_{2} to e_{3} . For those representing an event’s temporal properties ( r_{1}(e_{1}) ) such as duration or stationarity, we substitute the property to r_{2} or negate it to \neg r_{1} . To generate counterfactual questions, we design our model to dynamically create them rather than rely on a predefined rule-based template. While rule-based approaches like Chen et al. 2024 in logical reasoning constrain answers using a predefined question set, they limit flexibility to cover the broad range of temporal expressions. By comparison, our dynamic generation of constraints provides a more adaptable solution. We specify various counterfactual types through in-context learning (ICL) to control the relevance of these generated questions. The full prompts are in Appendix G.1 . 2.2 Counterfactual-Consistency Prompting After generating the counterfactual questions, we prompt the model again to