Counterfactual Samples Constructing and Training for Commonsense Statements Estimation

Chong Liu, Zaiwen Feng, Lin Liu, Zhenyun Deng, Jiuyong Li, Ruifang Zhai, Debo Cheng, Li Qin · 2024

Read Counterfactual Samples Constructing and Training for Commonsense Statements Estimation by Chong Liu, Zaiwen Feng, Lin Liu, Zhenyun Deng, Jiuyong Li, Ruifang Zhai, Debo Cheng, Li Qin (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-20563.txt.

Description

Abstract Plausibility Estimation (PE) plays a crucial role for enabling language models to objectively comprehend the real world. While large language models (LLMs) demonstrate remarkable capabilities in PE tasks but sometimes produce trivial commonsense errors due to the complexity of commonsense knowledge.

Catalog record

CREATORChong Liu, Zaiwen Feng, Lin Liu, Zhenyun Deng, Jiuyong Li, Ruifang Zhai, Debo Cheng, Li Qin
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-20563

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Counterfactual Samples Constructing and Training for Commonsense Statements Estimation Chong Liu, Zaiwen Feng, Lin Liu, Zhenyun Deng, Jiuyong Li, Ruifang Zhai, Debo Cheng, Li Qin Abstract Plausibility Estimation (PE) plays a crucial role for enabling language models to objectively comprehend the real world. While large language models (LLMs) demonstrate remarkable capabilities in PE tasks but sometimes produce trivial commonsense errors due to the complexity of commonsense knowledge. They lack two key traits of an ideal PE model: a) Language-explainable : relying on critical word segments for decisions, and b) Commonsense-sensitive : detecting subtle linguistic variations in commonsense. To address these issues, we propose a novel model-agnostic method, referred to as C ommonsense C ounterfactual S amples G enerating ( CCSG ). By training PE models with CCSG, we encourage them to focus on critical words, thereby enhancing both their language-explainable and commonsense-sensitive capabilities. Specifically, CCSG generates counterfactual samples by strategically replacing key words and introducing low-level dropout within sentences. These counterfactual samples are then incorporated into a sentence-level contrastive training framework to further enhance the model’s learning process. Experimental results across nine diverse datasets demonstrate the effectiveness of CCSG in addressing commonsense reasoning challenges, with our CCSG method showing 3.07 % \% improvement against the SOTA methods. [style=chinese] [style=chinese] [style=chinese] [style=chinese] [style=chinese] [style=chinese] [style=chinese] [style=chinese] keywords Plausibility Estimation ,Counterfactual Reasoning ,Large Language Models ,Commonsense Biases ,Contrastive Learning † † credit: Writing – original draft, Conceptualization, Methodology, Investigation, Software † † credit: Writing – review & editing, Conceptualization, Methodology † † credit: Writing – review & editing, Conceptualization † † credit: Writing – review & editing, Methodology † † credit: Writing – review & editing, Methodology † † credit: Writing – review & editing, Supervision, Validation † † credit: Writing – review & editing, methodology, Project administration, Conceptualization † † credit: Writing – review & editing, Conceptualization, Supervision, Funding acquisition † † Contributed equally to this work. † † corresponding: Corresponding author 1 Introduction Plausibility Estimation (PE), which assesses the plausibility of natural language sentences according to general knowledge that people commonly possess, is one of the fundamental capabilities of advanced Artificial intelligence (AI) agents. With the advancement of large language models (LLMs), PE has made considerable progress in distinguishing between commonsense and non-commonsense statements. However, due to inevitable manual annotation artifacts in real world datasets, existing PE models often fall into over-reliance on superficial linguistic correlations between sentences and their associated golden labels (also known as language biases) ( Bender and Koller, 2020 ; Marcus and Davis, 2023 ; Talmor et al., 2018 ) . For example, a model might childishly answer “ false ” for “ Mr. July ordered wires for dinner at a Chinese restaurant ” and still receive a satisfactory estimation. Figure 1: An ideal PE model should exhibit two indispensable characteristics: (a) Language-Explainable ability : The PE model should not only make a correct prediction but also base its prediction on the appropriate linguistic reference regions. (b) Commonsense-Sensitive ability : The PE model must be enough sensitive to commonsense variations in texts. For example, replacing the critical word “swim” with “run”, in a statement should result in opposite predicted labels, reflecting the underlying change in plausibility. Currently, the most prevalent solutions for PE fall into two categories: a) Relying on external knowledge ( Thorne et al., 2018 ; Ling et al., 2023 ) : These approaches leverage neural language models to extract reasoning chains from external knowledge bases, pinpointing solutions to commonsense questions while providing supporting knowledge assertions. However, they are heavily dependent on the availability and quality of external knowledge bases, which are costly and time-intensive to construct. b) Construction-based methods ( Zeng et al., 2020 ; Wang and Culotta, 2021 ; Udomcharoenchaikit et al., 2022 ; Tokpo and Calders, 2024 ) : These methods automatically generate parallel counterfactual data by intervening in existing observational examples to enhance the original dataset. The primary aim is to focus the model’s attention on critical samples, thereby strengthening the dataset’s quality and informativeness. Although construction-based methods show remarkable performance, they neglect to endow models with the two crucial characteristics necessary for an ideal PE mode: a) Language-Explainable ability : An ideal PE model should depend on the correct linguistic reference segments, adhering to the principle of being right for the right reasons ( Zeng et al., 2020 ; Ross et al., 2017 ) . As shown in Figure 1 (a), while two models might both predict “ It is wrong ” correctly, they may rely on different linguistic regions, leading to differing levels of explainability. b) Commonsense-Sensitive ability : An ideal PE model must be enough sensitive to commonsense linguistic variations in statements. For instance, as shown in Figure 1 (b), similar statements with identical structures (where the word “ swim ” is replaced by “ run ”), should yield different predictions, reflecting the change in background commonsense. The model must recognize this discrepancy and adjust its predictions accordingly. In this research, we propose a novel method, C ommonsense C ounterfactual S amples G enerating ( CCSG ), which could be used for a plug-and-play component to enhance the language-explainable and commonsense-sensitive abilities of PE models. Specifically, we develop a counterfactual samples constructor using low-level dropout and word-piece replacement, guided by word contributions. Our CCSG method is independent of pre-trained knowledge, making it a versatile mediator to guide language models (LMs) toward focusing on relevant linguistic regions while improving commonsense sensitivity. Additionally, CCSG significantly enhances the estimation performance of LMs on PE tasks. In summary, this paper makes the following key contributions: • We introduce a contrastive loss based on counterfactual samples and design a novel mechanism for generating positive and negative commonsense counterfactual samples. • We leverage the Structural Causal Model (SCM) to analyze the commonsense bias from a causal perspective and propose using a counterfactual samples constructor as an intermediary to mitigate this bias through causal inference techniques. • The proposed CCSG method has been validated on 9 distinct datasets, demonstrating its effectiveness in reducing bias and achieving new benchmarks in performance for the PE task. 2 Related Work In this section, we review key advancements and methodologies in PE, causality for language processing, and spurious correlation, highlighting their relevance to our work. 2.1 Plausibility Estimation Previous studies have extensively explored the concept of verifying commonsense statements. Models such as I2D2 ( Bhagavatula et al., 2022 ) were designed to evaluate the validity of commonsense statements generated by other models. Similarly, the ENTAILER model ( Tafjord et al., 2022 ) assesses the validity of provided hypotheses, albeit with incomplete training, limiting its robustness. However, these models, trained on limited and domain-specific datasets, show restricted applicability across broader commonsense domains. Additionally, some efforts ( Liu et al., 2023 ) have employed pretrained language models (PLMs) to validate commonsense sentences ( Jung et al., 2022 ) . 2.2 Causality for Language Processing Recent studies have increasingly explored the integration of causal inference with language models in the field of NLP ( Hu and Li, 2021 ) . Applications include tasks like controllable text generation ( Madaan et al., 2021 ; Goyal et al., 2020 ) and counterfactual reasoning ( Mu and Li, 2023 ; Chen et al., 2023b

Read the complete text