An Empirical Study of Causal Relation Extraction Transfer: Design and Data Conference: ; ; CCS: Computing methodologies Machine learning; Artificial intelligence CCS: Information systems Information retrieval CCS: Software and its engineering Software testing and debugging CCS: Computing methodologi
Sydney Anuyah , Jack Vanschaik , Palak Jain, Sawyer Lehman, Sunandan Chakraborty . · 2025
Read An Empirical Study of Causal Relation Extraction Transfer: Design and Data Conference: ; ; CCS: Computing methodologies Machine learning; Artificial intelligence CCS: Information systems Information retrieval CCS: Software and its engineering Software testing and debugging CCS: Computing methodologi by Sydney Anuyah , Jack Vanschaik , Palak Jain, Sawyer Lehman, Sunandan Chakraborty . (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06076.txt.
Introduction
Sydney Anuyah and co-authors report a 2025 study on teaching computers to pull cause-and-effect links out of text. Such extraction helps search and question tools, but models trained on one source often fail on another. The team tested which network designs and data mixes carry over best between sources with different writing and labeling styles.
A fairly simple model, pairing a biomedical language model with a two-way memory network, generalized better than fancier designs. The team also built a scoring metric that rewards finding the right noun phrases, not just exact tag matches. Mixing training data from varied domains and label styles improved transfer. Gains were largest when the mix held a fair share of both stated and implied cause sentences.
Catalog record
| CREATOR | Sydney Anuyah , Jack Vanschaik , Palak Jain, Sawyer Lehman, Sunandan Chakraborty . |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-06076
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An Empirical Study of Causal Relation Extraction Transfer: Design and Data Conference: ; ; CCS: Computing methodologies Machine learning; Artificial intelligence CCS: Information systems Information retrieval CCS: Software and its engineering Software testing and debugging CCS: Computing methodologies Sydney Anuyah , Jack Vanschaik , Palak Jain, Sawyer Lehman, Sunandan Chakraborty . Abstract . We conduct an empirical analysis of neural network architectures and data transfer strategies for causal relation extraction. By conducting experiments with various contextual embedding layers and architectural components, we show that a relatively straightforward BioBERT-BiGRU relation extraction model generalizes better than other architectures across varying web-based sources and annotation strategies. Furthermore, we introduce a metric for evaluating transfer performance, F 1 p h r a s e F1_{phrase} that emphasizes noun phrase localization rather than directly matching target tags. Using this metric, we can conduct data transfer experiments, ultimately revealing that augmentation with data with varying domains and annotation styles can improve performance. Data augmentation is especially beneficial when an adequate proportion of implicitly and explicitly causal sentences are included. 1. Introduction Causality is essential for understanding the relationships between variables, enabling us to identify actual cause-and-effect dynamics rather than mere correlations. This understanding is crucial in fields such as medicine ( Bollegala et al. 2018 ) , where identifying causal factors can lead to effective treatments, and finance ( Mariko et al. 2021 ) , where it informs decisions with wide-reaching impacts. In technology and AI, causality improves the reliability and interpretability of predictive models. Establishing causality leads to better outcomes across various domains, often requiring the extraction of causal relationships from free-text sources like scholarly articles and news reports, which is considered a key task in the field of Natural Language Processing (NLP). Although the extraction of causal relationships may have direct applications to many downstream tasks ( Zhou et al. 2024 ) , a major goal of causal relation extraction is to construct knowledge graphs that can facilitate downstream uses. ConceptNet ( Speer et al. 2017 ) and CauseNet ( Heindorf et al. 2020 ) are examples of this; however, the rule-based extraction methods used for these datasets limit the type of causal relations present in the knowledge graph. CauseNet, for example, identified causal sentences in Wikipedia and ClueWeb12 with syntactic templates then used a neural sequence tagger to label causes and effects in the identified sentences. Though this method was able to create the largest available causal knowledge base, it could be missing many implicit causal relations. In this sentence from CauseNet, “Insomnia is often caused by fear, stress, and anxiety." the relationship is made explicitly clear by the highlighted marker “caused". Other markers (e.g., “leads to", “associates", “trigger", etc.) comprise the source sentences underlying CauseNet. By contrast, take “The clock struck twelve with a loud chime that made me jump." from ( UzZaman et al. 2013 ) . Causal semantics are present, but the lack of a clear marker means such sentences would have been excluded from CauseNet. This under-representation of implicit sentences could bias knowledge graphs and knowledge bases when rule-based extraction is used on a large scale. We have yet to see a large-scale causal knowledge graph extracted using both implicit and explicit source sentences. This paper addresses the root cause of that problem; causal relation extraction models are trained on disparate benchmark datasets that vary significantly in lexical composition and annotation style. Recent efforts have employed increasingly complicated cause/effect/relation annotation schemes ( Li et al. 2021 ) , which produce over-specified models that are not well suited for open-domain extraction. To address this, our analysis examines transfer across six unique causal relation datasets that span varying domains, annotation styles, and implicit/explicit causality makeups. An empirical analysis of architectural components including choice of embedding, recurrent unit, and usage of conditional random fields (CRF) is conducted. In particular, we see transformer-based contextual word embeddings ( Devlin et al. 2018 ) with specialized pre-training datasets, such as BioBERT ( Lee et al. 2020 ) and RoBERTa ( Liu et al. 2019 ) , work best across all six target datasets. Experiments further reveal that across domains, common architectures for relation extraction, BiLSTM, and Conditional Random Field (CRF) ( Li et al. 2021 ) are perhaps unnecessary for an open domain causal relation extraction model as these components barely improve performance at the expense of speed and generalizability over a simpler BiGRU. Our current focus is on extracting intra-sentence causal relations, as they are easier to identify and evaluate using existing benchmarks, providing a foundation for more complex tasks. While newer models like Mistral 7b, Llama, and GPT may offer better performance, this work is limited to BERT and Flair embeddings due to their balance of computational efficiency and robust performance in domain-specific tasks. These embeddings are also well-studied and integrated into existing NLP pipelines, making them a practical choice for understanding causality. The introduction of our F1_{phrase} metric allows for a fair comparison of transfer performance by recognizing the target of causal relation extraction as the cause/effect phrase subtrees of the dependency tree. The application of this metric reveals that many target datasets can benefit from data augmentation, regardless of domain and annotation styles. Using F1_{phrase} and the best transfer models, we further see that improved transfer performance relates to implicit/explicit sentence makeup of a dataset and noun phrase coverage training size, rather than training data size. Causal relation extraction from the web sources is a challenging task, due to noise in the open domain and lack of consistency across viable training datasets. Through the lens of phrasal evaluation, our results imply that models like BioBERT-BiGRU are particularly well suited to causal relation extraction in this setting since they can best incorporate disparate data domains and annotation styles. Large causal knowledge bases have relied on rule-based methods for localizing causal relations ( Heindorf et al. 2020 ) , creating a bias for sentences with explicit expressions of causality. This work shows that BioBERT-BiGRU, trained on a collection of disparate data and evaluated with F1_{phrase} offers a path towards richer extraction of causal knowledge from noisy web-based sources. Our work contributes: (1) An empirical evaluation of causal relation extraction models across a combination of datasets showed the high performance of specially pre-trained BERT embeddings in the transfer setting, as well as the unnecessariness of Bi-LSTM and CRF layers. (2) The F1_{phrase} metric which facilitates the empirical evaluation analysis. (3) Furthermore, we conduct additional qualitative analysis to determine what data characteristics contribute to transfer performance. We show that the proportion of implicit causal sentences in the data improves transfer, more so than merely increasing dataset size. Our code files and supplementary materials will be made available after the review process. 2. Related Work An early example of a machine learning application for causal extraction is Blanco et al. 2008 , modified in ( Zhang et al. 2023 ) where Bagging and Decision Trees were used to classify cause-and-effect information. A study by Egami et al. 2018 provides a conceptual framework for text-based causal inferences, generalizing a codebook function to fit several models and linking higher-dimensional text to lower-dimensional representations for making inferences and estimating treatment effects. While machine