Synthetic Data Augmentation for Cross-domain Implicit Discourse Relation Recognition

Frances Yung, Varsha Suresh, Zaynab Reza, · 2025

Read Synthetic Data Augmentation for Cross-domain Implicit Discourse Relation Recognition by Frances Yung, Varsha Suresh, Zaynab Reza, (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20588.txt.

Introduction

Frances Yung and co-authors test a data trick for reading hidden links between sentences in this 2025 paper. Some text joins make sense only by inference: a cause, a contrast, a sequence, with no linking word. Naming that hidden relation is hard for language models, which lag far behind trained systems at it.

The team tried synthetic training data. A large language model wrote second sentences that follow a set relation, built from unlabeled text in a new domain. That extra data was meant to adapt a model trained on labeled text from another domain. On a large test set, the recipe brought no real gains. The authors conclude the generated samples were mostly unhelpful for this task. They also urge care: small score bumps without statistical backing can mislead.

Catalog record

CREATORFrances Yung, Varsha Suresh, Zaynab Reza,
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-20588

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Synthetic Data Augmentation for Cross-domain Implicit Discourse Relation Recognition Frances Yung, Varsha Suresh, Zaynab Reza, Abstract Implicit discourse relation recognition (IDRR) – the task of identifying the implicit coherence relation between two text spans – requires deep semantic understanding. Recent studies have shown that zero-/few-shot approaches significantly lag behind supervised models ( Chan et al., 2024 ; Yung et al., 2024 ) , but LLMs may be useful for synthetic data augmentation, where LLMs generate a second argument following a specified coherence relation. We applied this approach in a cross-domain setting, generating discourse continuations using unlabelled target-domain data to adapt a base model which was trained on source-domain labelled data. Evaluations conducted on a large-scale test set revealed that different variations of the approach did not result in any significant improvements. We conclude that LLMs often fail to generate useful samples for IDRR, and emphasize the importance of considering both statistical significance and comparability when evaluating IDRR models. 1 Introduction IDRR is the task of identifying the covert discourse relation (DRs) between two given text spans (the arguments: Arg1 and Arg2 ) in the absence of a specific discourse connective (DC) such as because or moreover . Explicit labelling of discourse structure is beneficial to inform LLMs in summarization tasks Li et al. (2016) ; Ishigaki et al. (2019) ; Xiao et al. (2020) ; Dong et al. (2021) ; Liu et al. (2023) ; Liu and Demberg (2024) . However, IDRR is challenging both for humans Hoek et al. (2021) and models ( Zeng et al., 2024 , SOTA, 56.50\% F1 and 64.87\% accuracy in) , particularly in a cross-domain setting Shi and Demberg (2019) ; Atwell et al. (2022) ; Liu and Zeldes (2023) ; Pyatkin et al. (2023) . Prompting of large pre-trained language models (LLMs), despite the human- or even superhuman-level performance in various reasoning tasks ( Mao et al., 2023 ; Bang et al., 2023 ; Gilardi et al., 2023 ; Törnberg, 2023 , e.g.,) , was found to be not successful in IDRR. Few-shot prompting using GPT-4 only reaches 30.90\% F1 and 29.40\% accuracy on the Penn Discourse Treebank ( Prasad et al., 2019 , PDTB 3.0,) , and 28.87\% F1 and 32.67\% accuracy on the multi-domain DiscoGeM corpus Scholman et al. (2022) , even with task-specific prompt engineering Chan et al. (2024) ; Yung et al. (2024) ; Omura et al. (2024) . On the other hand, recent work demonstrated that LLMs can instead be used to generate synthetic data to augment the PDTB 3.0, improving the performance of classes that the baseline struggles to predict Omura et al. (2024) . This work explores the application of synthetic data augmentation for cross-domain IDRR. Using raw texts from different target domains, we prompt LLMs to generate discourse continuations that express specific DRs. The generated data is then used to adapt the base model trained on human-annotated data of the source domain, which is the PDTB 3.0. We experimented with outputs of different LLMs, prompt templates, screening strategies and various advanced methods of domain adaptation, and evaluated on the entire DiscoGeM corpus, which is 4 times the size of the test set used in previous work Omura et al. (2024) . We only found marginal differences between models adapted to target-domain synthetic data compared with the base model. The benefit of generating synthetic data is unclear compared with direct application of the cross-domain base model, or using it to pseudo-label target-domain data. We have derived the following insights from the experimental results: 1. Synthetic data augmentation by LLMs does not improve IDRR under a cross-domain setting, where annotated target-domain data is unavailable for training nor validation. The synthetic data quality relies on the screening by the base model trained on large-scale annotated data from the source domain. 2. Previous positive results in in-domain IDRR may have been overly optimistic, as improvements were observed only under specific configurations, and the results exhibited high variability. 3. Fine-grained discourse inference remains a significant challenge for LLMs, compared with NLU tasks such as binary sentiment or topic classification Ubani et al. (2023) ; Piedboeuf and Langlais (2025) . 4. Manual analysis shows that the valid generated samples are often highly prototypical examples of the DR class. In contrast, DRs in real texts tend to be more ambiguous and rely on indirect inference. 2 Related Work 2.1 Synthetic data generated by LLMs Supervised learning algorithms rely on labeled data as training objectives but manual annotation is time- and cost-intensive. Using controlled text generation Hu et al. (2017) with LLMs through instructional prompts, large-scale training data was created for a range of NLP tasks, such as question-answering Puri et al. (2020) , textual similarity identification Schick and Schütze (2021) , NLU Meng et al. (2022) ; Liu et al. (2022) commonsense reasoning Yang et al. (2020) , and dialogue classification Sharma and Feldman (2023) , etc. Recent findings indicate that while some tasks benefit from synthetic augmentation, others do not Ubani et al. (2023) ; Møller et al. (2024) ; Piedboeuf and Langlais (2025) . For example, synthetic data augmentation has been shown to significantly improve performance in movie review classification (positive vs. negative) and question type identification (e.g., whether a question seeks a "reason" or a "number"). However, in ambiguous tasks like irony detection, models trained with synthetic data often underperform the baseline. 2.2 Implicit DR recognition The current SOTA of IDRR models are mostly based on fine-tuning or prompt-tuning of the RoBERTa model Xiang et al. (2022) ; Zhou et al. (2022) ; Zhao et al. (2023) ; Jiang et al. (2023b) ; Zeng et al. (2024) . Cross-domain IDRR remains challenging and understudied. We started our experiments with GOLF Jiang et al. (2023b) , one of the SOTA models trained on PDTB, but found no significant improvement over a standard RoBERTa model. Several previous studies have found that models do not generalize well to out-of-domain data Shi and Demberg (2019) ; Atwell et al. (2021) ; Liu et al. (2021) ; Scholman et al. (2021) ; Kurfalı and Östling (2021) ; Liu and Zeldes (2023) ; Braud et al. (2023) ; Li et al. (2024) . Furthermore, attempts to classify implicit DR by zero/few-shot prompting were not fruitful – both a standard multiple-choice template Chan et al. (2024) and multi-step templates with verification questions Yung et al. (2024) result in performance significantly below that of supervised models. Omura et al. (2024) introduced an approach to augment the PDTB with synthetic data. Specifically, given an Arg1 from the original PDTB training set and a DR label, an LLM is prompted to generate an alternative Arg2 . The generated samples undergo a secondary filtering stage via few-shot prompting to discard ambiguous cases. During model training, a weighted loss is applied to balance the original and synthetic samples. The augmentation strategy targets the most confusing DR classes, i.e. those with the lowest recall on the PDTB validation set. Performance gains were observed when augmenting the top-3 most confusing classes, but not top-1 nor top-5. 1 1 1 for the RoBERTa base classifier. When using RoBERTa large , augmentation of the top-5 classes was beneficial. Compared to a standard RoBERTa base model

Read the complete text