EventFull : Complete and Consistent Event Relation Annotation
Alon Eirew, Eviatar Nachshoni, Aviv Slobodkin, Ido Dagan · 2024
Read EventFull : Complete and Consistent Event Relation Annotation by Alon Eirew, Eviatar Nachshoni, Aviv Slobodkin, Ido Dagan (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-12733.txt.
Introduction
Alon Eirew and co-authors present a tool for marking how events in text relate. Events can precede, cause, or be the same as each other. Hand marking all pairs in a document is slow, since pairs grow with the square of the count. Most data sets thus skip pairs and lack full coverage.
Their tool, EventFull, guides markers through time, cause, and identity links in one joined flow. The design keeps marks consistent across the whole document. A pilot run shows the tool speeds the work and keeps quality high, with strong agreement between markers. The authors aim it at builders of richer event data sets.
Catalog record
| CREATOR | Alon Eirew, Eviatar Nachshoni, Aviv Slobodkin, Ido Dagan |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-12733
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EventFull : Complete and Consistent Event Relation Annotation Alon Eirew, Eviatar Nachshoni, Aviv Slobodkin, Ido Dagan Abstract Event relation detection is a fundamental NLP task, leveraged in many downstream applications, whose modeling requires datasets annotated with event relations of various types. However, systematic and complete annotation of these relations is costly and challenging, due to the quadratic number of event pairs that need to be considered. Consequently, many current event relation datasets lack systematicity and completeness. In response, we introduce EventFull , the first tool that supports consistent, complete and efficient annotation of temporal, causal and coreference relations via a unified and synergetic process. A pilot study demonstrates that EventFull accelerates and simplifies the annotation process while yielding high inter-annotator agreement. 1 1 1 The tool and code are publicly available at https://github.com/AlonEirew/EventGraphAnnot . 1 Introduction Identifying the semantic relations between events mentioned in a text, notably temporal, causal and coreference relations, has been a fundamental goal in NLP. Substantial efforts have been devoted to developing various datasets that capture some or all of these relations O’Gorman et al. 2016 ; Hong et al. 2016 ; Wang et al. 2022 . These datasets were then leveraged to develop and to evaluate corresponding models for detecting event-event relations Hu et al. 2023 ; Guan et al. 2024 . The output of such models has been utilized in a range of downstream applications, with recent examples including event forecasting Ma et al. 2023 , misinformation detection Lei and Huang 2023 , and treatment timeline extraction Yao et al. 2024 , among others. Figure 1: The EventFull annotation pipeline begins with a document containing marked targeted events and proceeds through three stages: [1] Temporal Relations: annotators establish temporal relations between event pairs, supported by three processes, including pair prioritization strategy, consistency checking, and transitive relation detection (§ 3.2 ); [2] Coreference: annotators identify coreferring event mentions; [3] Causal Relations: annotators determine causal relations for pairs of events. Models for detecting event relations are expected to produce complete output, that is identifying all event relations that can be inferred from the given text. Accordingly, such models should ideally be evaluated, and trained, on datasets in which event relation annotation is in itself complete, in the sense that each pair of targeted events has been classified for its potential relationships. Unfortunately, though, exhaustive manual annotation of all event-event relations is typically considered extremely challenging or impractical, since the number of event pairs to be considered is quadratic in the number of targeted event mentions in the text Naik et al. 2019 ; Rogers et al. 2024 . As the number of event mentions grows, this task quickly becomes both too time consuming and cognitively unmanageable. Faced with this inherent annotation complexity, many datasets adopted an annotation protocol that restricts the number of events or event pairs considered for annotation through various restrictions and heuristics. Notably, TB-Dense Chambers et al. 2014 , ECB+ Cybulska and Vossen 2014 , MEANTIME Minard et al. 2016 , and EventStoryLine Caselli and Vossen 2017 restrict event pairs to a span of two consecutive sentences. This limitation inherently prevents testing and training models on longer-range relations. Other datasets, such as TimeBank Pustejovsky et al. 2003b and MAVEN-ERE Wang et al. 2022 , did not publish a systematic annotation execution protocol that guarantees actual complete annotation, and were subsequently criticized for being incomplete in their relation annotation Pustejovsky and Stubbs 2011 ; Rogers et al. 2024 . Further, some researchers aimed to avoid the cost of manual annotation altogether and employed fully- or partly-automatic dataset creation methods Mirza et al. 2014 ; Madaan and Yang 2021 ; Tan et al. 2024 . These approaches inherently incorporate biases introduced by the employed automated method, making them less reliable for testing purposes while being prone to yield biased models when used for training. Finally, motivated by similar observations to ours, the recent NarrativeTime project Rogers et al. 2024 does emphasize relation annotation completeness, supported by a corresponding annotation tool. However, their work addresses only temporal relations, while employing a complex annotation scheme that requires expert annotators. Indeed, their actual annotation was performed by two of the authors, overall limiting the replicability of this approach. In this paper, we aim to close major gaps in prior annotation protocols. To that end, we introduce a simple-to-use annotation tool that facilitates future creation of event relation datasets (for covering multiple relation types, new genres and languages, etc.), which guarantees complete relation annotation . Our web-based tool, EventFull (Figure 1 ), fulfills four critical goals, which constitute our primary contributions: (1) supporting joint synergetic annotation of the three prominent event-event relation types: temporal, causal, and coreference; (2) for any given set of targeted events within the text, the gold output relations are guaranteed to be complete , classifying all event pairs for the three relation types; (3) supporting efficient annotation while guaranteeing annotation consistency, via automated transitive completion and consistency checks, while leveraging constraints imposed across the three relation types (see § 3.2 ); (4) ease of use by a non-expert annotator, lowering the bar for future dataset creation. To the best of our knowledge, EventFull is the first available tool that effectively supports and integrates all these goals. In the remainder of the paper, we first survey relevant background and related work (§ 2 ), followed by the presentation of EventFull’s input and output structure and its workflow (§ 3 ). Finally, we present a pilot study in which we demonstrate the effectiveness of EventFull (§ 4 ). 2 Background and Related Work This section first provides relevant background regarding event relations, followed by a short survey of prior annotation tools. 2.1 Event Relations We focus on the three event-event relations which have been addressed most broadly in the literature: temporal Do et al. 2012 — identifying the temporal order between events; coreference Raghunathan et al. 2010 — indicating whether two event mentions in the text refer to the same real-world event; and causal Mirza et al. 2014 --- detecting whether one event caused another event to happen. 2 2 2 Another relation type often considered is sub-event Glavaš et al. 2014 , which falls beyond the scope of this work. Event relations satisfy various constraints. Temporal relations, which connect any two events, must maintain transitive consistency Verhagen 2005 : if event A precedes B and B precedes C, then A necessarily precedes C Allen 1983 . Temporal order induces further constraints on the other two relations. In a causal relation, the causing event must precede the other Caselli and Vossen 2017 , while coreferring event mentions must temporally co-occur Cybulska and Vossen 2014 . As described in § 3.2 , we leverage these constraints in our tool to increase annotation efficiency while guaranteeing its consistency. Temporal and causal relations have been modeled at varying levels of granularity. The number of temporal relation types ranged from 13-14 in Allen’s interval algebra Allen 1984 and TimeML Pustejovsky et al. 2003a to 4-6 in recent approaches Chambers et al. 2014 ; Ning et al. 2018b ; Wang et al. 2022 , aiming to increase annotation consistency and scalability. Fine grained causal relations distinguish three sub-types — cause , precondition