Examining the Metrics for Document-Level Claim Extraction in Czech and Slovak
Lucia Makaiová, Martin Fajčík, Antonín Jarolím · 2025
Read Examining the Metrics for Document-Level Claim Extraction in Czech and Slovak by Lucia Makaiová, Martin Fajčík, Antonín Jarolím (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-14566.txt.
Introduction
Lucia Makaiova, Martin Fajcik, and Antonin Jarolim examine how to score document level claim extraction in Czech and Slovak. Claim extraction supports fact checking, yet scoring extracted claims is itself unsolved. The authors study ways to align two claim sets from the same document. A good alignment yields a similarity score for model output versus human labels.
The score can also measure agreement between human annotators. Experiments use a new data set drawn from comments under Czech and Slovak news stories. Informal language and strong local context make the task hard. Results expose limits of current scoring methods. The work aims at a more reliable evaluation framework for these languages.
Catalog record
| CREATOR | Lucia Makaiová, Martin Fajčík, Antonín Jarolím |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-14566
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Examining the Metrics for Document-Level Claim Extraction in Czech and Slovak Lucia Makaiová, Martin Fajčík, Antonín Jarolím Abstract Document-level claim extraction remains an open challenge in the field of fact-checking, and subsequently, methods for evaluating extracted claims have received limited attention. In this work, we explore approaches to aligning two sets of claims pertaining to the same source document and computing their similarity through an alignment score. We investigate techniques to identify the best possible alignment and evaluation method between claim sets, with the aim of providing a reliable evaluation framework. Our approach enables comparison between model-extracted and human-annotated claim sets, serving as a metric for assessing the extraction performance of models and also as a possible measure of inter-annotator agreement. We conduct experiments on newly collected dataset—claims extracted from comments under Czech and Slovak news articles—domains that pose additional challenges due to the informal language, strong local context, and subtleties of these closely related languages. The results draw attention to the limitations of current evaluation approaches when applied to document-level claim extraction and highlight the need for more advanced methods—ones able to correctly capture semantic similarity and evaluate essential claim properties such as atomicity, checkworthiness, and decontextualization. 1 Introduction Claim extraction is a key stage in the automatic fact-checking pipeline. Its purpose is to identify claims within a document that can be verified during the fact-checking process. In practice, this task is often coupled with the checkworthiness estimation. The resulting claims should therefore be atomic, decontextualized, and check-worthy, which we define as follows. • Atomic claim should express a single factual proposition. We also consider multi-part statements (e.g., “Marie Curie was a physicist and chemist.”) as atomic if they are presented as one inseparable unit in the source document. • Decontextualized claim should be self-contained and interpretable without relying on the surrounding text. • Check-worthy claim is factual claim that the general public will be interested in learning about its veracity [ 2 ] . Our work focuses on the problem of exhaustive document-level claim extraction. We investigate how to evaluate sets of extracted claims in a way that also properly reflects exhaustivity-ensuring that both under-generation and over-generation (i.e., generating too little, or too many claims respectively) are penalized. To this end, we propose and validate a simple evaluation framework based on an LLM-as-a-judge metric to assess claim extraction of language models on our collected datasets. They consist of claims extracted from comments under Czech and Slovak online news articles. Our goal is to find reliable ways to evaluate claim extraction for the purposes of fact-checking. We focus on news comments because they are a space where people actively discuss current topics and frequently spread potentially incorrect or misleading claims. This makes the domain especially important for developing tools that can help verify information and reduce the spread of misinformation. We select Czech and Slovak because these languages are underrepresented in current automated fact-checking research and lack large-scale, high-quality resources for claim detection and evaluation, which makes it challenging to build and improve systems for this task. Our contributions are: (i) Two novel datasets containing human-annotated document-level claims in Czech and Slovak. (ii) Proposed framework for measuring claim similarity at the document level. (iii) A human preference study to derive more meaningful interpretations of model scores across different evaluation metrics applied within our framework. 2 Related Work 2.1 Similarity Metrics We evaluate the quality of claim extraction by measuring the similarity between extracted and reference claims. Similarity metrics for comparing textual sequences vary in their approach and robustness. Purely lexical methods focus on a surface form of the word, and they fail to capture the claim’s similarity to the reference after paraphrasing [ 8 ] . They also fail to penalize the change of meaning properly when the words are kept the same. For example, if the reference is “The cat sat on a mat.” lexical metrics (such as ROUGE or edit distance) typically rate “The mat sat on a cat.” higher than “My cat sat on a rug.”, despite the former being semantically different (and nonsensical). Therefore, semantic metrics are prefered, because they can capture the similarity of semantics beyond surface-level lexical cues. BERTScore [ 8 ] is an embedding-based metric that computes similarity using contextualized BERT embeddings. Unlike metrics based solely on lexical overlap, BERTScore can capture paraphrasing and contextual differences. This is because the embeddings encode both token identity and surrounding context, causing semantically related words to have a low cosine distance. However, embedding-based semantic similarity metrics also have notable weaknesses. One of the main issues is their difficulty in handling words that reverse or substantially change the meaning of a sentence, most prominently negations [ 1 ] . For instance, the Czech sentences“Jan chodí do práce” (‘Jan goes to work.’) and “Jan nechodí do práce.”(‘Jan does not go to work.’) are likely to be assigned a high similarity score, despite conveying entirely opposite meanings. Following [ 6 ] , we implement an LLM-as-a-judge approach to evaluate similarity which is described in more detail in Section 3.5 . Because LLMs capture broader semantic and contextual nuances, this approach has the potential to alleviate the limitations observed in earlier metrics such as the negation problem. 2.2 Claim Extraction Currently, most of the fact-checking methods and benchmarks rely on sentence-level claim extraction, often from artificially constructed sentences. This makes the extraction task trivial. Document-level claim extraction, on the other hand, has been addressed scarcely. [ 3 ] focuses on document-level claim extraction, but in the end, they provide only one “central claim” per document as output, which they compare to the reference one generated by a professional fact checker. [ 7 ] introduce a document-level claim extraction and evaluation framework. They assess claims across three key attributes: entailment , coverage , and decontextualization . Their approach proposes a multi-stage evaluation network, rigorously designed to assess claim quality for use in decompose-then-verify fact-checking pipelines. Notably, in contrast to other works, they do not incorporate the atomicity and checkworthiness of claims, considering the former as lacking a well-defined endpoint and the latter as too subjective for reliable automated evaluation. We propose a simple evaluation framework that captures exhaustivity, penalizing both missing and redundant claims. This setup allows us to precisely examine how different metrics—specifically LLM-as-a-judge, BERTScore, and Edit similarity—perform and how their results diverge when compared to human judgment. We conduct this analysis on two novel datasets based on Czech and Slovak online news comments, which present unique linguistic challenges due to their informal and diverse nature. 3 Evaluation Method 3.1 Datasets The properties of our datasets used for evaluation are summarized in Table 1 . The comments used for their creation are sourced from articles in multiple Czech and Slovak online outlets between 2021 and 2024, and are included in a Hugging Face dataset available at https://huggingface.co/datasets/FactDeMice/CzechNews . Czech Gold Claims Dataset (CGCD) consists of 60 comments and their corresponding gold-standard claims, which were manually extracted by the first author. Each comment contains on average two claims, and the dataset is entirely in Czech. Czech-Slovak Agreement Dataset (CSAD) includes 120 comments with two-way annotations, meaning each comment was independently labeled by two different annotators. These comments contain on average