Retrieving Floods without Floodlights: Topic Models as Binary Classifiers for Extreme Climate Events in German News Abstract In studies of media coverage of extreme climate events, NLP methods have become indispensable for identifying relevant texts in large news databases. Still, enough annotated data to train accurate deep learning-based classifiers from scratch is often not available. Topic Models have the advantage of being both unsupervised and interpretable, but are typically used only for exploratory analysis or data characterisation. In this study, we investigate how to employ Topic Models as binary classifiers for refining the retrieval of relevant news about seven types of extreme climate events in the German media. Our method relies on the posterior distributions estimated by Topic Models to select relevant documents, without modifying their training procedure. Using an annotated sample to guide the evaluation, we show that the probabilities assigned to keywords used to query news databases can also be informative for selecting relevant topics and improve sample precision. We compare our results to a fine-tuned text embedding classifier and an open-weight LLM, discussing observed trade-offs, e.g. the LLM’s lowest precision. Moreover, we show that results are hazard-dependent, which speaks against considering climate events as a single category in NLP tasks. Keywords: extreme climate events, German news, topic models, text classification, document retrieval Retrieving Floods without Floodlights: Topic Models as Binary Classifiers for Extreme Climate Events in German News Brielen Madureira 1,2 , Mariana Madruga de Brito 2 , Andreas Niekler 1,3 1 LeipzigLab - Climate Discourse, Leipzig University, Germany 2 Helmholtz Centre for Environmental Research - UFZ, Germany 3 Computational Humanities, Leipzig University, Germany brielen.madureira@uni-leipzig.de mariana.brito@ufz.de aniekler@informatik.uni-leipzig.de Abstract content 1.  Introduction Assume we are gathering news about floods events to study collective attention in the media. Simply querying a news database to retrieve documents containing the string flood would not only match news reporting on actual floods, but also many false positives. Consider this (obviously constructed) example: “ Soccer fans experienced a flood of emotions witnessing floodlights being turned on as players flooded the field: the game could finally begin after the risk of a flash flood was ruled out. ” This illustrates a central challenge in information retrieval: the term flood can have metaphorical senses, be part of compound nouns unrelated to climate or refer to a merely hypothetical hazard. Thus, despite the repeated presence of the term flood , this text is rendered unrelated to actual flood events. Pitfalls like that can emerge at the intersection of environmental and social sciences, such as in text-based climate impact and adaptation research. This field often relies on NLP methods to process texts about climate events and their consequences ( 1 ; 32 , inter alia ) . In this context, dictionary-based retrieval is a typical procedure: large databases are queried using a curated list of hazard-related keywords to find potentially relevant documents about e.g. floods, droughts or wildfires (e.g. 41 ; 23 ). But as we just saw above, the mere presence of a keyword in a document does not guarantee its relevance. If term presence or frequency are directly used as predictors in quantitative assessments, research validity is impaired. Figure 1 : Relevant news articles can be identified based on the saliency of relevant topics in their representation estimated by a topic model. Although keyword-based matching serves as a reasonable prefiltering step for creating an initial collection of documents with good recall, retrieval results must be further refined using other classification methods to detect true and false positives and improve their precision. This is an open problem recently discussed by 15 . In this paper, we investigate the possibility of yielding a binary classification model for identifying relevant documents using the probabilities estimated by unsupervised vanilla Topic Models (TM), as the overview in Figure 1 . We assume a situation with a small amount of annotated data that is not enough for training deep learning-based models from scratch but still informative for evaluation. Our main contributions in this paper are: i a data analysis of news articles in German annotated with seven types of extreme climate events; ii the usage of TMs for relevance classification without any needed modification on the training regime and no direct human effort in topic interpretation; and iii evidence that TMs are, for some hazards, on par with deep learning alternatives, with the advantage of interpretability and a tendency to higher precision. 2.  Related Literature Retrieval of environment-related documents Document retrieval is an ubiquitous step in creating corpora for socio-environmental research. To name a few recent large-scale approaches, 20 implemented a graph-based heuristic on Wikipedia metadata of entries on climate topics, 18 relied on climate-related keywords to retrieve news and 7 used a hazard event database for a targeted query of news articles and refined results using a Large Language Model (LLM). Our work focuses on the step of refining an initial sample of documents retrieved via keyword-matching methods. Topic Models TMs such as Latent Dirichlet Allocation (LDA) ( 6 ) and Non-Negative Matrix Factorization (NMF) yield distributions of topics in documents in an unsupervised fashion. Many variations exist, e.g. keyword-assisted TMs ( 12 ) , which exploit keywords to guide clustering and circumvent post hoc topic interpretation, CorEx ( 14 ) , which relies on an information-theoretic framework, and Top2Vec ( 3 ) , which performs clustering in an embedding space shared by documents and words. TMs often aid characterising corpora in climate and ecology research ( 21 ; 42 ; 28 ; 44 ; 34 ; 5 ; 4 , inter alia ) . Tuning and assessing the quality of TMs is intricate if performed purely intrinsically ( 29 ) , but our evaluation is enhanced by annotated data that allows known document properties to be compared to the formed topics. TMs for text classification TMs have been widely used to map texts to classes, e.g. by feeding their outputs as input features for other classifiers ( 26 ; 2 ; 40 ) . Other works aligned topics to classes, either directly ( 37 ) , by experts ( 16 ) or by configuring the priors in a way that induces desired clusters ( 31 ; 36 ) , e.g. by relying on relevant keywords ( 10 ; 45 ; 22 ; 25 ) . Keyword selection can also derive a lower dimensional set of features for other types of classifier models ( 33 ) . 30 incorporated a response variable into the TM training, to jointly model documents and their classes or scores. While many procedures require adjusting priors or the modelling approach, we stick to standard LDA and NMF implementations, which are arguably more accessible for newcomers and researchers from other fields. Text classification in climate research Climate-related text classification is an established NLP task; in many settings, it remains an unsolved problem even for LLMs, with performance often well below 0.75 F1 in the ClimateEval benchmark ( 19 ) . In the study by 24 , a fine-tuned encoder achieved an F1 of 0.98 for identifying relevant documents on climate extreme impacts, but only in English and on a small sample of cleaner Wikipedia entries with climate-related keywords in their titles . This restriction likely ensured a majority of relevant matches, but resulted in an unknown number of missed cases. When full texts are considered (as we do), there is less room for false negatives while substantially increasing the need for filtering out false positives, especially with imbalanced datasets. The problem we tackle in this paper is similar to the work by 15 : corpus construction via keyword-based prefiltering and automatic classification. We differ by focusing on German, handling specific hazards separately and exploring TMs for classification, not only for topic analysis as that work did. Our design builds upon existing work with a novel perspective: we do not change the LDA and NMF internal mechanisms and explore the posterior probabilities (or normalised scores) they assign to keywords as a means to automatically partition topics and perform binary classification of news about extreme climate events. 3.  Methods This section formalises the task and explains how topic models are applied for binary classification. Then, it describes the two deep learning strategies used for comparison. 3.1.  Task Formalisation Let D be a set of documents d , each belonging to a binary class C=\{0,1\} , and V be the set of all tokens w that appear in D . Class 1 represents relevant documents. A document classifier is a function f\colon D\to C that maps documents to classes and can be approximated by various methods. Furthermore, let F\subseteq V be a set of feature tokens selected from V based on given criteria (e.g. minimum frequency and part-of-speech tags) and K\subset F be a small set of predefined tokens of interest which we name keywords . A trained topic model M with n topics T estimates two distributions: p_{feat} , the probability of a feature token in a topic and p_{topic} , the probability of a topic in a document. In other words, a document is represented as a probability distribution over topics and a topic, as a probability distribution over feature tokens. With M ’s estimations, we can define a binary relation R between D and T representing whether each document is related to each topic. To use M for binary classification, a partition of topics T with two sets is created, each corresponding to a class in C . The class of a topic must also correctly classify documents related to such a topic. 3.2.  Classification with Topic Models Firstly, a TM is trained on the entire collection of unique documents using selected hyperparameters, following standard procedures (described in Section 5 ). Then, two further steps are needed: (i) assigning topics to documents and (ii) identifying which topics are to be regarded as relevant. For (i), we define the relation R as p_{topic}(t,d)\geq\theta with 0\geq\theta\geq 1 . That means that if the proportion of a topic t in a document d is at least a threshold, we consider that d discusses topic t (as in Figure 1 ). For (ii), we propose two ways to partition topics into two classes, relevant and not relevant, avoiding the usual post hoc human interpretation in TMs: • keyword proximity : topic t is assigned to the relevant class if \exists w\in K:p_{feat}(w,t)>\gamma . In other words, if the topic assigns a high enough probability to at least one keyword, the topic belongs to the partition of the relevant class. • top terms : if there is a keyword among the top k features of a topic (ranked by probability), the topic is assigned to the relevant class. The actual classification of each document is made as follows: if the document is related to at least one of the relevant topics, we consider it to be relevant. Otherwise, it is classified as not relevant. 3.3.  Deep Learning Classifiers The performance of our TM approach is compared to two deep learning alternatives: a fine-tuned text embedding model and an LLM. The first is a binary classifier trained using the SetFit framework ( 43 ) which fine-tunes a pretrained text embedding model with a classification head, aiming at optimising task-specific embeddings based on a set of contrastive examples. The latter prompts an LLM to generate a binary label classifying the document as relevant or not. The implementation details are explained in Section 5 . 4.  Data The data for this study derives from an ongoing project on the collective attention to extreme climate events in the German media. Seven types of hazards were selected (cold waves, droughts, floods, heat waves, landslides, storms and wildfires). The wiso-net news aggregation database 1 1 1 https://www.wiso-net.de/ was queried using a pre-defined list of hazard-related keywords, similar to ( 24 ; 28 ; 8 , see Appendix) . The retrieved collection contained 13,771,411 German news articles from around 370 outlets, spanning from 2000 to 2024, split into separate sub-collections for each extreme climate event. Figure 2 : Wildfires as the main topic (top) or mention (bottom) in news excerpts. Figure 3 : % of documents per year (left) and % of documents per extreme climate event (right). We make a distinction between two types of relevant news: main , in which the extreme climate event is the main topic, and mention , that refer to it en passant , of secondary importance among other more prominent topics, as shown in Figure 2 . Both forms count towards levels of collective attention, but automated identification of the latter is more challenging due to brevity and underspecification. We are primarily interested in the coverage of international events in German news, so basic rule-based procedures were implemented to reduce the amount of local or unrelated news. As the data we are working with was queried via unrestricted keyword matching, many false positives occurred. Of particular relevance to this paper is the filter for what we call “intruder” keywords, i.e. words that derive from valid hazard-related keywords but are unrelated to climate events. For instance, the name Dürrenmatt , retrieved by Dürre (drought), Flutlicht (floodlight), by Flut (flood) and Stürmer (forward player in football), by Stürme (storms). To reduce the number of unrelated documents, we excluded all instances that contained only intruder keywords. Other preprocessing steps for filtering and cleaning the data are summarised in the Appendix. Final document collection The previous steps resulted in a sample of 2,438,275 documents (17.71% of the originally retrieved instances). They have, on average, 537.19 tokens (std = 362.06) and 28.66 sentences (std = 20.12). The news database has an inherent temporal bias towards recent years, illustrated in Figure 3 (left). The distribution over types of hazards is also shown in Figure 3 (right). Gold standard A sample of 3,150 documents was randomly selected while ensuring uniform distribution across hazard types (450 each) and years (18 each per hazard) and no duplicates per hazard. Two annotators classified the news as relevant or not (see annotation instructions in the Appendix) while also judging whether the event of interest was the news’ main topic or just a mention. Table 1 shows the percentage of relevant documents identified in the annotated sample. The initial effectiveness of the dictionary-based approach, together with the rule-based preprocessing, depends on the type of extreme climate event: while the landslide portion already reached a precision of almost 0.6, cold and heat waves stayed below 0.2. Identifying relevant documents is not as straightforward as it may sound. A subset of 100 news was annotated by both annotators independently. The overall agreement proportion in the primary binary decision (relevant or not) was 0.77 ( \kappa= 0.53), indicating that there are sources of legitimate disagreement in this decision. Apart from potential errors, disagreements may stem from differences in perception of what constitutes an extreme and concrete natural event. Some cases which may have involved such subjectivity were (translated from German): • cold wave : “…the son, travelled on Monday during the snow chaos from Cologne to Wismar in order stay by his mother.” • drought : “the mine was found two weeks ago due to the low water level in the Rhine river.” • landslide : “we knew the situation when a country closed the border or a street was blocked for a week due to a landslide or something else.” relevant main topic cold wave 14.44 4.22 drought 36.00 4.22 flood 43.33 12.44 heat wave 19.78 2.44 landslide 58.67 17.56 storm 27.56 7.33 wildfire 41.56 21.11 Table 1 : Percentage of relevant documents for each type of extreme climate event in the gold standard. 5.  Experiments The varying estimated proportions of relevant documents for each hazard sample suggest that these phenomena manifest differently not only in their nature but also in their coverage and linguistic features. Therefore, each classification strategy was conducted for each type of extreme climate event separately. The annotated sample was randomly split into a training and a test set with 350 and 100 instances, respectively, for each hazard. The presented results were computed in the test split. 2 2 2 Note that the use of train/test splits depends on the classifier. TMs’s unsupervised fitting included all unique documents, since the objective here is not to generalise to unseen data but to optimise for topics that best fit our own documents. Still, only the train split was used to select the best model configuration to avoid overfitting to the test data in this choice. The text embeddings model used the train split for fine-tuning. The LLM was directly prompted with the test data in a zero-shot approach. Topic models Documents were preprocessed to extract their features partially based on the procedure by 15 . We used Spacy’s 3 3 3 https://spacy.io/ model de_core_news_lg to tokenize, lemmatise and label tokens with their part-of-speech tags. Tokens with less than 3 characters and stopwords were removed, as well as non-alphabetical characters. All tokens were lowercased. The feature selection involved two criteria: the term’s document frequency and part-of-speech tag. All keywords were kept as features, even if they did not meet the minimum frequency threshold, to ensure they had a chance to contribute to forming a topic. To avoid the induction of topics based on duplicated news, only one instance of texts with high similarity was included. Gensim’s 4 4 4 https://radimrehurek.com/gensim/ implementation of the LDA and NMF methods was used to train topic models. The number of topics was a hyperparameter. For LDA, the eta and alpha arguments were set to auto . We run various combinations of the three hyperparameters (minimum document frequency, part-of-speech tags and number of topics) and, for each model, we computed results varying the values for k , \gamma , for top term and keyword proximity, and \theta . For each hazard, we selected the best-performing models in the training split. Specific parameters and the final configuration that produced the results are in the Appendix. The code is available at https://codeberg.org/briemadu/tm-as-classifier . Text embeddings This classifier was trained via the Small-Text ( 38 ) wrapper implementation around SetFit (with its default configuration in HuggingFace) and Sentence Transformers ( 35 ) . In this method, the classification is performed by a logistic regression component on top of the fine-tuned text embeddings. We opted for the BAAI/bge-m3 text embeddings released by ( 9 ) due to the model’s multilingual capabilities and longer context length (8,192 tokens), since standard Sentence Transformers that typically allow only up to 512 tokens would not suffice for longer news articles. Training was performed with a batch size of 16 instances and a learning rate of 10^{-5} . LLM Since the purpose of this paper is not to benchmark LLM performance, we chose only one model to serve as a reference. Results were produced by mistralai/ministral-3-14b-reasoning . 5 5 5 https://huggingface.co/mistralai/Ministral-3-14B-Reasoning-2512 We selected an open-weight model that could be run locally and keep the data in our own infrastructure. 6 6 6 We did not compare results to closed commercial models as they are at odds with open science principles. The prompt contained instructions similar to those given to the annotators, including the definition of the hazard and of the labels, the hazard’s keywords and the main body of the news article. The exact prompt and values are in the Appendix. We had to programatically parse answers that included spurious prefixes before the actual label. Evaluation The models’ performance was quantitatively assessed with conventional binary classification metrics: precision, recall and F1 score of the positive class. The test sample’s precision and a presumed recall of 1 were used as a baseline to measure how much the classifiers improve retrieval precision without reducing its recall. The evaluation was enriched with a detailed analysis of the TM results. We present results for three variations of TMs: tm-f1 was run with the configuration that resulted in the highest F1 score (on the training split) in our hyperparameter search; tm-b uses the configuration that balanced precision and recall to be both as high as possible; and tm-p has the configuration with the highest precision while retaining some level of recall. We also compare results to an ensemble strategy that performs classification via majority voting across the outputs of tm-b , fine-tuned text embeddings and LLM classifiers. 6.  Results P R F1 n main baseline 0.350 1.000 0.519 58 tm-f1 0.637 0.710 0.672 56 tm-b 0.710 0.649 0.678 55 tm-p 0.808 0.396 0.532 47 fine-tuned 0.647 0.853 0.736 57 llm 0.583 0.976 0.730 58 majority 0.701 0.890 0.784 58 Table 2 : Aggregated results: binary precision, recall and F1 score of all classifiers in the test split and the number of news of type main correctly identified. P R F1 cold wave baseline 0.170 1.000 0.291 tm-f1 0.471 0.471 0.471 tm-b 0.583 0.412 0.483 tm-p 0.500 0.059 0.105 fine-tuned 0.297 0.647 0.407 llm 0.455 0.882 0.600 majority 0.542 0.765 0.634 drought baseline 0.440 1.000 0.611 tm-f1 0.517 0.682 0.588 tm-b 0.622 0.523 0.568 tm-p 0.938 0.341 0.500 fine-tuned 0.686 0.795 0.737 llm 0.525 0.955 0.677 majority 0.692 0.818 0.750 flood baseline 0.360 1.000 0.529 tm-f1 0.605 0.639 0.622 tm-b 0.595 0.611 0.603 tm-p 0.750 0.167 0.273 fine-tuned 0.737 0.778 0.757 llm 0.600 1.000 0.750 majority 0.738 0.861 0.795 heat wave baseline 0.200 1.000 0.333 tm-f1 0.423 0.550 0.478 tm-b 0.600 0.450 0.514 tm-p 0.600 0.150 0.240 fine-tuned 0.439 0.900 0.590 llm 0.322 0.950 0.481 majority 0.429 0.900 0.581 landslide baseline 0.580 1.000 0.734 tm-f1 0.785 0.879 0.829 tm-b 0.778 0.845 0.810 tm-p 0.816 0.690 0.748 fine-tuned 0.877 0.983 0.927 llm 0.826 0.983 0.898 majority 0.826 0.983 0.898 storm baseline 0.270 1.000 0.425 tm-f1 0.680 0.630 0.654 tm-b 0.800 0.593 0.681 tm-p 0.625 0.185 0.286 fine-tuned 0.558 0.889 0.686 llm 0.614 1.000 0.761 majority 0.714 0.926 0.806 wildfire baseline 0.430 1.000 0.601 tm-f1 0.773 0.791 0.782 tm-b 0.825 0.767 0.795 tm-p 0.844 0.628 0.720 fine-tuned 0.750 0.837 0.791 llm 0.662 1.000 0.796 majority 0.809 0.884 0.844 Table 3: Detailed results: binary precision, recall and F1 score of all classifiers in the test split for each hazard type. Aggregated results We first examine results aggregated over the whole test split ( n=700 ), i.e. including all extreme climate events. Table 2 shows precision, recall and F1 score for all classifiers. The rightmost column shows the number of news articles of type main that were correctly identified as relevant. All classifiers succeeded in considerably increasing the low proportion of relevant documents in the keyword-based sample, but TMs and deep learning strategies behaved differently in how precision and recall were balanced. While the LLM had almost maximum recall with a substantial margin over other models, its precision was the lowest. tm-p had the highest precision but at the cost of low recall. tm-b achieved the second highest precision with a more reasonable recall. The majority voting approach led to the highest F1 score. If we focus on the identification of news of type main , all classifiers (apart from tm-p ) performed very well, identifying at least 55 out of the 58 instances. Results by hazard Aggregated results can mask variations in performance for each underlying hazard. Table 3 summarises results by hazard type, in line with the fact that models were trained separately. We can see that metrics varied greatly depending on the phenomenon: the lowest best F1 score of 0.59 occurred for heat wave whereas the highest best of 0.92 was observed for landslide. Majority voting achieved the best F1 scores for five hazards and the fine-tuned text embeddings for the other two. The LLM consistently held the highest recall in all hazards. tm-b had the best precision in the three most imbalanced (cold waves, heat waves and storms). Figure 4 : High-level overview of how relevant topics were identified by tm-b . In the first row, bars represent the maximum probability each topic (x-axis) assigns to keywords. The second row highlights in dark the positions in the rank that contain keywords in each topic (x-axis). The purple horizontal bars represent the optimal \gamma and k , respectively, in our experiments (see Appendix for exact numerical values). Discussion In aggregated results, TM performance was indeed lower than that of deep learning strategies, but the moderate reduction of only around 0.06 in F1 score still provides a much-desired gain in interpretability: we can explain exactly why each document was classified as relevant. The deep learning strategies tended to incur more false positives whereas TMs could reduce the proportion of unrelated documents while causing more false negatives. Models with higher precision but low recall, like tm-p , can still be useful when sample precision is a priority, since a sample with low recall may still be representative and of enough size in large datasets. High precision helps reduce the impact of unrelated documents in downstream analyses. For situations in which news of type main are more important than mention , the two best TMs and deep learning classifiers worked in like manner, not missing the vast majority of instances. The majority voting approach seemed to draw out advantages from each model, achieving the best F1 score. Still, employing three computationally costly models for this task is hardly justifiable in practice, given the modest overall increase in the aggregated F1 score compared to single models. Classification of news turned out to be hazard-dependant. There was no one-size-fits-all best solution across all hazards. The fine-tuned text embeddings balanced precision and recall well in general, but in three hazards (cold waves, storms and wildfires) the F1 score of the TM approaches was on par or better than it, which is a very interesting finding given that TMs are unsupervised and do not rely on the currently prevailing deep learning paradigm. Landslides and wildfires were the easiest to identify with all metrics above 0.75 (except for the LLM’s precision for wildfire) in all models apart from tm-p . Cold and heat waves were the most challenging with suboptimal results even for the majority voting method. Note, however, that comparisons between models should be done with caution, as these experimental estimates by hazard type were computed from samples of only 100 documents each. Rare events become very sensitive to individual predictions in such a small sample. For instance, cold waves contain only 17 relevant documents on which to measure precision and recall, so that a single swapped prediction by a model would already cause a 5.8% increase or decrease in recall. 7.  Analysis Figure 5 : Top 10 terms in each of the topics considered as relevant for drought and wildfire. In this section, we explore TM’s interpretability by providing more details on the tm-b models’ behaviour. In our non-exhaustive hyperparameter search, LDA achieved the best performance for six hazard types, while NMF was superior only for drought. The optimal thresholds \theta for assigning topics to documents were between 0.028 and 0.076. Figure 4 illustrates how relevant topics were selected. The top terms decision method achieved the highest performance across five hazards with k values ranging from 1 to 5. Keyword proximity was superior only for drought and heat wave using \gamma= 0.09 and 0.036, respectively. The number of selected topics for each hazard varied from 1 to 6. Here we focus on wildfires and droughts as they had the smallest and largest differences in F1 score, respectively, in relation to deep learning strategies. For wildfires, topics were considered relevant if a hazard-related keyword was among their top 3 most probable terms. That resulted in 2 out of 50 topics being considered as relevant. For drought, keyword proximity selected 3 out of 500 topics as relevant. The top 10 lemmas representing these topics are shown in Figure 5 . Figure 6 illustrates the effect of \theta for wildfire’s leftmost relevant topic: how well (not) relevant documents are classified based on the \theta parameter for the rightmost relevant topic in fire: documents with topic probability above \theta are classified as relevant, with a few wrong predictions. Figure 6 : Example of the \theta threshold for assigning a relevant fire topic to documents. Above the purple horizontal line, documents are classified as relevant, with a few mistakes with respect to the gold standard. Wildfires Out of the 100 test instances, 83 were correctly classified. We inspected the 17 errors. The 7 false positives are texts that do refer to fires, but they are either not wildfires (e.g. fires in houses or industrial plants) or discuss technologies to combat wildfires. One of the documents describes a wildfire warning that also refers to an extinguished fire, which may have been missed by the annotators. Among the 10 false negatives, most contain mentions to wildfires occurring in discussion about other main topics (financial markets, heat waves, conferences) in documents that are a concatenation of various news articles (a problem we inherited from the original database were not able to fully solve automatically). In such cases, the fire-related topic may not have been salient enough to be assigned to the document. Although two topics were selected as relevant, only the left one was responsible for making all predictions on the test set. The second topic appeared in only one text, indicating that, although potentially relevant, it may have specialised too much during training. All documents of type main were correctly identified. Drought For this hazard, 65 documents were correctly classified, with 14 false positives and 21 false negatives. Many false positives treat drought as a broad phenomenon rather than a concrete event, for example, when discussing drought-tolerant plants, vegetation stress, or climate change. Such cases are difficult to distinguish using TMs and may also reflect ambiguities in the annotation. False negatives show no clear patterns. Since this model relied on 500 topics, the drought concept may have been diffuse across multiple topics: in Figure 4 , it is evident that drought keywords appear in various topics not selected as relevant. The only main document that was incorrectly classified has only one keyword, a compound noun ( Dürregebieten ), which was not included as a feature. Including all words containing keywords as features could have prevented this but it introduces additional noise from low-frequency terms that form topics. 8.  General Discussion This work was primarily motivated by the lack of a comprehensive global database of extreme climate disasters. Existing disaster databases, for instance the EM-DAT ( 11 ) , are shaped by reporting practices and inclusion thresholds (e.g. at least 10 fatalities), which have been widely discussed for their biased coverage toward large, well-documented events and wealthier regions, systematically under-representing some regions and hazards ( 17 ) . Our method is designed to support bottom-up data-driven analyses by bypassing the inherent incompleteness and structural biases of top-down lists of worldwide extreme climate events ( 13 ) . Our procedure permits the inclusion of news about events that did not meet the arbitrary inclusion criteria of disaster databases. Rather than claiming the superiority of one model type for news classification, we have provided evidence that the results are hazard-dependent. This is an important finding for climate impact and adaptation research: the way different hazards are reported in the news varies, so solutions that treat all climate-related hazards as a single category (i.e. disasters in general) risk masking important performance variations, leading subsequent conclusions to be biased towards those that are easier to identify. In this context, an advantage of our approach is that we explicitly consider hazards separately, enabling more reliable downstream analyses. The exact reasons for such differences require further investigation. First, each hazard is inherently distinct in the abruptness of its onset, its duration, its frequency, and its perceived severity. Then, media coverage can differ depending on socio-economic and geo-political factors. Finally, there is linguistic and discourse-related variation. For instance, while some keywords are very specific to climate events, others are polysemic and appear in multi-word expressions. The interdependencies among these layers are worth studying. Some events are hard to pinpoint even for humans, which can impact gold standards. Treating extreme climate events as a monolithic concept is thus not advisable in NLP tasks. Besides, since multi-hazard events occur in reality, another promising way forward is to analyse how they also co-occur in news. We aimed to reduce human input in TM interpretation by selecting thresholds automatically and minimizing hyperparameter choices. Further work can investigate whether manual selection of keywords and topics can improve results. Our preliminary experiments with CorEx and Top2Vec yielded comparable results, so we prioritised the more traditional LDA/NMF methods in this study. However, other TM variations can be further investigated, including tuning priors to promote clearer keyword-related topics. The fine-tuned text embeddings achieved some of the highest F1 scores using only 350 documents and can potentially be further improved with active learning ( 39 ) . LLMs are being uncritically employed for many NLP tasks. We have shown that even a model with 14b parameters was not sufficiently precise. Our results add to the evidence that LLMs require careful evaluation as any other model. If LLM-based approaches are to be used, TMs can still be helpful in shrinking the amount of unrelated documents (e.g. by excluding those that have high probability for totally unrelated topics), thereby reducing the considerable environmental and financial costs of using LLMs. 9.  Conclusion We have presented a comparative analysis of three binary classifiers for refining collections of news articles on extreme climate events retrieved via keyword-based approaches. Although the LLM and the fine-tuned text embeddings had a higher F1 score in general, the drop in comparison to TMs was 0.148 on the worst case (drought) but also only 0.001 on the best case (wildfire). This is remarkable given TMs’ unsupervised training and the simplicity of the keyword-guided topic selection process. Depending on the use case, this difference may be acceptable given other advantages, such as higher precision. Besides, the reason for deep learning-based predictions are beyond human comprehension, whereas decisions based on TMs are fully transparent and explainable. Limitations The rule-based filtering may have excluded relevant documents, although it was a price worth paying to reduce the immense volume of unrelated news and to keep the task computationally tractable. Although we are seeking to identify extreme climate events, other types of disasters (.e.g. urban fires and industrial accidents that cause dam collapse) could not yet be fully distinguished by our methods. The test samples for each extreme climate event contain only 100 documents each, which may obscure variance in the estimates. More definitive claims about differences in models’ behaviour require cross-validation and, ideally, a larger sample. The performance of the classifiers is bounded by the quality of the annotation. Despite best efforts, ambiguity is not always easy to resolve and arbitrary decisions can impact models’ training and evaluation. We presented results for varying TM set-ups as we opted for selecting the best-performing configurations. Still, keeping it constant would facilitate the direct comparison across hazards. The hyperparameter search for TM considered only a few dozen combinations of the number of topics, POS-tags and minimum document frequency. This can potentially be further refined for each hazard separately. We did not perform extensive prompt engineering for the LLM, as these models are supposed to parse natural language instructions well; still, given their unpredictable nature, minor changes to the prompt might have led to different outcomes. Larger models may yield better results, but our focus here was on lower-scale, local solutions. Acknowledgements We thank Marc Keuschnigg for his contribution in conceptualising the research project that motivates this paper, as well as Maike Reichel and Julius Hehenkamp for their help in annotating the data. We also thank the anonymous reviewers for their valuable feedback. 10.  Bibliographical References Appendix A Appendix Further details about preprocessing and filtering The lists of German keywords used for each hazard are shown in Figures 8 , 9 , and 10 . Although derecho was included as a keyword initially, texts containing only this keyword were removed in a preprocessing step. The inclusion of Regenfälle (rainfalls) as a keyword for flood resulted in the inclusion of some texts that may not be about floods. Here, we provide a summary of the preprocessing and filtering steps applied to the original document collection. Exact implementation details are documented in the preprocessing code which is available upon request. We removed exactly duplicated instances, i.e. those pairs or groups of documents for which all metadata values were exactly the same. Documents with the same text but published by different outlets or on different dates were kept as they count separately towards media attention. The regex pattern ’<.*?>’ was used to remove reminiscent html content. To split (at least part of) the documents that have been concatenated as a single instance, despite being composed of several different pieces of news, we used another regex pattern with frequent news agencies abbreviations (e.g. dpa and afp) that often appeared in parentheses in between such concatenations. Approximately duplicated texts were identified using the MinHash algorithm to estimate Jaccard similarity, with a threshold of 0.8 slightly more conservative than the empirical choice in ( 27 ) . This was not used to exclude any document, but helped ensure that annotators did not annotate the same text (from different news sources) for the same hazard and that TMs were not trained on similar texts that would form spurious clusters. Spacy’s German model de_core_news_lg was used to parse each text and retrieve tokens and sentence counts. We also applied filters to reduce the number of unrelated documents and local news. The inclusion criteria were as follows: • The document contains at least one keyword related to its assigned hazard. Although this was an imposed criterion for the database query, after splitting concatenated documents, were a few cases of texts that no longer contained keywords. • The document contains at least one keyword of its assigned hazard which is not an intruder. • The document’s outlet reportedly belongs to the German press. • In case of exact duplicates (regarding all fields), only one instance was kept. • The document’s ressort does not contain the word lokal , since we are only interested in international extreme climate events. • The number of tokens is at least 30 and no more than 1,700. The thresholds were selected based on empirical observations of the distribution’s histogram and by taking into account an initial batch of annotated documents. • The document contains at least one of the following: a country name, a nationality (as an adjective or a noun) or a city name. • The first token of the document is not the name of a German city followed by a full stop. • The proportion of non-alphabetical characters is less than 0.11. The threshold was selected based on empirical observations of the distributions, also considering an initial batch of annotated documents. Further details about the annotation Figure 7 shows the instructions given to the two annotators. They also identified the sentences that refer to each type of hazard and the country where it occurred. These variables will be used in future studies. Figure 7 : Instructions given to the annotators. Further details about the classifiers LLM Figures 8 , 9 , and 10 show the values used to fill in the hazard-dependent slots in the prompt for the LLM, which is shown in Figure 11 . Definitions were translated from the EM-DAT’ glossary 7 7 7 https://doc.emdat.be/docs/data-structure-and-content/glossary/ , except for storm, not defined by EM-DAT, for which we used Wikipedia 8 8 8 https://en.wikipedia.org/wiki/Storm . Topic Models The number of iterations and passes were fixed at 400 and 20, respectively. The random seed was set to 123. Table 4 shows the number of documents used to train the TMs for each extreme climate event, i.e. the unique texts in the collection. Table 5 shows the selected hyperparameters for the topic model configuration of each extreme climate event. In Table 6 we show all topics selected as relevant for tm-b characterised by their top 10 terms with the highest probability. cold 91,140 drought 96,441 flood 334,607 heat 197,800 landslide 45,039 storm 488,068 fire 78,426 Table 4 : Number of documents used to train the topic models for each hazard. model decision method \theta \gamma or k min. doc freq tags topics tm-f1 landslide lda top terms 0.058 5.000 100 noun, verb, adj 100 fire lda top terms 0.040 3.000 50 noun, verb, adj 50 flood lda top terms 0.060 2.000 1000 noun, verb, propn 50 storm lda top terms 0.024 2.000 10000 noun, verb, adj 50 drought nmf keyword proximity 0.016 0.198 500 noun 500 heat lda keyword proximity 0.024 0.108 100 noun, verb, propn 100 cold lda top terms 0.024 5.000 500 noun, verb, adj 300 tm-b landslide lda top terms 0.076 5.000 100 noun, verb, adj 100 fire lda top terms 0.054 3.000 50 noun, verb, adj 50 flood lda top terms 0.062 2.000 1000 noun, verb, propn 50 storm lda top terms 0.030 1.000 10000 noun, verb, adj 50 drought nmf keyword proximity 0.028 0.090 500 noun 500 heat lda keyword proximity 0.052 0.036 100 noun, verb, adj 50 cold lda top terms 0.028 5.000 500 noun, verb, adj 300 tm-p landslide lda keyword proximity 0.064 0.054 50 noun, verb, adj 100 fire lda top terms 0.064 1.000 500 noun, propn 100 flood lda top terms 0.054 1.000 1000 noun, verb, propn 50 storm lda top terms 0.120 5.000 10000 noun, verb, adj 50 drought nmf top terms 0.034 1.000 500 noun, verb, adj 500 heat lda top terms 0.148 2.000 500 noun, verb, propn 50 cold lda keyword proximity 0.146 0.036 5000 noun, verb, adj 50 Table 5: Hyperparameters of all topic models that produced the presented results. Figure 8 : Keywords and values used in the prompts for each hazard (1/3). ⬇ landslide : hazard : Erdrutsch keywords : Erdrutsch , Felssturz , Felsst ü rz , Schlammlawine , Massenbewegung , Hangrutsch , Hangbewegung , Rutschung , Bodenrutsch , Hangabrutschung , Murgang , Ger ö lllawine , Rutschhang , Rutschh ä ng , Rutschgefahr , Felslawine , Mure hazard_event : Erdrutschereignisse definition : Jede Art von m ä ß iger bis schneller Bodenbewegung , einschlie ß lich Lahare , Schlammlawinen und Murg ä nge ( unter trockenen / nassen Bedingungen ). Ein Erdrutsch ist eine durch die Schwerkraft gesteuerte Bewegung von Erde oder Gestein , deren Geschwindigkeit in der Regel zwischen langsam und schnell liegt , jedoch nicht sehr langsam ist . Er kann oberfl ä chlich oder tief sein , aber das Material muss eine Masse bilden , die einen Teil des Hangs oder den Hang selbst ausmacht . Die Bewegung muss nach unten und nach au ß en mit einer freien Fl ä che erfolgen . ODER Jede Art von Abw ä rtsbewegung von Erdmaterialien unter hydrologisch trockenen Bedingungen . ODER Arten von Massenbewegungen , die auftreten , wenn starker Regen oder schnelle Schnee -/ Eisschmelze gro ß e Mengen an Vegetation , Schlamm oder Gestein unter dem Einfluss der Schwerkraft einen Hang hinunterbef ö rdern . fire : hazard : Waldbrand keywords : Fl ä chenbrand , Fl ä chenbr ä nd , Waldbrand , Waldbr ä nd , Wildfeuer , Landschaftsbrand , Landschaftsbr ä nd , Buschfeuer , Vegetationsbrand , Vegetationsbr ä nd , Naturbrand , Naturbr ä nd , Gro ß brand , Gro ß br ä nd , Forstbrand , Forstbr ä nd , Heidebrand , Heidebr ä nd hazard_event : Waldbrandeereignisse definition : Jede unkontrollierte und nicht vorgeschriebene Verbrennung oder das Abbrennen von Pflanzen in einer nat ü rlichen Umgebung wie Wald , Grasland , Buschland oder Tundra , die nat ü rliche Brennstoffe verbraucht und sich aufgrund von Umweltbedingungen ( z . B . Wind oder Topografie ) ausbreitet . Waldbr ä nde k ö nnen durch Blitzeinschl ä ge oder menschliches Handeln ausgel ö st werden . Figure 9 : Keywords and values used in the prompts for each hazard (2/3). ⬇ cold : hazard : K ä ltewelle keywords : K ä ltewelle , K ä lteeinbruch , K ä ltestress , extreme K ä lte , extremer K ä lte , extremen K ä lte , Frost , strenger Winter , Wintereinbruch , Wintereinbr ü ch , K ä lteperiode , K ä lterekord , arktische K ä lte , arktischer K ä lte , arktischen K ä lte , K ä ltewarnung , Eisregen , Eisesk ä lte , Schneechaos hazard_event : K ä ltewellenereignisse definition : Eine Periode mit ungew ö hnlich kaltem Wetter . In der Regel dauert eine K ä ltewelle zwei oder mehr Tage und kann durch starke Winde noch verst ä rkt werden . Die genauen Temperaturkriterien f ü r eine K ä ltewelle k ö nnen je nach Standort variieren . heat : hazard : Hitze keywords : Hitze , extreme Temperaturen , extremen Temperaturen , Temperaturrekord , Tropennacht , Tropenn ä cht , ü berhitzung , Rekordhitze , Hitzetag hazard_event : Hitzewellenereignisse definition : Eine Periode mit ungew ö hnlich hei ß em und / oder ungew ö hnlich feuchtem Wetter . In der Regel dauert eine Hitzewelle zwei oder mehr Tage . Die genauen Temperaturkriterien f ü r eine Hitzewelle k ö nnen je nach Standort variieren . drought : hazard : D ü rre keywords : D ü rre , Rekordd ü rre , Trockenperiode , Trockenheit , Wasserknappheit , Niedrigwasser , Wassermangel , Niederschlagsmangel , Niederschlagsdefizit , Bodenfeuchte - Defizit , Bodenfeuchteverlust hazard_event : D ü rreereignisse definition : Ein l ä ngerer Zeitraum mit ungew ö hnlich geringen Niederschl ä gen , der zu einer Wasserknappheit f ü r Menschen , Tiere und Pflanzen f ü hrt . D ü rren unterscheiden sich von den meisten anderen Gefahren dadurch , dass sie sich langsam , manchmal sogar ü ber Jahre hinweg , entwickeln und ihr Beginn in der Regel schwer zu erkennen ist . D ü rren sind nicht nur ein physikalisches Ph ä nomen , da ihre Auswirkungen durch menschliche Aktivit ä ten und den Wasserbedarf noch verst ä rkt werden k ö nnen . D ü rren werden daher oft sowohl konzeptionell als auch operativ definiert . Operative Definitionen von D ü rre , d . h . der Grad der Niederschlagsverringerung , der eine D ü rre ausmacht , variieren je nach Ort , Klima und Umweltbereich . Figure 10 : Keywords and values used in the prompts for each hazard (3/3). ⬇ flood : hazard : Hochwasser keywords : Ü berschwemmung , Flut , Hochwasser , Ü berflutung , Flusshochwasser , Regenf ä lle , Sturzflut , Gletscherseeausbruch , Gletscherseeausbr ü ch , Gletschersee - Ausbruchsflut , Jahrhunderthochwasser hazard_event : Hochwasserereignisse definition : Ein allgemeiner Begriff f ü r das Ü berlaufen von Wasser aus einem Flussbett auf normalerweise trockenes Land in der Aue ( Fluss ü berschwemmung ) , ü berdurchschnittlich hohe Wasserst ä nde entlang der K ü ste ( K ü sten ü berschwemmung ) und in Seen oder Stauseen sowie Wasseransammlungen an oder in der N ä he des Ortes , an dem der Regen gefallen ist ( Sturzfluten ). storm : hazard : Sturm keywords : Sturm , St ü rm , Unwetter , Orkan , Blizzard , Derecho , Hagel , Zyklon , Gewitter , Tornado , Mikroburst , Hurrican , Hurrikan , Taifun , Blizzard hazard_event : Sturmereignis definition : Ein Sturm ist jeder gest ö rte Zustand der nat ü rlichen Umwelt oder der Atmosph ä re eines astronomischen K ö rpers . Er kann durch erhebliche St ö rungen der normalen Bedingungen gekennzeichnet sein , wie z . B . starker Wind , Tornados , Hagel , Donner und Blitz ( Gewitter ) , starke Niederschl ä ge ( Schneesturm , Regensturm ) , starker Eisregen ( Eissturm ) , starke Winde ( tropischer Wirbelsturm , Sturm ) , Wind , der bestimmte Substanzen durch die Atmosph ä re transportiert , wie z . B . bei einem Staubsturm , sowie andere Formen von Unwettern . ⬇ Du bist ein Experte f ü r die Klassifikation von Nachrichtenartikel bez ü glich der Existenz von Referenzen auf $hazard und extreme $hazard_event . Definition von $hazard : $definition Synonyme f ü r $hazard : $keywords . Die Nachrichtenartikel m ü ssen mit einem dieser Labels klassifiziert werden : - Label 1 : Das Dokument behandelt $hazard , extreme $hazard_event oder damit verbundene Auswirkungen . - Label 0 : Das Dokument hat KEINE Verbindung zu $hazard oder extremen $hazard_event . Es sind nur konkrete , spezifische Naturereignisse in der realen Welt relevant . Artikel , die sich lediglich mit der M ö glichkeit eines Ereignisses befassen , metaphorische Verwendungen , allgemeine Diskussionen ü ber die Art der Gefahr oder Ereignisse sind nicht relevant . Analysiere den Inhalt des Dokuments Satz f ü r Satz sehr sorgfaltig und vergib das Label 1 auch wenn nur ein Satz im Dokument relevant ist . Entscheide , welches Label f ü r diesen Nachrichtenartikel das richtige ist , und beginne deine Antwort entsprechend mit 0 oder 1. Klassifiziere den folgenden Nachrichtenartikel : $text Figure 11 : Prompt used for the LLM experiment. 1 2 3 4 5 6 7 8 9 10 landslide 98 see bergbau erdrutsch wasser rutschung tagebau gefahr siedlung erklären bereich 99 tal felssturz fels stein gestein meter kubikmeter berg stürzen groß 80 tote erdrutsch zahl leiche bergen opfer begraben vermissen vermisst verschütten 22 mensch erdrutsch leben überschwemmung haus schwer behörde heftig sterben region 60 haus bewohner unglück erdrutsch gebäude wohnung wohnen früh bürgermeisterin ursache wildfire 1 feuer brand waldbränd flamme waldbrand hektar feuerwehr region groß kontrolle 26 buschfeuer australisch koala bundesstaat kontinent buschbränd australier tier villa ostküste flood 43 mensch überschwemmung haus leben region dpa behörde stadt land angabe 20 hochwasser sachsen elbe dresden polen pegel meter donau brandenburg tschechien storm 16 unwetter wasser hochwasser überschwemmung heftig region schwer regen betreffen schaden 13 gewitter wetter regen blitz absagen regnen mark himmel heftig schlecht 7 hurrikan sturm bundesstaat treffen kilometer wirbelsturm land schaden windgeschwindigkeit stunde drought 457 dürre notstand ernteausfall vieh regenzeit ausmaß zentrum versicherung wasserreservoir helmholtz 246 mitte niedrigwasser tag fisch wasser stoff fischsterben sand kreis jugendliche 319 trockenheit waldbrandgefahr feuchtigkeit kanton brandgefahr ernteausfall nässe stress notstand leiter heat wave 22 grad temperatur hitze celsius tag hitzewelle sommer liegen wetter mensch cold wave 1 mensch sterben leben kältewelle erfrieren behörde obdachlose tote zahl dutzend 194 schneefall heftig stark schneechaos teil fallen sperren schneemasse verkehr blockieren 137 eisregen verspätung reisend glatt mittag behindern glätte vereist schiene störung 12 winter kalt mild wehen östlich wintermonat stark atlantik flachland luft 111 kälte warm wärme eisig frieren decke kleidung thermometer anziehen klirrend 119 wetter regen kälteeinbruch regnen wetterlage kalender bauernregel wetterstation eisheilig schlecht Table 6: Top 10 terms of each topic considered as relevant in the tm-b experiments. The second column contains an arbitrary topic ID.