Novel Benchmark for NER in the Wastewater and Stormwater Domain
Franco Alberto Cardillo, Franca Debole, Francesca Frontini, Mitra Aelami, Nanée Chahinian, Serge Conrad · 2025
Read Novel Benchmark for NER in the Wastewater and Stormwater Domain by Franco Alberto Cardillo, Franca Debole, Francesca Frontini, Mitra Aelami, Nanée Chahinian, Serge Conrad (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01938.txt.
Introduction
Teaching software to tag the technical terms in water utility documents is hard for want of practice material, and Franco Alberto Cardillo and five co-authors built some in a 2025 paper. They assembled a French-Italian corpus of wastewater and stormwater texts, marked up with the domain's entities, places, chemicals, processes, and infrastructure.
Against that benchmark they ran current named-entity recognition methods, including large language model approaches, to set a baseline for the field. The work also explores projecting annotations automatically from one language to another, so the corpus can grow into new languages without starting the labeling over. The authors frame the benchmark as the first step toward extracting structured knowledge from the reports and regulations that water managers drown in.
Catalog record
| CREATOR | Franco Alberto Cardillo, Franca Debole, Francesca Frontini, Mitra Aelami, Nanée Chahinian, Serge Conrad |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01938
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Novel Benchmark for NER in the Wastewater and Stormwater Domain Franco Alberto Cardillo, Franca Debole, Francesca Frontini, Mitra Aelami, Nanée Chahinian, Serge Conrad Abstract Effective wastewater and stormwater management is essential for urban sustainability and environmental protection. Extracting structured knowledge from reports and regulations is challenging due to domain-specific terminology and multilingual contexts. This work focuses on domain-specific Named Entity Recognition (NER) as a first step towards effective relation and information extraction to support decision making. A multilingual benchmark is crucial for evaluating these methods. This study develops a French-Italian domain-specific text corpus for wastewater management. It evaluates state-of-the-art NER methods, including LLM-based approaches, to provide a reliable baseline for future strategies and explores automated annotation projection in view of an extension of the corpus to new languages. 1 Introduction The effective management of wastewater and stormwater systems is crucial for urban sustainability and environmental protection. These systems, which form an integral part of public infrastructure, require structured information for monitoring, planning, and maintenance. However, much of the relevant information exists in unstructured textual formats, such as technical reports, regulatory documents, and maintenance logs. Extracting information from these sources is a key challenge, due to domain-specific terminology and the multilingual nature of regulatory and operational contexts. Typically a wastewater management information extraction application will require domain-specific entity recognition, followed by the extraction of relations between entities to support decision-making, automated reasoning, and linking to existing knowledge bases. The recent progresses in domain-specific Named Entity Recognition (NER) have the potential to greatly facilitate the development of such applications. However, to effectively evaluate this first and crucial step of the extraction pipeline, it is essential to establish a clearly defined set of extractable entities and construct a multilingual benchmark corpus . Building on previous work - carried out within the framework of a national project on just one language - we propose the following contributions: • The starwars corpus , an aligned French-Italian corpus containing domain-specific texts. • Experimental evaluation of domain-specific NER method LLM-based approaches. • Experimental evaluation of automated projection of annotations from one language to another, to facilitate the multilingual extension of the benchmark. The resulting multilingual benchmark provides a foundation for further research and practical applications in the field of environmental information systems, first and foremost relation extraction (RE), and entity liking. However, such aspects will require further work. The remainder of this paper describes the corpus creation and annotation process, and NER experiments on both the Italian and French corpus, as well as the projection of annotation from one corpus to the other, and an analysis of the results. 2 Related work Benchmarks, providing annotated corpora with domain-specific labels, as well as evaluation metrics, have been developed to test the effectiveness of NER in various domains. In the biomedical domain for instance, the BC5CDR corpus [ 1 ] consists of 1500 PubMed articles with 4409 annotated chemicals, 5818 diseases and 3116 chemical-disease interactions. The SciERC [ 2 ] is a collection of 500 scientific abstracts annotated with scientific entities, their relations, and co-reference clusters. The BiodivNERE [ 3 ] is a gold-standard corpus specifically designed for NER and RE tasks within the biodiversity domain. This resource aims to enhance information extraction from biodiversity literature by providing annotated data tailored to the field’s unique terminology and relationships. The corpus includes detailed annotations for entities such as organisms, phenomena, relying also on an underlying ontology for classes and relations. The emergence of such domain specific benchmarks is paralleled by the development of domain-tuned NER models, handling terminology and contexts in various fields. For example, the use of models such as BioBERT [ 4 ] , that are fine-tuned on domain corpora, can achieve high accuracy on field-specific entity types [ 5 ] . More recently Large Language Models (LLMs) have begun to transform information extraction in technical and scientific domains, with the capacity to handle the various steps of entity extraction (extraction, normalisation, linking to existing knowledge) with little or no fine-tuning required. While some domains have been highly investigated, and with some evidence of good performances even with zero or few-shot approaches [ 6 ] , there is still a lack of exploration in water sciences. Some experiments on question answering, such as WaterGPT [ 7 ] , show that fine-tuning may be required. For the specific field of Wastewater networks, NER is further complicated by the lack of an agreed upon annotation schema and domain ontology, although the French Covadis standards 1 1 1 http://www.geoinformations.developpement-durable.gouv.fr/ constitutes an excellent starting point. Multilinguality is another important dimension to consider, since texts containing relevant information are produced by local institutions in different languages. In general, available multilingual NER benchmarks such as MultiCoNER [ 8 ] do not contain texts and entities for all specific domains. Multilingual LLMs can be successfully applied in specialized fields (medicine, law, etc.) across different languages with or without fine-tuning, but performances may vary across language and yielding uneven results [ 9 ] . Table 1: The set of 14 tags in three languages French Italian English Appareillage Attrezzatura Equipment Spatial Spaziale Spatial Evénéments Evento Event Forme Forma Shape Fonction Funzione Function Defaillance Guasto Fault Matériau Materiale Material Mesure Misura Measurement Mode_circulation Modalità_circolazione Circulation_Mode Organisation Organizzazione Organization Ouvrage Struttura Structure Temporel Temporale Temporal Type_reseau Tipo_rete Network_Type Canalisation Tubazione Pipeline The most relevant datasets to date have been developed as part of the ‘‘Megadata, Linked Data and Data Mining for Wastewater Networks’’ (MeDo) project 2 2 2 http://webmedo.msem.univ-montp2.fr/ , funded by the Occitanie Region, in France, and carried out in collaboration between hydrologists, computational linguists and computer scientists. An information extraction platform was designed [ 10 ] and a corpus composed of 1,557 HTML and PDF documents was collected in July 2018 using a set of Google queries with a combination of keywords. A Gold standard containing the documents used to train the NER modules and the corresponding annotation files is available in an open access repository [ 11 ] . While constituting a good basis, the corpus contains only documents in French and its spatial extent is limited to Metropolitan France. 3 Corpus creation and annotation The starwars Corpus , currently in French and Italian, was created by consolidating and extending the MeDo corpus to obtain a multilingual parallel benchmark. 3.0.1 Annotation scheme The starwars annotation scheme 3 3 3 Excerpts from the starwars corpus alongside the annotation guidelines, describing the tags and their application in more detail, will be made available and cited in the final version of this paper. is largely inspired by the UML schema of the RAEPA geostandard [ 12 ] and the application ontology developed by [ 13 ] . It contains 14 entities in total, 11 of which are common to RAEPA. The set of tags is in Table 1 . Table 2: Corpus-level Statistics French Italian Documents 110 110 Sentences 3 ,103 3 ,103 Annotated sentences (with 1+ tags) 1,726 1,726 Tokens 77 ,530 72 ,972 Annotated tokens (either B- or I-) 12 ,415 13 ,470 Entities (tokens annotated with B-) 5 ,357 5 ,324 3.0.2 French Corpus The French corpus consists of 110 documents, 45 of which are from the MeDo Gold Standard, while the remaining were collected