Profiling German Text Simplification with Model-Fingerprints
Lars Klöser, Mika Elias Beele, Bodo Kraft · 2026
Read Profiling German Text Simplification with Model-Fingerprints by Lars Klöser, Mika Elias Beele, Bodo Kraft (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-13050.txt.
Introduction
Lars Kloeser, Mika Elias Beele, and Bodo Kraft look at how well large language models make German text simpler. Simple text helps many readers, yet teams who build such tools lack a good way to check what a model tends to do. Style can shift in ways that plain scores miss.
The team built a tool they call the Simplification Profiler. It turns a batch of plain texts from one model into a rich print of that model. The print joins many traits into one view that people can read. By stacking prints from many texts, teams can spot habits, compare models, and repeat the check later. The aim is a full, fast, and repeatable test of how each model simplifies.
Catalog record
| CREATOR | Lars Klöser, Mika Elias Beele, Bodo Kraft |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-13050
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Profiling German Text Simplification with Model-Fingerprints Lars Klöser, Mika Elias Beele, Bodo Kraft Abstract While Large Language Models (LLMs) produce highly nuanced text simplifications, developers currently lack tools for a holistic, efficient, and reproducible diagnosis of their behavior. This paper introduces the Simplification Profiler, a diagnostic toolkit that generates a multidimensional, interpretable fingerprint of simplified texts. Multiple aggregated simplifications of a model result in a model’s fingerprint. This novel evaluation paradigm is particularly vital for languages, where the data scarcity problem is magnified when creating flexible models for diverse target groups rather than a single, fixed simplification style. We propose that measuring a model’s unique behavioral signature is more relevant in this context as an alternative to correlating metrics with human preferences. We operationalize this with a practical meta-evaluation of our fingerprints’ descriptive power, which bypasses the need for large, human-rated datasets. This test measures if a simple linear classifier can reliably identify various model configurations by their created simplifications, confirming that our metrics are sensitive to a model’s specific characteristics. The Profiler can distinguish high-level behavioral variations between prompting strategies and fine-grained changes from prompt engineering, including few-shot examples. Our complete feature set achieves classification F1-scores up to 71.9%, improving upon simple baselines by over 48 percentage points. The Simplification Profiler thus offers developers a granular, actionable analysis to build more effective and truly adaptive text simplification systems. 1 Introduction Automatic Text Simplification (ATS) endeavors to rephrase complex texts into more easily understandable versions to broaden information accessibility. Research in German ATS often focuses on replicating specific, fixed simplification styles inherent in reference texts [ 21 , 29 , 11 ] . However, simplification must be tailored to specific circumstances to enhance the practical relevance of ATS, which includes explicitly considering the overall context. This context encompasses both the target audience 1 1 1 For example, children, individuals with low literacy, or domain newcomers and other situational requirements 2 2 2 such as desired output length, preservation of specific contents, or an appropriate tone . The rise of Large Language Models (LLMs) has made generating such context-aware simplifications more feasible. For example, through straightforward prompt variations. While this flexibility is a significant advancement, it introduces a critical challenge for developers: How do we move beyond simple good/bad scores to understand and steer the nuanced properties of these outputs? - To create solutions that are perfectly adjusted to the needs of different target groups, developers require tools that can characterize and diagnose model behavior. Current approaches to ATS evaluation, however, do not always deliver nuanced insights in the model’s behavior. The evaluation of ATS systems has long relied on reference-based metrics like BLEU [ 19 ] or SARI [ 31 ] . Reference-based metrics, by design, are inflexible. Evaluations with distinct language, domain, or simplification styles require creating novel reference simplifications to compare against. Reference-free approaches aim to avoid this need, notably learnable metrics such as REFeREE [ 9 ] or LENS [ 14 ] . These learnable metrics face further fundamental challenges. Firstly, there is the issue of representational breadth . As motivated, ATS should aim for versatility. Evaluation metrics should be flexible enough to manage this versatility reliably. However, existing human evaluation datasets, if used for training or validation, typically embody only one specific, often implicit, standard of good simplification [ 14 , 2 , 26 , 16 ] . Creating the multitude of high-quality, comprehensive datasets needed to cover this broad application spectrum is likely prohibitively expensive and practically unfeasible. A challenge especially for ATS in non-English languages like German, which serves as the primary linguistic focus for the empirical validation in this paper [ 4 , 17 , 22 ] . Secondly, even when using the available data, there is the problem of learning reliability . We know from other natural language processing tasks like natural language inference (NLI), question answering, or relation extraction that AI models can learn spurious patterns – correlations present in limited data but irrelevant to the actual task [ 6 , 7 , 25 , 12 ] . Given the limited scale of ATS data, learnable evaluation models are particularly susceptible to these pitfalls. These potentially faulty decision patterns create a high risk that these metrics will inadvertently embed biases and fail to generalize, especially when faced with diverse or out-of-domain simplifications. Thirdly, many such learnable metrics operate as black boxes , presenting a significant challenge in explainability. When an evaluation metric does not provide clear insights into how it arrives at a score, it becomes exceedingly difficult to diagnose its failures, understand its sensitivities, or make targeted improvements to the evaluated ATS systems. This lack of transparency harms iterative LLM development, as it provides feedback that can hardly be used for model improvements. Low scores fail to inform developers what to fix, hindering the targeted improvements necessary for creating highly adaptive ATS systems. These issues highlight the need for a paradigm shift from monolithic evaluation to granular analysis. This paper lays the conceptual and methodological foundations for the Simplification Profiler : a diagnostic toolkit designed to characterize ATS outputs across linguistically grounded properties. This approach provides a multi-dimensional, interpretable fingerprint of each simplification, making critical trade-offs (e.g., between simplicity and content preservation) transparent to the developer. These properties can be divided into universal quality criteria and context-dependent parameters: • Universal: Linguistic Correctness (Grammar and Spelling), Factual Accuracy (Adequacy), Coherence. • Context-Dependent: Linguistic Level (Complexity), Content Scope (Compression), Terminology Use, Text Length. The key advantage of our approach is its compositional nature. Instead of building a single, monolithic, task-specific AI model for evaluation, we leverage robust, well-established tools for specific aspects, such as NLI models, grammar checkers, and readability indices. This compositional approach allows our Profiler to stand on the shoulders of significant existing expert knowledge and broad training data, potentially increasing the individual measurements’ generalizability and reliability. These observations lead to our central hypothesis: A multi-dimensional analysis of specific text properties provides an informative and nuanced fingerprint of ATS system performance, enabling targeted development. We test this by demonstrating that our toolkit is sensitive enough to detect the fingerprints of different development choices, using a linear classifier to distinguish outputs generated by various models and prompts. Our main technical contribution is the implementation of the toolkit and all code to reproduce the results of this paper in an open-source GitHub repository 3 3 3 https://github.com/MSLars/German-Simplification-Profiling . 2 Related Work Core Principles of ATS Evaluation. The evaluation of Automatic Text Simplification (ATS) targets qualities like simplicity, fluency, and adequacy [ 24 , 20 ] , with recent LLM-focused work adding criteria like factuality and user engagement [ 30 , 18 ] . While human evaluation is the gold standard [ 1 , 23 ] , its scalability issues necessitate automatic metrics [ 26 ] . Reference-Based and -Free Metrics. Automatic approaches include reference-based metrics like BLEU [ 19 ] , despite its known correlation issues [ 27 ] , the ATS-specific SARI [ 31 ] , and semantic methods like BERTScore [ 32 ] . The dependency on high-quality references spurred reference-free approaches, from traditional readability formulas to linguistically-informed structural metrics like SAMSA [ 28 ] . Modern Model-Based Evaluation. A