Granuscore: A Reference-Free Measure of Granularity for Text Analysis and Question Answering

Lukas Ellinger, Alexander Fichtl, Miriam Anschütz, Georg Groh · 2026

Read Granuscore: A Reference-Free Measure of Granularity for Text Analysis and Question Answering by Lukas Ellinger, Alexander Fichtl, Miriam Anschütz, Georg Groh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26620.txt.

Introduction

Lukas Ellinger and three others introduce Granuscore, a yardstick for how fine or coarse wording is, needing no reference text. Older gauges mostly count surface detail or sentence specificity. Granuscore instead reads structure in a layered embedding space, recovering known orderings on a test set and tracking shifts across discourse settings. It also explains wiggles in specificity beyond sentence length. On four question-answering sets, it exposes steady gaps between questions, gold answers, and model replies.

Catalog record

CREATORLukas Ellinger, Alexander Fichtl, Miriam Anschütz, Georg Groh
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60484
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26620

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Granuscore: A Reference-Free Measure of Granularity for Text Analysis and Question Answering Lukas Ellinger, Alexander Fichtl, Miriam Anschütz, Georg Groh Abstract Natural language conveys information at varying levels of granularity, from fine-grained references to broad descriptions. While granularity is fundamental to human communication, existing measures mostly capture surface detail or sentence specificity. We introduce Granuscore , a reference-free measure of granularity that leverages structural properties of a hierarchical embedding space. Granuscore reliably recovers hierarchical orderings on the Granola-EQ dataset and captures expected differences in granularity across discourse contexts. Across domains, we further show that Granuscore explains non-linear variation in sentence specificity beyond sentence length. Finally, we apply Granuscore to four question-answering benchmarks and analyze how granularity differs for questions, gold answers, and model outputs across response outcomes. The analysis reveals consistent differences in model behavior and provides a principled lens for characterizing the difficulty of QA datasets. Together, the results position Granuscore as a scalable, broadly applicable tool for analyzing granularity in text. 1 Introduction Natural language varies not only in what information is conveyed, but also in how coarsely or finely that information is expressed. Consider the sentences in Figure 1 . A speaker may refer to a person as Tony Hawk , a skateboarder , or a sportsman , and may locate an event in San Diego , California , or the United States . These alternatives preserve the underlying fact while referring to it at different levels. We refer to this dimension as granularity : the level of abstraction at which entities or events are represented in language ( 25 ; 31 ; 14 ) . Granularity is not incidental: speakers adapt the level of abstraction of their descriptions depending on conversational context and task requirements ( 25 ; 14 ) . When uncertain, speakers often prefer coarser fact descriptions that remain informative without overcommitting. Conversely, when common ground is established, more fine-grained references become appropriate ( 43 ) . Granularity should therefore be understood as a deliberate strategy that balances reliability and audience expectations. Figure 1: Sentences with referential units varying in granularity. Units that differ across sentences are underlined. Replacing fine-grained terms with coarser alternatives increases sentence granularity: lower Granuscores indicate finer expressions. Prior work suggests that linguistic granularity affects how information is perceived and used. In dialogue systems, too fine-grained or coarse responses can reduce user satisfaction ( 1 ; 36 ) . Similarly, in simplified language settings, controlling granularity is important for accessibility and comprehension as it reduces the cognitive load ( 26 ; 3 ) . However, studying such effects systematically is difficult because existing approaches do not provide a scalable, reference-free measure of granularity at the sentence level. Our contributions are as follows: • We introduce Granuscore , a reference-free measure of granularity that exploits structural properties of a hierarchical embedding space. • We validate Granuscore both empirically and conceptually. It reliably recovers human-annotated orderings on Granola-EQ ( 43 ) and captures expected granularity differences across discourse contexts. • We show that across domains Granuscore explains non-linear variation in sentence specificity beyond sentence length. • We demonstrate the practical relevance of Granuscore for question answering. Evaluating six language models on four QA benchmarks, we identify consistent differences in granularity between questions, gold answers, and model outputs across response outcomes. These patterns provide a principled lens for characterizing QA dataset difficulty and analyzing model behavior. • We release Granuscore as a pip package to ensure reproducibility and enable its usage for further research or production. 2 Background and Related Work Granularity 25 describe granularity in natural language as shifts between coarse and fine descriptions, where higher-level representations abstract from more detailed components. Related perspectives appear in cognitive science, where concepts are organized at different levels of abstraction within taxonomies ( 31 ) . Further, foundational work by 14 argues that intelligent reasoning requires representing the world at multiple levels of granularity and switching between them as needed, allowing complex phenomena to be modeled through simpler abstractions. A related property is term specificity , which refers to identifying index terms distinguishing one class of documents from others. In particular, 19 describe hierarchical specificity as a term’s position within a generic–specific hierarchy, where narrower terms correspond to more specific concepts, matching the notion of granularity. We capture these ideas using structural properties of a hierarchical embedding space. Unlike approaches relying on manually constructed hierarchies, this enables estimating granularity without being restricted to predefined vocabularies. Sentence Specificity Sentence specificity refers to the extent to which a sentence conveys concrete information and supports consistent interpretation across readers ( 21 ; 20 ) . Prior work has shown its relevance for reading comprehension ( 9 ) and establishing common ground in dialogue ( 10 ) . Although finer-grained references often increase sentence specificity, granularity and sentence specificity capture different properties. Sentence specificity reflects the amount of descriptive information conveyed by a sentence, whereas granularity describes the level at which referential expressions occur within a semantic hierarchy. A sentence can therefore become more specific by adding descriptive details without changing the granularity of its referents. For example, “The skateboarder won the competition” becomes more specific in “The skateboarder won the competition and set a new record.” . The referents remain at the same granularity level, but the sentence conveys more information. Granularity Evaluation While granularity has been implicitly discussed in work on specificity, informativeness, and semantic hierarchies ( 36 ; 1 ; 20 ; 21 ) , existing automatic evaluations typically rely on taxonomy depth (e.g., WordNet hypernym levels ( 24 ) or hierarchical relations in knowledge graphs such as Wikidata ( 37 ; 16 ) ). However, these approaches require entities to exist in the underlying taxonomy and therefore provide limited coverage for free-form text. In contrast, embedding-based approaches can operate directly on arbitrary text. 16 propose an automatic benchmark for measuring specificity using transitive relations derived from Wikidata. However, the induced orderings can yield unintuitive comparisons, for example, ranking Mexico as more granular than Colombia , or historian as more granular than writer . We therefore acknowledge this dataset but refrain from using it in our experiments. 43 introduce Granola-EQ , a question answering dataset with explicitly controlled answer granularity levels. They show that standard decoding methods tend to produce overly granular and often incorrect answers. We build on this dataset to train Granuscore and extend their analysis by applying granularity estimation to a broader set of QA datasets, studying how granularity relates to model outputs, correctness, and dataset difficulty. Training Signals for Informativeness and Interestingness The informativeness of model responses plays a central role in user engagement and response quality ( 1 ; 36 ) . While early work relies on human annotation to supervise informativeness ( 1 ; 36 ) , more recent approaches use LLM-based judges to obtain relative preference signals by comparing response pairs ( 41 ) . Relatedly, 28 introduce interestingness as a training signal and assign scores using an LLM judge. In contrast to these approaches, which depend on human supervision or pairwise or model-based judgments, Granuscore provides a reference-free, scalable signal that measures granularity on an absolute and interpretable scale. 3 Granuscore Figure 2: Granuscore pipeline : extraction of

Read the complete text