Autoencoder-Based Framework to Capture Vocabulary Quality in NLP

Vu Minh Hoang Dang, Rakesh M. Verma · 2025

Read Autoencoder-Based Framework to Capture Vocabulary Quality in NLP by Vu Minh Hoang Dang, Rakesh M. Verma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-00209.txt.

Introduction

Vu Minh Hoang Dang and Rakesh Verma gauge vocabulary quality in language data sets with an autoencoder based frame. Word richness shapes model skill, but classic gauges miss context, meaning depth, and build complexity. Their work came out in twenty twenty-five.

Network capacity stands in for richness, diversity, and complexity, giving a live read of how vocabulary size, sentence build, and context depth interact. Tests ran on a fraud text set spanning many domains, and on a classic book set across tongues, genres, and eras. Results back the method's toughness and flex. The authors offer guidance for data curation and model design.

Catalog record

CREATORVu Minh Hoang Dang, Rakesh M. Verma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-00209

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Autoencoder-Based Framework to Capture Vocabulary Quality in NLP Vu Minh Hoang Dang, Rakesh M. Verma Abstract Linguistic richness is essential for advancing natural language processing (NLP), as dataset characteristics often directly influence model performance. However, traditional metrics such as Type-Token Ratio (TTR), Vocabulary Diversity (VOCD), and Measure of Lexical Text Diversity (MTLD) do not adequately capture contextual relationships, semantic richness, and structural complexity. In this paper, we introduce an autoencoder-based framework that uses neural network capacity as a proxy for vocabulary richness, diversity, and complexity, enabling a dynamic assessment of the interplay between vocabulary size, sentence structure, and contextual depth. We validate our approach on two distinct datasets: the DIFrauD dataset, which spans multiple domains of deceptive and fraudulent text, and the Project Gutenberg dataset, representing diverse languages, genres, and historical periods. Experimental results highlight the robustness and adaptability of our method, offering practical guidance for dataset curation and NLP model design. By enhancing traditional vocabulary evaluation, our work fosters the development of more context-aware, linguistically adaptive NLP systems. 1 Introduction The growing prevalence of natural language processing (NLP) applications has heightened the need for effective measures to evaluate the quality of linguistic datasets. Linguists, however, have been interested in vocabulary measurement for over 50 years, exploring ways to capture lexical richness and diversity across different contexts. Their lexical metrics such as TTR (type-token ratio) [ 22 ] , MTLD (Measure of Lexical Text Diversity) [ 15 ] and VOCD (Vocabulary Diversity) [ 13 ] have been widely adopted, but they exhibit notable limitations. These popular measures often fail to account for critical factors such as the contextual relationship between tokens, the impact of duplication, and the variation in sentence length distributions across datasets [ 14 , 15 ] . Consequently, their applicability is constrained, particularly when addressing datasets with diverse linguistic and structural characteristics. This problem is particularly challenging due to the inherent complexity of language. Datasets vary in vocabulary size, sentence structure, and contextual depth. While some approaches normalize vocabulary measures or incorporate sentence-length metrics [ 16 ] , they often rely on oversimplified assumptions. These metrics fail to address key dimensions of dataset complexity, leading to potentially flawed evaluations in NLP tasks. Our research addresses these gaps by focusing on vocabulary quality assessment through an autoencoder-based methodology. We demonstrate that reconstruction complexity can effectively measure vocabulary quality, providing insights into the underlying dataset characteristics. Through extensive validation, our framework establishes a robust approach for evaluating vocabulary quality in NLP datasets. Our contributions include: 1. A systematic critique of existing metrics, highlighting their limitations in real-world scenarios. 2. An autoencoder-based framework that leverages neural network capacity as a proxy for vocabulary diversity and complexity. 3. Evaluation of the framework using two datasets: DIFrauD [ 2 ] for deceptive text and Project Gutenberg 1 1 1 https://huggingface.co/datasets/manu/project_gutenberg for diverse domains and languages. 2 Related Work 2.1 Lexical Metrics Lexical metrics like TTR, VOCD, and MTLD assess lexical diversity in linguistic datasets [ 22 , 13 , 15 ] . Each has strengths and limitations. TTR, simple and efficient, is unreliable for longer texts [ 20 ] . VOCD uses a probabilistic model but is complex and hard to interpret [ 14 ] . MTLD is robust to text length but introduces inconsistencies and lacks semantic considerations [ 15 ] . Newer approaches like HD-D and MTLD+ address some shortcomings but still have limitations [ 10 , 7 ] . These metrics overlook sentence-level variations, contextual relationships, and duplication impact. They treat all tokens equally, disregarding semantic and syntactic roles, and are sensitive to dataset size and domain-specific characteristics [ 21 , 9 ] . This highlights the need for more robust, context-aware metrics in modern NLP applications. 2.2 Vocabulary Quality in NLP Vocabulary quality is critical for dataset evaluation, directly influencing language model performance. Existing metrics like token counts or frequency distributions [ 19 ] often neglect semantic richness, syntactic variability, and cross-domain generalizability. Normalizing vocabulary size by dataset size as in Data Quality Index 1 (DQI 1) [ 16 ] fails to account for sentence length differences, while duplication inflates scores without adding diversity. Research highlights biases in quality metrics due to oversampling or poor curation [ 6 ] . A comprehensive taxonomy of NLP metrics, including vocabulary quality, diversity, and inter-class differences, is provided in [ 5 ] . Cross-genre and multilingual datasets pose challenges due to distinct linguistic features [ 12 ] and language-specific variations [ 4 ] . Recent advancements address these issues by evaluating contextual and structural properties. Tools assess domain diversity and lexical richness [ 8 ] , while token-level attention metrics measure vocabulary informativeness [ 23 ] . Metrics like overlap percentage and token recall [ 3 ] align vocabulary evaluation with downstream requirements, aiding domain-specific applications. Despite these advancements, comprehensive vocabulary quality evaluation remains a challenge. Many metrics focus on surface-level characteristics, neglecting contextual relationships, semantic richness, and structural complexity. Our work proposes a novel autoencoder-based framework that evaluates vocabulary quality across multiple linguistic dimensions, providing insights for dataset development and analysis. 3 Critique of DQI 1 and Other Vocabulary Metrics Figure 1: This plot shows the change in the DQI 1 metric (Y-axis), which measures vocabulary quality, across different numbers of unique rows in a fixed sample size of 1000 rows (X-axis). The DQI 1 value decreases as the number of unique rows increases. Vocabulary quality is a critical aspect of evaluating NLP datasets. Several metrics have been proposed to measure vocabulary richness and diversity, including DQI 1, VOCD, and MTLD. While these metrics offer valuable insights, they exhibit notable limitations. This section provides a detailed critique of DQI 1 and compares it with the shortcomings of other metrics, such as TTR, VOCD, and MTLD. 3.1 Critique of DQI 1 The authors of [ 16 ] propose DQI 1, which aims to measure dataset quality through a combination of two components: vocabulary and sentence length. However, the formulation of DQI 1 exhibits several critical issues, which limit its applicability and interpretability. Let X be the dataset, v be the vocabulary, s be the sentence length, S be the set of all sentences in the dataset, a and b be the lower and upper thresholds of sentence length, and size be the total number of samples, then the DQI 1 formula is as follows: Vocabulary =\frac{v(X)}{size(X)}+\sigma(s(X))\cdot\frac{\sum_{S}(sgn((s-a)(b-s))}{size(S)} Firstly, the name “Vocabulary” in the DQI 1 metric is misleading. While the metric claims to focus solely on vocabulary, it incorporates sentence length into the formula, causing ambiguity. A more precise name, such as “Vocabulary and Sentence-Length Composite,” would better reflect its scope. The vocabulary component oversimplifies richness by ignoring the saturation of unique tokens as datasets grow. Its normalization by dataset size assumes uniformity across domains, which is invalid for domains like social media with inherently shorter texts. Similarly, the sentence length component overlooks the impact of duplication, as normalizing by unique sentences distorts diversity representation. Figure 1 demonstrates this flaw, showing that duplication paradoxically increases the DQI 1 score, contrary to expectations. While removing the sentence length component might seem like a solution, this would not address the fundamental issues with vocabulary normalization and would leave us with an incomplete measure of

Read the complete text