Iterative Definition Refinement for Zero-Shot Classification via LLM-Based Semantic Prototype Optimization

Naeem Rehmat, Muhammad Saad Saeed, Ijaz Ul Haq, Khalid Malik · 2026

Read Iterative Definition Refinement for Zero-Shot Classification via LLM-Based Semantic Prototype Optimization by Naeem Rehmat, Muhammad Saad Saeed, Ijaz Ul Haq, Khalid Malik (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27335.txt.

Introduction

Rehmat and three colleagues propose a training-free iterative definition refinement framework for zero-shot web content sorting. Web filtering needs accurate sorting to block threats, prevent theft, and meet rules. But the modern web changes fast. Vector-based zero-shot methods map content and category descriptions into a shared meaning space, assigning labels without labeled training data. Yet they stay highly sensitive to definition quality. Poorly specified or unclear definitions create meaning overlap and wrong labels, which step-by-step definition tuning aims to reduce.

Catalog record

CREATORNaeem Rehmat, Muhammad Saad Saeed, Ijaz Ul Haq, Khalid Malik
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35998
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.27335

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Iterative Definition Refinement for Zero-Shot Classification via LLM-Based Semantic Prototype Optimization Naeem Rehmat, Muhammad Saad Saeed, Ijaz Ul Haq, Khalid Malik Abstract Web filtering systems rely on accurate web content classification to block cyber threats, prevent data exfiltration, and ensure compliance. However, classification is increasingly difficult due to the dynamic and rapidly evolving nature of the modern web. Embedding-based zero-shot approaches map content and category descriptions into a shared semantic space, enabling label assignment without labeled training data, but remain highly sensitive to definition quality. Poorly specified or ambiguous definitions create semantic overlap in the embedding space, leading to systematic misclassification. In this paper, we propose a training-free, adaptive iterative definition refinement framework that improves zero-shot web content classification by progressively optimizing category definitions rather than updating model parameters. Using LLMs as feedback-driven definition optimizers, we investigate three refinement strategies namely example-guided, confusion-aware, and history-aware, each refining class descriptions using structured signals from misclassified instances. Furthermore, we introduce a human-labeled benchmark of 10 URL categories with 1,000 samples per class and evaluate across 13 state-of-the-art embedding foundation models. Results demonstrate that iterative definition refinement consistently improves classification performance across diverse architectures, establishing definition quality as a critical and underexplored factor in embedding-based systems. The dataset is available here . 1 Introduction Figure 1 : Comparison of URLs classification paradigms. Unlike (a) supervised fine-tuning and (b) static zero-shot methods, our proposed approach (c) iteratively refines category definitions using LLM feedback, without updating model parameters. The classification of web contents into semantic categories is a critical component in a wide range of applications, including content filtering, malicious URL detection, network traffic analysis, and web indexing Le et al. 2018 ; Saxe and Berlin 2017 . Traditional supervised approaches to URL classification require large annotated datasets and are brittle to the rapidly evolving nature of web content Maneriker et al. 2021 . Embedding-based zero-shot classification offers an alternative by representing both URLs and category descriptions in a shared semantic space, enabling label inference through similarity computation without task-specific training Reimers and Gurevych 2019 ; Yin et al. 2019 . The effectiveness of this paradigm, however, is fundamentally tied to the quality of the category definitions. In embedding space, each category definition acts as a semantic prototype where the class boundary is implicitly determined by the position of this prototype relative to document embeddings Wang et al. 2022a ; Muennighoff et al. 2023 . When definitions are vague, overly general, or semantically similar across classes, the resulting prototype placement causes systematic misclassifications even when the underlying embedding model possesses strong representational capacity. As illustrated in Figure 1 , this limitation is shared by both supervised fine-tuning approaches Yin et al. 2019 which update model parameters at significant cost and static zero-shot methods Su et al. 2023 . This observation motivates a key question: can classification performance be improved by refining the category definitions themselves, rather than by retraining or fine-tuning the embedding model? We answer this question affirmatively by proposing an iterative definition refinement framework in which LLMs are used to progressively improve class definitions based on feedback from classification results. The proposed framework introduces three complementary refinement strategies. Example-guided refinement updates definitions using representative samples to ground them in real data. Confusion-aware refinement targets the most frequently misclassified class pairs to improve discriminability at decision boundaries.History-aware refinement incorporates the trajectory of previous definitions and performance scores to guide further improvements, enabling a form of iterative knowledge evolution. Our main contributions are as follows: (1) We introduce the first framework that applies iterative semantic prototype refinement to web content classification, achieving competitive performance without model training or fine-tuning. Unlike prior supervised Maneriker et al. 2021 ; Le et al. 2018 and static zero-shot approaches Yin et al. 2019 ; Su et al. 2023 , our framework dynamically adapts category definitions from classification feedback, leaving all embedding models frozen. (2) We propose example-guided , confusion-aware , and history-aware refinement each targeting a distinct failure mode (data grounding, inter-class confusion, and definition trajectory). specifically we combined Monte Carlo acceptance rule in history-aware strategy, enabling principled exploration of the definition space without getting trapped in local optima. (3) We systematically benchmark all strategy-model combinations (13 state-of-the-art embedding foundation models and 4 LLM), identifying which pairings yield the most discriminative definitions and providing deployment-ready guidance for practitioners. (4) We release a B2MWT-10C (Benchmark Multilingual Multi-Web Text) dataset of 10,000 URLs across 10 semantically overlapping categories. 2 Related Work Early web content classification methods relied on hand-crafted lexical and structural features extracted from URL strings, including token n-grams, character-frequency distributions, and domain-level statistics. Deep learning shifted the paradigm toward end-to-end representation learning. Convolutional neural networks (CNNs) applied at the character level demonstrated strong performance for malicious URL detection, learning local structural patterns indicative of phishing and spam Le et al. 2018 ; Saxe and Berlin 2017 . Recurrent architectures, particularly LSTMs, subsequently captured sequential dependencies across URL path and query components Huang et al. 2020 . Transformer-based models fine-tuned on URL classification tasks achieved state-of-the-art results on standard benchmarks Devlin et al. 2019 , benefiting from subword tokenization and contextual pre-training. More recently, graph neural networks have incorporated link structure and co-occurrence patterns, framing URL classification as node classification in a web graph Liu et al. 2022 . Despite their strong supervised performance, these approaches require substantial labeled data and retraining whenever category definitions evolve, and offer no mechanism for zero-shot generalization to novel categories. Zero-shot classification methods address the labeled-data bottleneck. Early approaches based on natural language inference (NLI) frame classification as textual entailment, scoring candidate labels via a premise–hypothesis structure Yin et al. 2019 . Embedding-based classification represents documents and category definitions independently in a shared vector space and assigns labels via cosine similarity Reimers and Gurevych 2019 ; Pratt et al. 2023 . The quality of embeddings from modern sentence transformers and instruction-tuned models Wang et al. 2022a ; Su et al. 2023 ; Xiao et al. 2024 has made this paradigm increasingly competitive, with comprehensive benchmarking provided by MTEB Muennighoff et al. 2023 . LLMs have also been applied directly to classification through prompt engineering Brown et al. 2020 ; Touvron et al. 2023 , where the model assigns a label given the input and a description of candidate categories. While effective, this requires a full LLM forward pass per sample, which is computationally prohibitive at scale. Automatic prompt optimization methods Zhou et al. 2022 ; Pryzant et al. 2023 are conceptually related. These methods improve LLM performance by modifying inputs rather than weights. However, these methods target generative tasks and optimize prompts for auto regressive models. Our framework instead optimizes semantic prototype definitions for embedding-based classifiers, involving different objectives, feedback signals, and optimization dynamics. To our knowledge, no prior work has systematically studied iterative definition refinement for zero-shot web content classification across diverse embedding architectures. Figure 2 : Workflow diagram of proposed method. 3 Methodology 3.1 Problem Formulation Let \mathcal{X}=\{(x_{i},y_{i})\}_{i=1}^{n} denote a labeled dataset where x_{i} represents the textual content extracted from the HTML of a webpage associated with a URL and y_{i}\in\mathcal{C}=\{c_{1},\ldots,c_{k}\} denotes the ground truth category. For each class c_{j} , we maintain a textual definition d_{j} ; a short natural-language description

Read the complete text