UniDex: Rethinking Search Inverted Indexing with Unified Semantic Modeling Conference: Proceedings of the xxx; Beijing; China the Preprint Version of UniDex, Beijing, China DOI: 10.1145/xxxxx ISBN: xxxxx/2025/09 CCS: Information systems Novelty in information retrieval

Zan Li † , Jiahui Chen † , Yuan Chai ∗† , Xiaoze Jiang ∗† , Xiaohua Qi † , Zhiheng Qin, Runbin Zhou, Shun Zuo, Guangchao Hao, Kefeng Wang, Jingshan Lv, Yupeng Huang, Xiao Liang, Han Li · 2025

Read UniDex: Rethinking Search Inverted Indexing with Unified Semantic Modeling Conference: Proceedings of the xxx; Beijing; China the Preprint Version of UniDex, Beijing, China DOI: 10.1145/xxxxx ISBN: xxxxx/2025/09 CCS: Information systems Novelty in information retrieval by Zan Li † , Jiahui Chen † , Yuan Chai ∗† , Xiaoze Jiang ∗† , Xiaohua Qi † , Zhiheng Qin, Runbin Zhou, Shun Zuo, Guangchao Hao, Kefeng Wang, Jingshan Lv, Yupeng Huang, Xiao Liang, Han Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24632.txt.

Introduction

Zan Li and Jiahui Chen share first credit on this paper from two thousand twenty-five, with a long author list. It presents UniDex, a model-based remake of the inverted index at the heart of search.

Classic indexes match raw terms, which misses sense and ages poorly. UniDex maps both queries and documents to semantic IDs through a part called UniTouch. A second part, UniRank, then ranks by semantic match. The pair swaps hand-built rules for one learned frame, which cuts upkeep. Tests on plant-scale data and live traffic show clear gains. The system now serves the short-video search of Kuaishou at vast scale.

Catalog record

CREATORZan Li † , Jiahui Chen † , Yuan Chai ∗† , Xiaoze Jiang ∗† , Xiaohua Qi † , Zhiheng Qin, Runbin Zhou, Shun Zuo, Guangchao Hao, Kefeng Wang, Jingshan Lv, Yupeng Huang, Xiao Liang, Han Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-24632

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UniDex: Rethinking Search Inverted Indexing with Unified Semantic Modeling Conference: Proceedings of the xxx; Beijing; China the Preprint Version of UniDex, Beijing, China DOI: 10.1145/xxxxx ISBN: xxxxx/2025/09 CCS: Information systems Novelty in information retrieval Zan Li † , Jiahui Chen † , Yuan Chai ∗† , Xiaoze Jiang ∗† , Xiaohua Qi † , Zhiheng Qin, Runbin Zhou, Shun Zuo, Guangchao Hao, Kefeng Wang, Jingshan Lv, Yupeng Huang, Xiao Liang, Han Li Abstract . Inverted indexing has traditionally been a cornerstone of modern search systems, leveraging exact term matches to determine relevance between queries and documents. However, this term-based approach often emphasizes surface-level token overlap, limiting the system’s generalization capabilities and retrieval effectiveness. To address these challenges, we propose UniDex, a novel model-based method that employs unified semantic modeling to revolutionize inverted indexing. UniDex replaces complex manual designs with a streamlined architecture, enhancing semantic generalization while reducing maintenance overhead. Our approach involves two key components: UniTouch, which maps queries and documents into semantic IDs for improved retrieval, and UniRank, which employs semantic matching to rank results effectively. Through large-scale industrial datasets and real-world online traffic assessments, we demonstrate that UniDex significantly improves retrieval capabilities, marking a paradigm shift from term-based to model-based indexing. Our deployment within Kuaishou’s short-video search systems further validates UniDex’s practical effectiveness, serving hundreds of millions of active users efficiently. 1. Introduction Inverted indexing has long served as a foundational component of modern search systems due to its efficiency in large-scale information retrieval ( Bai et al. 2020 ; Yang et al. 2017 ; Formal et al. 2021a ) . The conventional paradigm tokenizes documents into discrete terms and builds an index mapping each term to its associated documents ( Formal et al. 2021b ; Formal et al. 2021a ) . Relevance between queries and documents is then primarily determined through exact or approximate term matches ( Roelleke and Wang 2008 ; Pang et al. 2017 ) . While effective in terms of efficiency, this approach inherently emphasizes surface-level lexical overlap, thereby limiting generalization and hindering retrieval effectiveness ( Dai and Callan 2019 ) . Breaking away from this term-centric paradigm to enhance semantic retrieval remains an open challenge. Figure 1. Comparison of Conventional Inverted Indexing and UniDex. UniDex substitutes the various manually designed term-based Touch and Rank components with the unified semantic modeling approaches, UniTouch and UniRank. As illustrated in the blue blocks of Figure 1 , industrial-scale inverted indexing systems are typically organized into two stages: the Touch stage and the Rank stage. The Touch module performs term-level retrieval to construct a candidate set, while the Rank module scores these candidates with respect to the query. Over the years, a variety of manually engineered heuristics have been introduced at both stages to improve recall and ranking quality ( Yang et al. 2017 ; Bai et al. 2020 ; Bruch et al. 2024 ) . For example, synonym expansion, stopword handling, and term normalization are employed during Touch to enlarge the candidate pool, while heuristic strategies such as BM25 ( Yang et al. 2017 ) or TF–IDF ( Roelleke and Wang 2008 ) , often combined with handcrafted term-importance features, are used in Rank to refine relevance estimation ( Yang et al. 2017 ; Roelleke and Wang 2008 ; Nogueira et al. 2019b ) . Despite their widespread adoption ( Dai and Callan 2019 ) , these heuristics are costly to maintain, brittle across domains, and fundamentally constrained by the limitations of term-level modeling. With the rise of deep semantic models ( Khattab and Zaharia 2020 ; Jiang et al. 2022 ; Wang et al. 2025 ) , many recent studies attempt to integrate neural representations into inverted indexing ( Formal et al. 2021a ; Formal et al. 2021b ; Lassance et al. 2024 ) . However, most of these works remain tied to the term-based paradigm: semantic models are often used to generate richer sets of query or document terms, but the retrieval process itself still relies on lexical overlap. Consequently, such methods ( Bai et al. 2020 ; Lassance et al. 2024 ) inherit the weaknesses of traditional indexing, including reliance on handcrafted matching strategies and limited semantic generalization. Motivated by these limitations, we propose UniDex , a unified model-based framework that rethinks inverted indexing from a semantic perspective. As shown in the green blocks of Figure 1 , UniDex replaces manually designed term-level components in both Touch and Rank with two semantic modeling modules: UniTouch and UniRank . Specifically, UniTouch maps queries and documents into discrete semantic IDs (SIDs) through a dual-tower encoder with an integrated quantization module. To better capture the inherent ambiguity of queries and the compositional semantics of documents, UniTouch represents each input with multiple learnable tokens, each token corresponding to a potential semantic aspect. For efficient retrieval, we design a Max–Max matching strategy that aligns query tokens with document tokens in a manner consistent with the inverted indexing lookup logic. This design ensures that a document can be retrieved if it matches at least one semantic aspect of the query, while preserving the scalability of inverted indexing. As a result, UniTouch can recall documents whose semantics are close to the query even when they do not share overlapping terms, thus significantly improving generalization over traditional term-based inverted indexes. On top of this retrieval stage, UniRank employs another semantic model with token-level interactions to precisely rank the retrieved candidates, going beyond conventional heuristic term-matching approaches. Together, UniTouch and UniRank form UniDex: a unified retrieval–ranking pipeline that combines the scalability of inverted indexing with the expressive power of semantic modeling. Unlike traditional pipelines, UniDex eliminates dependence on handcrafted rules, generalizes more effectively across diverse queries and domains, and substantially reduces engineering overhead, making it a practical solution for real-world search platforms. We comprehensively evaluate UniDex on large-scale industrial datasets as well as real-world online traffic. The main contributions of this work are summarized as follows: (1) We present the first paradigm shift of inverted indexing from a term-based to a model-based framework. This reformulation opens new directions for the evolution of retrieval systems and carries significant implications for both research and industry. (2) We propose UniDex , a unified semantic modeling framework that replaces manually engineered term-matching and multi-path relevance computations with semantic ID–based indexing and model-driven ranking. This design substantially reduces maintenance overhead while providing stronger semantic generalization. (3) To the best of our knowledge, we achieve the first successful large-scale deployment of a model-based inverted indexing system in industry. UniDex has been integrated into Kuaishou’s short-video search platform, where extensive online A/B testing confirms its effectiveness and efficiency. It currently serves hundreds of millions of active users, demonstrating both the practicality and impact of our approach. Figure 2. Overview of our framework. (a) The UniTouch model architecture, consisting of a Query Encoder, a Document Encoder, and Quantization Modules. (a1) The internal design of the Quantization Module. (a2) The token matching strategy in UniTouch, defined as the maximum entry of the token-level matching matrix. (b) The end-to-end retrieval pipeline of UniDex: a query is first discretized into semantic IDs via UniTouch, relevant documents are retrieved from the semantic inverted indexing, and the final candidate set is ranked by UniRank. 2.

Read the complete text