DeepRAG: Building a Custom Hindi Embedding Model for Retrieval Augmented Generation from Scratch

Nandakishor M · 2025

Read DeepRAG: Building a Custom Hindi Embedding Model for Retrieval Augmented Generation from Scratch by Nandakishor M (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08213.txt.

Introduction

Models that search text for the right passage to feed a generator work well in English, but Hindi has lacked a search model built for it. Nandakishor M builds one from scratch in 2025, named DeepRAG, starting with more than 2.7 million Hindi text samples. A custom tokenizer was shaped around how Hindi words are built, and the model learned to place related passages near each other.

The new model beat the general multilingual models in use by twenty-three percent in retrieval precision. It also plugs into a common toolkit for building full Hindi question answering systems. The author offers the process as a template for other languages short on tailored tools. Language specific building, the work argues, pays off.

Catalog record

CREATORNandakishor M
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-08213

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DeepRAG: Building a Custom Hindi Embedding Model for Retrieval Augmented Generation from Scratch Nandakishor M Abstract While Large Language Models (LLMs) excel in text generation, their performance in Retrieval Augmented Generation (RAG) systems heavily depends on the quality of text embeddings. For non-English languages like Hindi, the lack of high-quality, dedicated embedding models remains a significant challenge. This paper presents DeepRAG, a comprehensive framework for developing custom Hindi-specific text embeddings from scratch for RAG applications. We detail our end-to-end process, including corpus collection from diverse Hindi sources, specialized tokenizer training with SentencePiece, transformer architecture design with semantic pooling strategies, and model training with contrastive learning techniques. Our evaluation demonstrates that DeepRAG embeddings significantly outperform multilingual alternatives in Hindi semantic similarity tasks, with a 23% improvement in retrieval precision. We further demonstrate the integration of our embeddings with LangChain for building effective Hindi RAG systems. The detailed methodology provides a roadmap for creating domain-specific embeddings for low-resource languages, addressing critical gaps in multilingual NLP infrastructure. I Introduction The advent of large language models (LLMs) has revolutionized natural language processing capabilities, enabling sophisticated text generation across domains. However, these models face challenges with factuality, knowledge limitations, and hallucinations [ 1 ] . Retrieval Augmented Generation (RAG) offers a solution by retrieving relevant documents before generation, enhancing both accuracy and factuality. A critical, often overlooked component in the RAG pipeline is the quality of text embeddings used for retrieval. While numerous embedding models exist for English, high-quality embeddings for other languages remain limited. This gap is particularly pronounced for Hindi, one of the world’s most widely spoken languages. Existing multilingual embedding solutions like multilingual-E5 [ 2 ] and LaBSE [ 3 ] provide some coverage for Hindi, but these models sacrifice language-specific performance for multilingual capabilities. They typically under-represent Hindi semantic nuances, resulting in suboptimal retrieval performance when applied to Hindi RAG systems. In this paper, I present DeepRAG, a complete framework for building custom, high-performance Hindi embeddings specifically designed for RAG applications. Rather than fine-tuning existing models, DeepRAG was built entirely from scratch—from corpus collection and tokenizer training to model architecture design and optimization. This ground-up approach allowed us to incorporate language-specific considerations throughout the development process. The key contributions of this work include: • A comprehensive methodology for building language-specific embedding models from scratch, with each component optimized for Hindi. • A specialized SentencePiece tokenizer trained on a diverse corpus of over 2.7 million Hindi texts, incorporating linguistic-aware subword segmentation. • A custom transformer architecture with enhanced attention mechanisms and pooling strategies specifically designed for Hindi semantic representation. • A multi-stage training process using contrastive learning and synthetic data generation for robust embeddings. • Integration techniques with LangChain for building end-to-end Hindi RAG systems. • Extensive comparative evaluation demonstrating significant improvements over multilingual approaches. Our findings show that a dedicated, language-specific approach yielded substantial gains in embedding quality, with DeepRAG embeddings demonstrating a 23% improvement in retrieval precision over the best multilingual alternatives. I believe our methodology provides valuable insights for developing similar solutions for other low-resource languages where general-purpose multilingual models fall short. II Related Work II-A Multilingual Embedding Models Several established multilingual embedding models currently support Hindi to varying degrees. Multilingual-E5 [ 2 ] covers 100+ languages including Hindi, while LaBSE (Language-agnostic BERT Sentence Embeddings) [ 3 ] was designed specifically for cross-lingual alignment across 109 languages. These models, while impressive in their breadth, face inherent trade-offs. The "curse of multilinguality" [ 4 ] highlights how performance on individual languages degrades as more languages are added to a model. In my personal testing with Hindi documents, I’ve found this degradation to be particularly pronounced for semantic search applications. II-B Language-Specific Embedding Models Language-specific embedding models have demonstrated significant performance improvements over multilingual alternatives. English-specific models like MPNet [ 5 ] and E5 [ 2 ] consistently outperform their multilingual counterparts in English tasks. Similarly, BGE [ 6 ] for Chinese and KUZU [ 7 ] for Korean have shown the value of language-dedicated approaches. For Hindi, however, the landscape remains surprisingly sparse. Work by Kakwani et al. [ 8 ] introduced IndicBERT, but focused primarily on classification rather than embedding quality. Kumar et al. [ 9 ] explored Hindi word embeddings, but comprehensive sentence embedding models remain notably absent. II-C RAG Systems and Embedding Quality Recent work by Lewis et al. [ 10 ] introduced the RAG paradigm, while Gao et al. [ 11 ] emphasized the critical impact of embedding quality on RAG performance. Their findings indicate that retrieval quality is often the primary limiting factor in RAG systems. Through my experiments with Hindi RAG, I’ve consistently found that existing embedding models fail to capture the nuanced semantic relationships in Hindi documents. This observation aligns with findings from Ruder et al. [ 12 ] , who noted that language-specific pretraining significantly improves downstream task performance. III Corpus Collection and Analysis The foundation of any high-quality embedding model is a diverse and representative text corpus. For Hindi, this presented unique challenges due to the relative scarcity of digitized content compared to languages like English. III-A Data Sources and Collection I prioritized diversity of both content and style to ensure robust embeddings. Our final corpus comprised text from the following sources: • IITB Parallel Corpus: 1.2 million Hindi sentences from various domains • Samanantar: 750,000 samples of general Hindi text • Oscar Hindi: 450,000 sentences from web crawls • CC-100 Hindi: 300,000 sentences of web content • Hindi Wikipedia: 150,000 articles covering encyclopedic knowledge • Hindi news articles: 100,000 news pieces covering current events • XNLI Hindi: 50,000 premise-hypothesis pairs useful for semantic reasoning • IndicGLUE: 30,000 samples from diverse tasks • Hindi literature: 5,000 passages from classic and modern Hindi literature Several of these datasets required custom extraction scripts to isolate the Hindi portions and remove non-Hindi text contamination. After collection, the raw corpus totaled approximately 3.1 million text samples. III-B Corpus Cleaning and Preprocessing Raw datasets contained significant noise, including code-mixing with English, HTML artifacts, and irregular Unicode representations. Our cleaning pipeline included: 1. Unicode normalization (NFC form) 2. Removal of non-Hindi text segments via Unicode range filtering 3. Deduplication through exact and near-duplicate detection 4. Length filtering (removing very short or excessively long texts) 5. Specialized Hindi normalization using IndicNLP’s normalizer 6. Removal of frequent patterns indicating low-quality content The cleaning process reduced our corpus to approximately 2.7 million high-quality Hindi text samples. An example of our cleaning function is shown below: Algorithm 1 Hindi Text Cleaning 1: Input: Raw Hindi text T 2: Output: Cleaned text T^{\prime} 3: T\leftarrow\text{strip}(T) 4: T\leftarrow\text{replace\_urls\_with\_placeholder}(T) 5: T\leftarrow\text{unicode\_normalize\_NFC}(T) 6: T\leftarrow\text{hindi\_normalizer.normalize}(T) 7: T\leftarrow\text{remove\_repeating\_chars}(T) 8: T\leftarrow\text{replace\_numbers\_with\_placeholder}(T) 9: T\leftarrow\text{normalize\_whitespace}(T) 10: return T^{\prime} III-C Corpus Analysis To inform our tokenizer and model design decisions, I conducted extensive analysis on the cleaned corpus, examining: • Character and word frequency distributions • Sentence length statistics • Common word collocations • Subword unit effectiveness through various segmentation approaches • Topic diversity via clustering analysis This analysis revealed several insights specific to Hindi that informed our approach: 1. Hindi’s agglutinative features create long compound words that benefit from subword tokenization 2. The frequent absence of spaces between certain Hindi words requires careful tokenization 3. Common transliterations from English technical

Read the complete text