Advancing Vietnamese Information Retrieval with Learning Objective and Benchmark
Phu-Vinh Nguyen, Minh-Nam Tran, Long Nguyen, Dien Dinh · 2025
Read Advancing Vietnamese Information Retrieval with Learning Objective and Benchmark by Phu-Vinh Nguyen, Minh-Nam Tran, Long Nguyen, Dien Dinh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07470.txt.
Introduction
Vietnamese search tools lack a home-grown yardstick, and a 2025 paper by Phu-Vinh Nguyen and co-authors supplies one. Search sits inside many modern AI systems, yet no standard benchmark existed to grade Vietnamese embedding models, which slowed the whole local field.
The team built a benchmark centered on finding and re-ranking documents. They also propose a training objective that reshapes a standard contrastive loss to suit retrieval better, and they trained their own Vietnamese embedding model with it. Tests then vary the temperature setting in both the old and new losses to show its effect. The paper gives Vietnamese language research both a measuring stick and a stronger baseline model.
Catalog record
| CREATOR | Phu-Vinh Nguyen, Minh-Nam Tran, Long Nguyen, Dien Dinh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-07470
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Advancing Vietnamese Information Retrieval with Learning Objective and Benchmark Phu-Vinh Nguyen, Minh-Nam Tran, Long Nguyen, Dien Dinh Abstract With the rapid development of natural language processing, many language models have been invented for multiple tasks. One important task is information retrieval (IR), which requires models to retrieve relevant documents. Despite its importance in many real-life applications, especially in retrieval augmented generation (RAG) systems, this task lacks Vietnamese benchmarks. This situation causes difficulty in assessing and comparing many existing Vietnamese embedding language models on the task and slows down the advancement of Vietnamese natural language processing (NLP) research. In this work, we aim to provide the Vietnamese research community with a new benchmark for information retrieval, which mainly focuses on retrieval and reranking tasks. Furthermore, we also present a new objective function based on the InfoNCE loss function, which is used to train our Vietnamese embedding model. Our function aims to be better than the origin in information retrieval tasks. Finally, we analyze the effect of temperature, a hyper-parameter in both objective functions, on the performance of text embedding models. 1 Introduction With the born of transformer architecture Vaswani et al. (2017) since 2017, many language models such as BERT Devlin et al. (2019) , GPT Brown et al. (2020) , and T5 Raffel et al. (2020) have been developed and have strong performance in many natural language tasks. Furthermore, the rise of many large language models (LLMs) recently, such as Llama Touvron et al. (2023) , Mixtral Jiang et al. (2024) , Qwen Bai et al. (2023) , and Phi Gunasekar et al. (2023) , has gained strong attention for the research community due to their exceptional performance in text generation. However, LLMs have one disadvantage, they cannot access the custom data and new information to update their knowledge, which makes them unable to shift their knowledge to fit different applications. Consequently, Retrieval-Augmented Generation Lewis et al. (2020) systems (or RAG) are invented to handle the problem by utilizing retrieval systems to search for relevant information from the database before feeding those information to LLMs as an extra context. This shows the necessity and importance of embedding language models for retrieval and reranking tasks in the era of LLMs. Despite the importance of retrieval systems for LLMs, in Vietnam, the number of existing benchmarks for retrieval and reranking tasks are limited, which leads to the difficulty in comparing and assessing the performance of many Vietnamese embedding language models on those two tasks. Despite there are some Vietnamese benchmarks like ViGLUE Tran et al. (2024a) , ViNLI Huynh et al. (2022) , VMNLU 1 1 1 https://github.com/ZaloAI-Jaist/VMLU.git , and VSFC Nguyen et al. (2018a) , none of them evaluate performance of language models on retrieval and reranking tasks. This paper attempts to address the need for those benchmarks by introducing a new benchmark, the Vietnamese Context Search (or the VCS) to evaluate the ability of text embedding models to search for relevant Vietnamese documents. This benchmark is constructed using existing Vietnamese datasets with modifications in their structure and tasks. Despite having a simple construction process, this benchmark effectively provides different inspections of Vietnamese text embedding models. The VCS serves as a standard and high-quality benchmark to evaluate and compare different Vietnamese embedding models on retrieval and reranking tasks. Furthermore, this work also introduces a new training objective to train Vietnamese embedding language models on retrieval and reranking tasks. This training objective aims to yield better performance of embedding language models compared to the InfoNCE loss function, which is usually used in contrastive learning. The research will experiment with different training objectives with two training methods, including in-batch negative and curated hard-negative to compare the ability of two loss functions. Next, the evaluation of some existing Vietnamese embedding language models on the VCS benchmark is conducted to examine their ability in context search. Lastly, an empirical study is conducted to understand the effect of temperature \tau in the loss function on the overall performance of embedding models. Different training methods are included in the study to further investigate the impact of temperature on the loss function. To conclude, this work includes three primary contributions: • First, introduce a new Vietnamese benchmark, the VCS, to evaluate Vietnamese language models in their ability to search relevant documents. This benchmark evaluates models on two tasks, retrieval and reranking tasks. • Second, introduce a new training objective function to train text embedding models on retrieval and reranking tasks • Lastly, we conduct an empirical study to investigate the impact of temperature, a hyper-parameter, of the InfoNCE and our loss functions in the performance of embedding language models on reranking and retrieval tasks. 2 Related Work In the era of large language models (LLMs), not only does the development of different generative language models such as Gemma Team et al. (2024) , SeaLLM Nguyen et al. (2023) , and Mamba Dao and Gu (2024) gains the attention from the community, but also do embedding language models, especially those support searching text documents like GTE Li et al. (2023) , NV-Embed Lee et al. (2024) , BGE Luo et al. (2024) , or GritLM Muennighoff et al. (2024) , become more important due to their applications in RAG systems, which provide more context and information for LLMs to generate correct answers. Consequently, many works aim to provide a benchmark to evaluate language models on their ability in information retrieval (IR) such as BEIR Thakur et al. (2021) , MTEB Muennighoff et al. (2023) , BRIGHT Su et al. (2024) , and ReQA Ahmad et al. (2019) . Those benchmarks advance the development of many text embedding language models and the research of natural language processing (NLP) by supporting the research community with resources to compare and evaluate text embedding models. However, similar and comparable benchmarks for Vietnamese embedding language models are limited. While there are some benchmarks like ViGLUE Tran et al. (2024a) , ViQuAD Nguyen et al. (2020) , ViSFD Luc Phan et al. (2021) , VMLU, VSMEC Ho et al. (2020) , and VSFC Nguyen et al. (2018b) , they mostly focus on question-answering and natural language understanding aspects of language models and completely ignore the ability of language models in retrieval and reranking tasks. That leads to the difficulty in evaluating and comparing Vietnamese text embedding models in their ability of retrieve relevant information. Despite some Vietnamese embedding language models being created, without a standard benchmark on this field, the Vietnamese research community is unable to know the benefits, pros, and cons of those language models, which can lead to misleading when applying them to applications (RAG systems) or research projects. In the early days of information retrieval, different systems were created to find relevant text information from large databases such as TF-IDF Sammut and Webb (2010) , BM25 Amati (2009) , and BM25F Pérez-Agüera et al. (2010) . However, those methods cannot capture the context of documents and use it for the retrieval process. Consequently, different retrieval methods using deep learning models are utilized to encode a piece of text to a vector that can present different or hidden aspects of it. Many pre-trained language models (PLMs), including BERT Devlin et al. (2019) ,