QZhou-Embedding Technical Report

Peng Yu, En Xu, Bin Chen, Haibiao Chen, Yinfei Xu · 2025

Read QZhou-Embedding Technical Report by Peng Yu, En Xu, Bin Chen, Haibiao Chen, Yinfei Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-21632.txt.

Introduction

Peng Yu, En Xu, Bin Chen, Haibiao Chen, and Yinfei Xu report QZhou-Embedding, a model that turns text into codes that hold its sense, for use in search and match tasks. Their report is from two thousand twenty-five.

The model is built on a known seven-billion-term base that follows commands. The team gives it one frame for many tasks, with set ways to shape data and train. A data pipeline uses a large model to reword, grow, and forge hard near-miss cases. Training runs in two stages: a first pass on search, then a full-task tune. The model tops two known leader boards for such codes, in English and in Chinese, at the time of the report.

Catalog record

CREATORPeng Yu, En Xu, Bin Chen, Haibiao Chen, Yinfei Xu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-21632

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

QZhou-Embedding Technical Report Thanks: https://kingsoft.com/ Peng Yu, En Xu, Bin Chen, Haibiao Chen, Yinfei Xu Abstract We present QZhou-Embedding, a general-purpose contextual text embedding model with exceptional text representation capabilities. Built upon the Qwen2.5-7B-Instruct foundation model, we designed a unified multi-task framework comprising specialized data transformation and training strategies. The data transformation scheme enables the incorporation of more diverse textual training datasets, while the task-specific training strategies enhance model learning efficiency. We developed a data synthesis pipeline leveraging LLM API, incorporating techniques such as Paraphrasing, Augmentation, and Hard negative example generation to improve the semantic richness and sample difficulty of the training set. Additionally, we employ a two-stage training strategy, comprising initial retrieval-focused pretraining followed by full-task fine-tuning, enabling the embedding model to extend its capabilities based on robust retrieval performance. Our model achieves state-of-the-art results on the MTEB and CMTEB benchmarks, ranking first on both leaderboards(August 27, 2025), simultaneously achieves state-of-the-art performance on tasks including Reranking, Clustering, etc. Our findings demonstrate that higher-quality, more diverse data is crucial for advancing retrieval model performance, and that leveraging LLMs’ generative capabilities can further optimize data quality for embedding model breakthroughs. Our model weights are released on HuggingFace 1 1 1 https://huggingface.co/Kingsoft-LLM/QZhou-Embedding under Apache 2.0 license. For reproducibility, we provide evaluation code and instructions on GitHub 2 2 2 https://github.com/Kingsoft-LLM/QZhou-Embedding . 1 Introduction Text embedding models, which transform natural language text into mathematical vector representations, play an indispensable role in text mining, question-answering systems, recommendation systems, and retrieval-augmented generation. Recently, LLM-based agent technology has experienced rapid development and widespread adoption, embedding models, which transform textual or multimodal data into vector representations for knowledge base construction, have significantly enhanced agent systems in terms of real-time performance, long-term memory, data privacy preservation, and knowledge integration capabilities. With the continuous advancement of neural networks and deep learning, text embeddings have evolved from early sparse representations (e.g., BM25 [ 1 ] ) to dense representations based on fine-tuned deep networks such as BERT [ 2 ] and T5 [ 3 ] , leading to significant performance improvements [ 4 ] [ 5 ] [ 6 ] [ 7 ] [ 8 ] . In 2022, the rise of large language models (LLMs), exemplified by ChatGPT [ 9 ] , ushered in a new era of text embeddings based on LLM representations, including models like text-embedding-3-large and RepLLaMA [ 10 ] . Recent research on optimizing text embedding models has explored diverse perspectives and focal points. For instance, to address the limitation of decoder-only architectures—where causal attention mechanisms restrict token embeddings to unidirectional semantic capture—several approaches have been proposed: Echo Embedding [ 11 ] employs input repetition and instruction design to enable preceding tokens to capture subsequent token semantics. LLM2Vec [ 12 ] modifies attention to bi-directional mechanism to remove backward dependency constraints. Conan-Embedding-v2 [ 13 ] proposes a novel soft masking mechanism combined with dynamic rank reduction. Another widely adopted approach is knowledge distillation, where text embeddings are treated as the ”signal states” representing textual semantics. By distilling knowledge from high-performing teacher models to student models, the objective is to optimize the embedding performance. For instance, Jasper [ 14 ] employs a multi-stage knowledge distillation framework, combining with multiple carefully designed loss functions and finally achieving superior results. Debater [ 16 ] proposes a step-by-step thinking mechanism for embedding generation, iteratively optimizing document representations through continuous COT. Distillation is applied to constrain the final token representation to learn the optimal semantic states from these thinking steps. Additionally, hard negative sampling has emerged as a crucial research direction in text embedding models, serving as a pivotal technique for model optimization. ANCE [ 18 ] identified that conventional dense retrieval training leads to diminishing gradient norms during optimization. Thus they developed an asynchronous Approximate Nearest Neighbor (ANN) indexing mechanism that periodically refreshes the negative sample pool using the current model parameters, thereby ensuring the maintenance of up-to-date and optimally challenging negative samples. Both Conan-Embedding [ 24 ] and its v2 version incorporated similar dynamic hard negative sampling techniques to enhance model performance. NV-Embed [ 19 ] implemented an alternative approach by leveraging their previously developed NV-Retriever’s [ 20 ] positive-aware negative mining strategy, including TopK-MarginPos and TopKPercPos filtering mechanisms. In this work, we present QZhou-Embedding, built upon the powerful Qwen2.5-7B-Instruct [ 21 ] model, which pushes the boundaries of text embedding capabilities. To enhance the model’s semantic understanding, we designed a unified multi-task learning framework that not only accommodates more diverse training data but also bring efficient learning across three key tasks: retrieval, natural language inference (NLI), and classification. Our framework comprises two core components: 1. Data Transformation: We carefully adapt data formats to the specific requirements of retrieval, NLI, and classification tasks, enabling effective feature extraction from heterogeneous data sources, significantly benefiting retrieval model training. 2. Training Strategy: We designed specialized loss functions based on each task’s characteristics, optimizing model training efficiency. To further improve the robustness and generalization of vector representation, we propose a data synthesis method by employing three techniques to address data scarcity: Paraphrasing & Data augmentation for limited datasets and Hard negative generation for negative sample enrichment. Building upon prior work, we designed a strategy named ”Data Grouping Strategy”, enabling batch sampling within single datasets, inadvertently increasing training difficulty through in-batch negative sampling from the same distribution. For model training, we used a two-phase training approach, through the first-stage retrieval training and second-stage full-capability training, our model acquires a solid foundation of retrieval capabilities, while effectively extending to multiple capability dimensions. Our model achieved state-of-the-art average scores on CMTEB [ 22 ] and MTEB [ 23 ] benchmarks, ranking first overall on both CMTEB and MTEB leaderboards, demonstrating the effectiveness of our approach. The contributions of our work are summarized as follows: • We propose a unified multi-task learning framework that systematically coordinates both data processing and training pipelines, enhancing diversity in datasets and efficiency in model training ; • We develop advanced data synthesis techniques powered by LLM, including Paraphrasing, Data augmentation, and Hard negative generation. These methods significantly enhance the quality of training corpora, thereby improving model’s robustness and generalization capabilities; • We emply a two-stage training paradigm: Stage 1 focuses exclusively on retrieval capability building, establishing strong foundational retrieval performance; and stage 2 implements balanced training with controled retrieval/non-retrieval task ratios, achieving superior performance on classification (CLS), pair classification (PairCLS), and semantic textual similarity (STS) tasks while maintaining retrieval effectiveness; • Our model achieves state-of-the-art performance on both MTEB and CMTEB benchmarks, which validates the effectiveness of our proposed methods. 2 Related Works 2.1 Text Embedding Models Text vector representation is a fundamental research area in natural language processing (NLP) and serves as the cornerstone for language understanding. Early approaches relied on sparse vector representations, such as TF-IDF [ 25 ] , BM25 [ 26 ] , and LSA [ 27 ] . With the advent of pretrained language models, dense contextualized representations based on architectures like BERT [ 2 ] and T5 [ 3 ] became widely studied and applied [ 4 ] [ 5 ] [ 6 ] . In the era of large language models (LLMs), major advancements have led to the development of LLM-based embedding models,

Read the complete text