Don’t Reinvent the Wheel: Efficient Instruction-Following Text Embedding based on Guided Space Transformation

Yingchaojie Feng, Yiqun Sun, Yandong Sun, Minfeng Zhu, Qiang Huang, Anthony K. H. Tung, Wei Chen · 2025

Read Don’t Reinvent the Wheel: Efficient Instruction-Following Text Embedding based on Guided Space Transformation by Yingchaojie Feng, Yiqun Sun, Yandong Sun, Minfeng Zhu, Qiang Huang, Anthony K. H. Tung, Wei Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24754.txt.

Introduction

Text embeddings that obey instructions usually demand re-encoding the whole corpus for every new instruction, and Yingchaojie Feng and co-authors refuse to pay that price. Their framework, GSTransform, transforms pre-computed generic embeddings in real time instead. The key insight is that instruction-relevant signals already hide inside generic embeddings, waiting to be rotated into view. A small set of instruction-labeled examples guides the transformation.

Across nine datasets and three downstream tasks, the method beat state-of-the-art instruction-following embedders on quality. It also ran 6 to 300 times faster on large collections. The team releases the code publicly.

Catalog record

CREATORYingchaojie Feng, Yiqun Sun, Yandong Sun, Minfeng Zhu, Qiang Huang, Anthony K. H. Tung, Wei Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24754

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Don’t Reinvent the Wheel: Efficient Instruction-Following Text Embedding based on Guided Space Transformation Yingchaojie Feng, Yiqun Sun, Yandong Sun, Minfeng Zhu, Qiang Huang, Anthony K. H. Tung, Wei Chen Abstract In this work, we investigate an important task named instruction-following text embedding, which generates dynamic text embeddings that adapt to user instructions, highlighting specific attributes of text. Despite recent advancements, existing approaches suffer from significant computational overhead, as they require re-encoding the entire corpus for each new instruction. To address this challenge, we propose GSTransform , a novel instruction-following text embedding framework based on G uided S pace Transform ation. Our key observation is that instruction-relevant information is inherently encoded in generic embeddings but remains underutilized. Instead of repeatedly encoding the corpus for each instruction, GSTransform is a lightweight transformation mechanism that adapts pre-computed embeddings in real time to align with user instructions, guided by a small amount of text data with instruction-focused label annotation. We conduct extensive experiments on three instruction-awareness downstream tasks across nine real-world datasets, demonstrating that GSTransform improves instruction-following text embedding quality over state-of-the-art methods while achieving dramatic speedups of 6 ∼ \sim 300 × \times in real-time processing on large-scale datasets. The source code is available at https://github.com/YingchaojieFeng/GSTransform . 1 Introduction Figure 1: An illustration of instruction-following text embeddings. Depending on different instructions, the embeddings reorganize their similarity relationships, forming distinct clusters that emphasize different aspects. Text embedding ( Mikolov et al., 2013 ; Pennington et al., 2014 ; Zhuo et al., 2023 ; Li and Li, 2024 ) is a fundamental NLP problem, serving as the backbone of numerous applications such as clustering ( Aggarwal and Zhai, 2012 ) , Semantic Textual Similarity (STS) ( Agirre et al., 2012 ; Agirre et al., 2013 ) , and information retrieval ( Karpukhin et al., 2020 ; Thakur et al., 2021 ) . The core objective of text embedding models is to convert textual data into fixed-length vector representations that encode semantic relationships, ensuring that similar texts remain close in the embedding space while dissimilar ones are well separated ( Muennighoff et al., 2023 ) . Despite their widespread use, generic embedding models are inherently static and inflexible, often failing to adapt to task-specific requirements. As illustrated in Figure 1 , real-world text data frequently encapsulates multiple semantic aspects (e.g., “intent” and “emotion”), yet traditional embeddings primarily focus on general semantics rather than user-specified perspectives. This limitation restricts their applicability in scenarios where users seek embeddings tailored to specific needs. To address this challenge, recent research has introduced instruction-following text embedding, which generates embeddings in different semantic spaces conditioned on user-provided instructions ( Su et al., 2023 ; Peng et al., 2024 ) . These models allow users to emphasize specific aspects of text, dynamically adjusting embeddings based on task-oriented instructions. Notably, InstructOR Su et al. (2023) concatenates instruction with text as input and fine-tunes a Transformer-based model across a diverse set of instructions to capture instruction-aware semantics. Alternatively, InBedder Peng et al. (2024) treats instructions as questions and encodes the corresponding answers as the final embeddings, leveraging both strengths of generative models and embedding models. Although these methods improve instruction-awareness, they suffer from significant efficiency limitations when applied to large-scale datasets. The primary issue stems from their dependence on full corpus re-encoding for each instruction. Given m instructions and n texts, both InstructOR and InBedder require O(m\times n) forward passes to generate embeddings, making it computationally expensive and impractical for real-world applications where pre-encoded embeddings are stored in vector databases. This inefficiency hinders experimentation with different instructions, as even minor changes require reprocessing the entire dataset. On the other hand, we observe that generic text embeddings already contain latent instruction-relevant information but do not prioritize it explicitly. Instead of re-encoding the full corpus for every new instruction, we propose a transformation-based approach that dynamically adjusts existing embeddings to align with user-specified instructions. As illustrated in Figure 1 , different users may prioritize distinct semantic attributes. For instance, when the focus is on the “intent” aspect, the transformation highlights the shape of embeddings to cluster texts based on their underlying purpose. Conversely, when the emphasis shifts to the “emotion” aspect, the transformation accentuates color, grouping texts by their sentiment. To this end, we introduce GSTransform , a novel instruction-following text embedding framework based on G uided S pace Transform ation, eliminating the need for exhaustive re-encoding. GSTransform consists of two components: • Instruction-based Label Construction: We build a label taxonomy from user instructions to categorize text representations according to instruction-specific semantics. • Label-guided Embedding Transformation: We adapt the original embedding space using these instruction-driven labels to align with user-specified information. Rather than regenerating embeddings from scratch, GSTransform treats instructions as transformation operators that strategically reorient attention within the existing semantic space, ensuring adaptability without excessive computation. The instruction-driven label data guides the training of transformation models, providing a clearer optimization objective than direct training on instructions. Contributions This work makes the following essential contributions: (1) Efficient Instruction-Following Embedding: We propose GSTransform, a novel framework that enables instruction-following text embeddings via guided space transformation, eliminating the need for re-encoding the entire corpus for each instruction. (2) Cost-Effective Instruction Adaptation: GSTransform introduces a Label-guided Embedding Transformation mechanism that adapts pre-computed embeddings to user instructions using a lightweight model trained on a small, annotated subset, requiring only a fixed number of LLM calls, making it highly cost-effective for large-scale applications. (3) Comprehensive Empirical Validation: We conduct extensive experiments on three instruction-awareness tasks across nine real-world datasets, showing that GSTransform improves embedding quality while significantly reducing computational overhead and latency on large-scale datasets compared to state-of-the-art baselines. 2 Related Work 2.1 Generic Text Embedding Text embedding has been a long-studied problem. Since word embeddings, people adopt self-supervised training in generating word embeddings, and pool the word embeddings to form text embeddings Mikolov et al. (2013) ; Pennington et al. (2014) . Recent advancements in context-aware semantic text embedding models leverage Transformer-based architectures Vaswani et al. (2017) ; Devlin et al. (2019) as their backbone, often employing customized objectives like contrastive loss to train the models Cer et al. (2018) ; Reimers and Gurevych (2019) ; Gao et al. (2021) ; Zhuo et al. (2023) . Moreover, state-of-the-art (SOTA) text embedding models have been further enhanced with techniques such as using large language models (LLMs) Wang et al. (2023) ; Muennighoff et al. (2025) ; Lei et al. (2024) and more sophisticated loss functions designed to address issues like cosine saturation Li and Li (2024) . Despite their effectiveness, these methods lack generalizability and fail to meet diverse user needs when downstream tasks require focusing on specific aspects beyond general semantics. 2.2 Instruction-Following Text Embedding Instruction-following text embedding Su et al. (2023) ; Peng et al. (2024) allows users to guide embedding generation through customized instructions. The model produces embeddings that align with users’ specific interests by considering both the input text and instructions. InstructOR Su et al. (2023) pioneered instruction-based embeddings by concatenating instructions with input texts and training the model using contrastive objectives across a diverse set of instructions. It adapts embeddings for varied semantic interpretations but does not explicitly model instruction-specific semantic aspects. InBedder Peng et al. (2024)

Read the complete text