GLoSS: Generative Language Models with Semantic Search for Sequential Recommendation
Krishna Acharya, Aleksandr V. Petrov, Juba Ziani · 2025
Read GLoSS: Generative Language Models with Semantic Search for Sequential Recommendation by Krishna Acharya, Aleksandr V. Petrov, Juba Ziani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01910.txt.
Introduction
Krishna Acharya, Aleksandr Petrov, and Juba Ziani present a recommendation system that writes its own search queries. Their system, called GLoSS, appeared in twenty twenty-five. It aims to guess what a shopper will want next from the order of past picks.
Earlier systems matched words exactly when searching for items. GLoSS instead searches by meaning, so it can find items that share no words with the query. A small, fine-tuned language model writes the queries, which keeps the cost low on modest hardware. On three Amazon review data sets, the system beat both classic methods and rival language model systems. It did especially well for new users with little history on two of the sets.
Catalog record
| CREATOR | Krishna Acharya, Aleksandr V. Petrov, Juba Ziani |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01910
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GLoSS: Generative Language Models with Semantic Search for Sequential Recommendation Krishna Acharya, Aleksandr V. Petrov, Juba Ziani Abstract . We propose G enerative Lo w-rank language model with S emantic S earch (GLoSS) , a generative recommendation framework that combines large language models with dense retrieval for sequential recommendation. Unlike prior methods such as GPT4Rec, which rely on lexical matching via BM25, GLoSS uses semantic search to retrieve relevant items beyond lexical matching. For query generation, we employ 4-bit quantized LlaMA-3 models fine-tuned with low-rank adaptation (LoRA), enabling efficient training and inference on modest hardware. We evaluate GLoSS on three real-world Amazon review datasets: Beauty, Toys, and Sports, and find that it achieves state-of-the-art performance. Compared to traditional ID-based baselines, GLoSS improves Recall@5 by 33.3%, 52.8%, and 15.2%, and NDCG@5 by 30.0%, 42.6%, and 16.1%, respectively. It also outperforms LLM-based recommenders such as P5, GPT4Rec, LlamaRec and E4SRec with Recall@5 gains of 4.3%, 22.8%, and 29.5%. Additionally, user segment evaluations show that GLoSS performs particularly well for cold-start users in the Amazon Toys and Sports datasets, and benefits from longer user histories in Amazon Beauty dataset, demonstrating robustness across different levels of interaction lengths. Our code and model checkpoints are publicly available at: https://github.com/krishnacharya/GLoSS Figure 1. GLoSS: a finetuned LLM generates queries, which are then used to perform semantic search over the item catalog. 1. Introduction Sequential Recommender Systems predict the next item in the sequence of user-to-item interaction. Until recently, id-based approaches dominated the field of sequential recommendation: these methods typically learn an embedding representation for every item in the catalog, as well as a sequence encoder —a model that represents the whole sequence of user-item interactions in the same space of embeddings. Id-based methods then compute item scores using a similarity function (e.g., dot product) between the sequence embedding and the item embedding. Some of the most notable examples of id-based methods include SASRec ( Kang and McAuley 2018 ) and BERT4Rec ( Sun et al. 2019 ) . Despite achieving high recommendation accuracy on many sequential datasets, id-based methods have a number of limitations. In particular, these id-based methods require item embeddings to be learned for every item in the catalog, meaning that new items can be only recommended after the recommendation model has been retrained. In addition, models like SASRec and BERT4Rec are trained only using data directly presented in the training set, which constrains their ability to generalize beyond seen patterns or infer relevance based on rich item metadata or natural language descriptions. In contrast, textual document retrieval systems do not suffer from these limitations: documents (equivalent of “items” in recommender systems) are typically indexed independently of the scoring model, and, hence can be retrieved immediately after being added to index. Moreover, the widespread adaptation of pre-trained language models (such as BERT ( Devlin et al. 2019 ) ), has enabled fine-tuning for retrieval, allowing the models to generalize beyond the training data and retrieve relevant documents even in the absence of direct lexical overlap, leading to substantial improvements in retrieval effectiveness ( Karpukhin et al. 2020 ; Reimers and Gurevych 2019 ) . Recent advances in generative language models have enabled their integration into recommendation pipelines. For instance, P5Rec ( Geng et al. 2022 ) adopts T5 ( Raffel et al. 2020 ) as a unified backbone to support multiple tasks including sequential recommendation, rating prediction, and review summarization. GPT4Rec ( Li et al. 2023b ) uses GPT-2 to generate multiple queries per user, which are then matched to items via a BM25 lexical retriever. More recently, LlamaRec ( Yue et al. 2023 ) employs LLaMA-2 as a reranker over candidates selected by an LRU-based recommender ( Yue et al. 2024 ) . Other methods, such as CALRec and GenRec ( Li et al. 2024 ; Ji et al. 2024 ) , follow a similar query generation and retrieval framework, combining LLMs like PaLM ( Anil et al. 2023 ) or LLaMA-2 ( Touvron et al. 2023 ) with BM25-based lexical search. However, many of these methods use BM25, a lexical matching algorithm that does not capture the semantic meaning of generated queries. Furthermore, approaches like P5, GPT4Rec, and CALRec rely on older language models (e.g., T5, GPT-2, PaLM) and use full fine-tuning, which is computationally intensive. Over the past five years, generative language models have improved significantly due to advancements in architecture, better training data, and more effective training methods. This progress has resulted in models like the LLaMA-3 series ( Grattafiori et al. 2024 ) , which are substantially more capable for tasks such as query generation. Additionally, parameter-efficient fine-tuning techniques such as QLoRA ( Dettmers et al. 2023 ) have dramatically reduced the resource requirements for training, enabling efficient fine-tuning on consumer GPUs. Inspired by the GPT4Rec approach and aiming to address its limitations, we propose G enerative Lo w-rank language model with S emantic S earch (GLoSS) , a generative recommendation framework for sequential recommendation. Unlike GPT4Rec, GLoSS uses semantic search (aka. dense retrieval) to enable item recommendation beyond direct lexical matching. Our query generation backbone uses 4-bit quantized LLaMA-3 models fine-tuned via low-rank adapters. We evaluate GLoSS on three real-world Amazon review datasets: Beauty, Toys, and Sports ( He and McAuley 2016a ) —and demonstrate that it achieves state-of-the-art performance. Compared to ID-based baselines, GLoSS yields substantial gains in Recall@5 (33.3% on Beauty, 52.8% on Toys, and 15.2% on Sports) and NDCG@5 (30.0%, 42.6%, and 16.1%, respectively). GLoSS also outperforms LLM-based recommenders such as P5, GPT4Rec, LlamaRec and E4SRec, with Recall@5 improvements of 4.3% on Beauty, 22.8% on Toys, and 29.5% on Sports—while maintaining competitive NDCG@5 without relying on a reranking stage. In summary, the contributions of this paper are as follows (1) We apply low-rank adaptation (LoRA) to fine-tune 4-bit quantized LLaMA-3 models for query generation, showing that high-quality generative performance can be achieved with minimal resources. (2) We integrate dense retrieval into the generative recommendation pipeline to enable retrieval beyond lexical matching. Our analysis in Section 4.2.2 shows that dense retrieval is a critical component, yielding substantial gains in NDCG@5 and Recall@5 over BM25. (3) Through extensive experiments on Amazon datasets, we show that GLoSS outperforms both ID- and LLM-based baselines with consistent double-digit improvements in Recall@5 and NDCG@5. (4) We analyze GLoSS across user segments based on interaction history length: cold-start, regular, and power users. To our knowledge, this is the first such study for LLM-based recommenders. GLoSS performs best on cold-start users in Toys and Sports, and benefits from longer histories in Beauty, demonstrating robustness across user interaction lengths. The rest of the paper is organised as follows: Section 2 contains the description of similar methods in RecSys and IR; Section 3 describes our proposed GLoSS; Section 4 describes experimental evaluation; Section 5 contains concluding remarks. We now describe prior related to ours. 2. Recommendation through Retrieval This section describes methods from the fields of Recommender Systems and Information Retrieval. Section 2.1 describes classic content-based methods; Section 2.2 describes recent generative methods; Section 2.3 discusses similar methods from the IR field. 2.1. Content-based filtering Our work retrieves items from the catalog using textual description, generated by a language model. Hence, our method has similarities with