Architecture is All You Need: Improving LLM Recommenders by Dropping the Text

Kevin Foley, Shaghayegh Agah · 2025

Read Architecture is All You Need: Improving LLM Recommenders by Dropping the Text by Kevin Foley, Shaghayegh Agah (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15833.txt.

Introduction

Kevin Foley, Shaghayegh Agah, and co-authors test a lean design for recommendation systems in this 2025 study. Many recent systems use large language models to suggest films, goods, or media. Those models read item text and carry broad world facts, but they are big, slow, and costly to run and tune.

The authors keep only the shape of such a model, shrunk down, and swap word pieces for plain tokens that stand for items. Their small system beat both classic recommenders and large language model systems on shared test sets. It did so at a small share of the size and cost. Their claim is that the shape of the model, not its stored world facts, does most of the work.

Catalog record

CREATORKevin Foley, Shaghayegh Agah
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-15833

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Architecture is All You Need: Improving LLM Recommenders by Dropping the Text Kevin Foley, Shaghayegh Agah Abstract In recent years, there has been an explosion of interest in the applications of large pre-trained language models (PLMs) to recommender systems, with many studies showing strong performance of PLMs on common benchmark datasets. PLM-based recommender models benefit from flexible and customizable prompting, an unlimited vocabulary of recommendable items, and general “world knowledge” acquired through pre-training on massive text corpora. While PLM-based recommenders show promise in settings where data is limited, they are hard to implement in practice due to their large size and computational cost. Additionally, fine-tuning PLMs to improve performance on collaborative signals may degrade the model’s capacity for world knowledge and generalizability. We propose a recommender model that uses the architecture of large language models (LLMs) while reducing layer count and dimensions and replacing the text-based subword tokenization of a typical LLM with discrete tokens that uniquely represent individual content items. We find that this simplified approach substantially outperforms both traditional sequential recommender models and PLM-based recommender models at a tiny fraction of the size and computational complexity of PLM-based models. Our results suggest that the principal benefit of LLMs in recommender systems is their architecture, rather than the world knowledge acquired during extensive pre-training. Keywords Multitask, Recommendation, Personalization, Large Language Models, Content Retrieval 1 Introduction Recent research in the field of recommender systems has begun to explore the capabilities of large language models (LLMs) as bases for fine-tuned next-item recommendation models. These LLM-based recommenders have shown strong performance and techniques like Supervised Fine-Tuning (SFT) [ 1 ] and Direct Preference Optimization (DPO) [ 2 , 3 ] have been used to further tune language models’ outputs to produce high-quality recommendations. The LLM model architectures used in large pretrained language models (PLMs) offer many advantages over earlier generations of sequential model architectures. Above all, LLMs are excellent pattern learners. Decoder-only transformer architectures with causal attention masks excel at next-token prediction in settings with long context windows and complex dependencies and interactions between input tokens. This is an advantage in language modeling that can just as easily be applied to the next-item prediction task that is at the center of most recommendation systems. Figure 1: Model architecture. With the exception of the SiLU activation to reduce parameter count, the architecture is the same as that of the Llama3 family of models. In addition to architectural advantages of LLMs, PLMs can contribute to recommender systems with world knowledge . They excel at generalizing and synthesizing information from massive collections of text, and can be quickly adapted to new tasks through fine tuning or in-context learning and few-shot prompting [ 4 ] . When used in recommendation systems, PLMs have shown a promising ability to generate high quality recommendations on little or no training data and to recommend cold start items not previously seen in the training data. Several recent PLM recommender models – BigRec [ 1 ] , TALLRec [ 5 ] , SPRec [ 6 ] , S-DPO [ 3 ] , RoseDPO [ 7 ] , LLara [ 8 ] , and GPT4Rec [ 9 ] – have shown strong performance against popular benchmarks. On the other hand, PLM recommender systems are computationally expensive, often thousands of times larger than sophisticated transformer-based recommender models. They require many more tokens to be processed per recommended item in both input prompts and generated output. The world knowledge of PLMs has been shown to degrade after finetuning [ 10 ] , meaning that attempts to improve recommendation performance through fine tuning could limit the advantage of the PLM’s world knowledge. Additionally, a recent study [ 11 ] finds that the PLMs used for recommender models may have been exposed during training to common benchmark datasets, calling into question the strong results that have been reported by these models. In this work, we develop LSRec, a lightweight sequential recommender, and compare the benefits of the LLM architecture to the benefits of PLM’s world knowledge. LSRec shares the same causal decoder-only transformer architecture as the Llama family of open-weight PLMs without using text input or pre-trained weights. We compare our model performance with SASRec, a leading transformer-based next-item recommender operating directly on item tokens, and with BigRec, a recent fine-tuned PLM recommender that shows strong performance on limited training data. 2 Methodology Traditional next-item recommendation models typically take as input a sequence of item ids that a user has interacted with, and either score a single candidate item or output a vector of scores in the style of a multilabel classifier. PLM-based models typically take the title, year, and possibly additional metadata for each item watched, format the sequence of watched titles and metadata into a string as part of a larger text prompt explaining the recommendation task, and generate the title and year of an output item. The generated text serves as the model’s recommendation and must be mapped to item ids. We propose a hybrid approach combining the architecture of a causal, decoder-only transformer model with the item-level tokenization of a traditional sequential recommendation model. We base our model on the Llama 3 architecture [ 12 ] . This is preferred over alternative LLM architectures due to its relatively simple design and wide adoption in applied research papers, including in PLM recommendation model research. We reduce model size by specifying a shorter context window, fewer layers and hidden dimensions, a single key-value attention head instead of the multiple key-value heads in a typical LLM, and limiting vocabulary size to the full set of ids in the item vocabulary plus a handful of special tokens for multi-task training (explained in the following section). Other configuration differences between a pretrained Llama model and LSRec are shown in Table 1 . To further reduce the size of our models, we use sigmoid linear unit (SiLU) activation instead of the Swish-gated Linear Unit (SwiGLU) activation function used in the Llama family of LLMs. We specify three models ranging in size from 768 thousands to 7.1 million parameters. With these modifications, LSRec ranges from around 150 to 1,000 times smaller than even the smallest pretrained Llama models. Simplified model architecture is shown in Figure 1 . Parameter Llama 3.2-1b LSRec-768k LSRec-2.2M LSRec-7.1M params 1B 768k 2.2M 7.1M attn dropout 0.0 0.2 0.2 0.2 hidden dims 2048 64 128 384 intermediate dims 8192 128 384 512 context length 131k 200 200 200 attn heads 32 1 2 2 layers 16 2 4 4 kv heads 8 1 1 1 vocab size 128k 10k 10k 10k Table 1: Key Configuration Differences vs. Llama3.2-1b 2.1 Data preparation We focus our analysis on recommendations in the movie domain and train and evaluate various LSRec versions on the MovieLens-10M dataset, a widely-used benchmark dataset in recommender system papers. The dataset consists of 10 million user ratings from 72,000 users across 10,000 movies, with genre tags for each individual movie. User-item interactions are ordered by timestamp into user-item sequences. Users with fewer than five interacted items are dropped from the dataset, and sequences are truncated to a maximum length of 200 items. We prepare our training data for supervised fine tuning (SFT) using a

Read the complete text