Tokenize Once, Recommend Anywhere: Unified Item Tokenization for Multi-domain LLM-based Recommendation
2025
Read Tokenize Once, Recommend Anywhere: Unified Item Tokenization for Multi-domain LLM-based Recommendation (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-12922.txt.
Description
Yu Hou and Won-Yong Shin unify how language-model recommenders turn items into tokens. Recommender systems built on language models must translate products, songs, or videos into discrete tokens the model can reason over. Existing tokenizers are trained per domain, so a shop and a music service each need their own, and knowledge does not carry across. UniTok instead projects items from all domains into one latent space. Then routes them through a mixture of experts: domain specialists for each field's quirks, plus a shared expert that is always on for transferable knowledge.
A calibration step keeps each domain's tokens equally informative. Across wide-ranging real datasets, UniTok improved over strong benchmarks by large margins and kept working in new domains without retraining. The design is aimed at platforms that run recommendations across many catalogs at once and cannot afford a bespoke tokenizer for each.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-12922