Tokenize Once, Recommend Anywhere: Unified Item Tokenization for Multi-domain LLM-based Recommendation

2025

Read Tokenize Once, Recommend Anywhere: Unified Item Tokenization for Multi-domain LLM-based Recommendation (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-12922.txt.

Description

Yu Hou and Won-Yong Shin unify how language-model recommenders turn items into tokens. Recommender systems built on language models must translate products, songs, or videos into discrete tokens the model can reason over. Existing tokenizers are trained per domain, so a shop and a music service each need their own, and knowledge does not carry across. UniTok instead projects items from all domains into one latent space. Then routes them through a mixture of experts: domain specialists for each field's quirks, plus a shared expert that is always on for transferable knowledge.

A calibration step keeps each domain's tokens equally informative. Across wide-ranging real datasets, UniTok improved over strong benchmarks by large margins and kept working in new domains without retraining. The design is aimed at platforms that run recommendations across many catalogs at once and cannot afford a bespoke tokenizer for each.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-12922

Browse the full library · About the Tower of Babel