Concept Tokens: Learning Behavioral Embeddings Through Concept Definitions

Ignacio Sastre, Aiala Rosá · 2026

Read Concept Tokens: Learning Behavioral Embeddings Through Concept Definitions by Ignacio Sastre, Aiala Rosá (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-04465.txt.

Description

Ignacio Sastre and Aiala Rosá teach a language model a new word for an idea, and nothing else, in a 2026 arXiv paper. Their Concept Tokens method adds one special token to a pretrained model's vocabulary and learns only that token's embedding. From natural-language definitions of a target concept in which the concept's name is replaced by the token. The model itself stays frozen throughout.

Two testbeds show the token genuinely steers behaviour. A token trained on hallucination can be asserted to increase invented answers in closed-book questions, or negated to suppress them, mostly by making the model abstain. A token for recasting, a language teacher's correction technique, shows the same directional control and follows other instructions better than stuffing the definitions into the prompt. A qualitative study with the Eiffel Tower and an invented Austral Tower probes what the embedding has and has not captured.

Catalog record

CREATORIgnacio Sastre, Aiala Rosá
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-04465

Browse the full library · About the Tower of Babel