Concept Tokens: Learning Behavioral Embeddings Through Concept Definitions
Ignacio Sastre, Aiala Rosá · 2026
Read Concept Tokens: Learning Behavioral Embeddings Through Concept Definitions by Ignacio Sastre, Aiala Rosá (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-04465.txt.
Description
Ignacio Sastre and Aiala Rosá teach a language model a new word for an idea, and nothing else, in a 2026 arXiv paper. Their Concept Tokens method adds one special token to a pretrained model's vocabulary and learns only that token's embedding. From natural-language definitions of a target concept in which the concept's name is replaced by the token. The model itself stays frozen throughout.
Two testbeds show the token genuinely steers behaviour. A token trained on hallucination can be asserted to increase invented answers in closed-book questions, or negated to suppress them, mostly by making the model abstain. A token for recasting, a language teacher's correction technique, shows the same directional control and follows other instructions better than stuffing the definitions into the prompt. A qualitative study with the Eiffel Tower and an invented Austral Tower probes what the embedding has and has not captured.
Catalog record
| CREATOR | Ignacio Sastre, Aiala Rosá |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-04465