Skill Neologisms: Towards Skill-based Continual Learning

Antonin Berthon, Nicolas Astorga, Mihaela van der Schaar · 2026

Read Skill Neologisms: Towards Skill-based Continual Learning by Antonin Berthon, Nicolas Astorga, Mihaela van der Schaar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04970.txt.

Description

Antonin Berthon, Nicolas Astorga, and Mihaela van der Schaar explore teaching language models new skills through special soft tokens they call skill neologisms. Adding skills by fine-tuning risks erasing old ones, while prompt-based methods are limited by context length and expressiveness. A learned token, by contrast, can be added to the vocabulary without changing model weights.

The team first shows pretrained models already carry tokens tied to procedural knowledge. On a controlled synthetic task, learned skill tokens improved the target skill and combined with unfamiliar skills, and separately trained tokens composed without further training. A natural-language benchmark confirmed the zero-shot composition, suggesting a scalable path to continual learning.

Catalog record

CREATORAntonin Berthon, Nicolas Astorga, Mihaela van der Schaar
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.04970

Browse the full library · About the Tower of Babel