Shaping capabilities with token-level data filtering

Neil Rathi, Alec Radford · 2026

Read Shaping capabilities with token-level data filtering by Neil Rathi, Alec Radford (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21571.txt.

Description

Neil Rathi and Alec Radford shape what a language model can do by filtering its training data before training starts. Most attempts to remove unwanted skills happen after training and can be bypassed. Filtering the data itself attacks the skill at its root.

On a test task of removing medical skills, simple filtering proved highly effective. It also held up against attempts to undo it and cost little at large scale. The result suggests data curation is a strong safety lever. Builders could decide early which skills a model should never gain, rather than patching them out later.

Catalog record

CREATORNeil Rathi, Alec Radford
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21571

Browse the full library · About the Tower of Babel