Shaping capabilities with token-level data filtering
Neil Rathi, Alec Radford · 2026
Read Shaping capabilities with token-level data filtering by Neil Rathi, Alec Radford (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21571.txt.
Description
Neil Rathi and Alec Radford shape what a language model can do by filtering its training data before training starts. Most attempts to remove unwanted skills happen after training and can be bypassed. Filtering the data itself attacks the skill at its root.
On a test task of removing medical skills, simple filtering proved highly effective. It also held up against attempts to undo it and cost little at large scale. The result suggests data curation is a strong safety lever. Builders could decide early which skills a model should never gain, rather than patching them out later.
Catalog record
| CREATOR | Neil Rathi, Alec Radford |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 83 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-21571