Refusal-Feature-guided Teacher for Safe Finetuning via Data Filtering and Alignment Distillation

Seokil Ham, Yubin Choi, Seungju Cho, Yujin Yang, Younghun Kim, Changick Kim · 2025

Read Refusal-Feature-guided Teacher for Safe Finetuning via Data Filtering and Alignment Distillation by Seokil Ham, Yubin Choi, Seungju Cho, Yujin Yang, Younghun Kim, Changick Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07356.txt.

Description

Seokil Ham and co-authors defend in 2025 a service that fine-tunes language models on customer data without letting safety slip. Such services are open to attack when user data hides harmful prompts, which can train the model to drop its guard. The standard defense starts from an already safety-tuned model, then fine-tunes it.

The authors find that safety-tuned starting weights actually learn the customer's task worse. Their fix keeps a safety-tuned model on the side as a teacher instead. The base model is fine-tuned directly, while the teacher filters harmful prompts out of the data and streams safety lessons in during training. Tests showed fewer harmful outputs and better scores on customer tasks at once. The paper reframes safe fine-tuning as guided learning from a plain start, not as decorating a pre-guarded model.

Catalog record

CREATORSeokil Ham, Yubin Choi, Seungju Cho, Yujin Yang, Younghun Kim, Changick Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-07356

Browse the full library · About the Tower of Babel