From Parameter Dynamics to Risk Scoring: Quantifying Sample-Level Safety Degradation in LLM Fine-tuning

Xiao Wang, Yifei Zhang †, YongKang Liu, Xiaocui Yang, Zihan Wang, Shi Feng, Daling Wang · 2026

Read From Parameter Dynamics to Risk Scoring: Quantifying Sample-Level Safety Degradation in LLM Fine-tuning by Xiao Wang, Yifei Zhang †, YongKang Liu, Xiaocui Yang, Zihan Wang, Shi Feng, Daling Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04572.txt.

Description

Xiao Wang and six co-authors uncover a mechanism by which fine-tuning a language model on harmless data can quietly erase its safety training. Comparing model weights only before and after training misses the process, so they track how parameters move during fine-tuning itself.

They find benign training pushes parameters cumulatively toward directions aligned with dangerous behaviour, progressively weakening safety. Building on that, they score each training sample by how much its induced update projects onto the danger direction versus the safety direction. Those continuous risk scores let practitioners flag risky samples in advance, turning a vague fragility into something measurable per example.

Catalog record

CREATORXiao Wang, Yifei Zhang †, YongKang Liu, Xiaocui Yang, Zihan Wang, Shi Feng, Daling Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.04572

Browse the full library · About the Tower of Babel