Semantic-aware Wasserstein Policy Regularization for Large Language Model Alignment

Byeonghu Na, Hyungho Na, Yeongmin Kim, Suhyeon Jo, HeeSun Bae, Mina Kang, Il-Chul Moon · 2026

Read Semantic-aware Wasserstein Policy Regularization for Large Language Model Alignment by Byeonghu Na, Hyungho Na, Yeongmin Kim, Suhyeon Jo, HeeSun Bae, Mina Kang, Il-Chul Moon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-01685.txt.

Description

Byeonghu Na and colleagues align language models with human preferences by fixing a gap in training. Training seeks reward under KL divergence penalty, but KL and f-divergence variants compare token chances at same indices, missing semantic similarity. Their Wasserstein Policy Regularization uses the entropy Wasserstein distance, which uses token space geometry. Its dual form turns the penalty into reward terms through dual variables, giving a usable objective for RL algorithms. Tests show it beats KL and f-divergence baselines, showing the value of semantic policy distances for alignment.

Catalog record

CREATORByeonghu Na, Hyungho Na, Yeongmin Kim, Suhyeon Jo, HeeSun Bae, Mina Kang, Il-Chul Moon
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76965
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602.01685

Browse the full library · About the Tower of Babel