Robust Multi-Objective Controlled Decoding of Large Language Models

Seongho Son, William Bankes *, Sangwoong Yoon *, Shyam Sundhar Ramesh *, Xiaohang Tang, Ilija Bogunovic · 2025

Read Robust Multi-Objective Controlled Decoding of Large Language Models by Seongho Son, William Bankes *, Sangwoong Yoon *, Shyam Sundhar Ramesh *, Xiaohang Tang, Ilija Bogunovic (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08796.txt.

Description

Son and a team of colleagues introduced a decoding method in twenty twenty-five that aligns language models to many human goals at once. Goals such as helpfulness and safety can conflict. Standard controlled decoding balances average rewards, which can leave the worst goal poorly served.

Their method, RMOD, instead maximizes the worst-case reward across goals. They frame the problem as a two-player game between hostile reward weights and the sampling policy, solved at a Nash equilibrium. That game reduces to a convex problem for the worst weights, with the best policy derived directly. A practical version adds little computing cost over standard decoding. Tests with up to ten objectives on alignment datasets beat baselines on worst-case rewards and win rates, they report.

Catalog record

CREATORSeongho Son, William Bankes *, Sangwoong Yoon *, Shyam Sundhar Ramesh *, Xiaohang Tang, Ilija Bogunovic
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-08796

Browse the full library · About the Tower of Babel