Self-Harmony: Learning to Harmonize Self-Supervision and Self-Play in Test-Time Reinforcement Learning

Ru Wang, Wei Huang, Qi Cao, Yusuke Iwasawa, Yutaka Matsuo, Jiaxian Guo, The University of Tokyo, RIKEN Center for Advanced Intelligence Project, Google Research, Australia {ru.wang, qi.cao}@[email protected]@google.com · 2025

Read Self-Harmony: Learning to Harmonize Self-Supervision and Self-Play in Test-Time Reinforcement Learning by Ru Wang, Wei Huang, Qi Cao, Yusuke Iwasawa, Yutaka Matsuo, Jiaxian Guo, The University of Tokyo, RIKEN Center for Advanced Intelligence Project, Google Research, Australia {ru.wang, qi.cao}@[email protected]@google.com (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01191.txt.

Description

Ru Wang and colleagues improve test time reinforcement learning, a label free way to adapt a language model at inference using only signals it generates itself. The catch is building a trustworthy signal: majority voting, the standard trick, often crowns answers that are popular but wrong.

Their framework, Self-Harmony, rests on a simple test: a correct answer should survive a rephrasing of the question. One model plays two roles: a solver that answers and a reframer that paraphrases. Candidate answers are pooled across the original and reframed views using the harmonic mean. That favors answers stable under reframing. No human labels or helper models are needed. Across reasoning benchmarks the method ranked first in nearly all settings tried, with no training failures.

Catalog record

CREATORRu Wang, Wei Huang, Qi Cao, Yusuke Iwasawa, Yutaka Matsuo, Jiaxian Guo, The University of Tokyo, RIKEN Center for Advanced Intelligence Project, Google Research, Australia {ru.wang, qi.cao}@[email protected]@google.com
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-01191

Browse the full library · About the Tower of Babel