Free Energy-Driven Reinforcement Learning with Adaptive Advantage Shaping for Unsupervised Reasoning in LLMs
Yiming Huang, Zhenbo Shi, Xincheng Wen, Jichuan Zeng, Cuiyun Gao, Peiyi Han, Chuanyi Liu · 2026
Read Free Energy-Driven Reinforcement Learning with Adaptive Advantage Shaping for Unsupervised Reasoning in LLMs by Yiming Huang, Zhenbo Shi, Xincheng Wen, Jichuan Zeng, Cuiyun Gao, Peiyi Han, Chuanyi Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04065.txt.
Description
Yiming Huang and six co-authors introduce FREIA, an unsupervised reinforcement learning algorithm that lets large language models improve their reasoning without ground-truth labels. Existing unsupervised methods often cannot adapt to the model's evolving reasoning ability during training, so they can misdirect policy updates. FREIA is built from two ideas aimed at that gap.
Its Free Energy-Driven Reward adapts rewards to balance consensus among sampled answers against exploration, drawing on the Free Energy Principle. Its Adaptive Advantage Shaping adjusts learning signals according to the shape of the sampled rewards. Across nine datasets and three reasoning tasks, FREIA outperformed other unsupervised reinforcement learning baselines. On a distilled reasoning model, mathematical reasoning gains ran from roughly half a point to several points in Pass@1.
Catalog record
| CREATOR | Yiming Huang, Zhenbo Shi, Xincheng Wen, Jichuan Zeng, Cuiyun Gao, Peiyi Han, Chuanyi Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.04065