Take Off the Training Wheels! Progressive In-Context Learning for Effective Alignment

Zhenyu Liu 1 , Dongfang Li 1 , Xinshuo Hu 1 , Xinping Zhao 1 , Yibin Chen 2 , Baotian Hu 1 🖂 , Min Zhang 1 1 Harbin Institute of Technology (Shenzhen), Shenzhen, China 2 Huawei Cloud, Huawei Technologies Ltd. [email protected] , [email protected] , {lidongfang,hubaotian,zhangmin2021}@hit.ed · 2025

Read Take Off the Training Wheels! Progressive In-Context Learning for Effective Alignment by Zhenyu Liu 1 , Dongfang Li 1 , Xinshuo Hu 1 , Xinping Zhao 1 , Yibin Chen 2 , Baotian Hu 1 🖂 , Min Zhang 1 1 Harbin Institute of Technology (Shenzhen), Shenzhen, China 2 Huawei Cloud, Huawei Technologies Ltd. [email protected] , [email protected] , {lidongfang,hubaotian,zhangmin2021}@hit.ed (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-09958.txt.

Description

Liu and six colleagues propose Progressive In-Context Alignment for language models in twenty twenty-five. In-context learning teaches a model a task from examples in the prompt. How those examples shape token representations was unclear for alignment tasks. The team finds the learned task function is stored in the separator token.

Once the first response tokens are set, the examples become redundant, they show. Their method first runs a few shot stage and extracts that stored vector. A zero shot stage then uses the vector alone to guide generation. No further examples are needed in that stage. Tests beat standard in-context learning and matched tuning based alignment methods. The training free method also cut time cost in their experiments.

Catalog record

CREATORZhenyu Liu 1 , Dongfang Li 1 , Xinshuo Hu 1 , Xinping Zhao 1 , Yibin Chen 2 , Baotian Hu 1 🖂 , Min Zhang 1 1 Harbin Institute of Technology (Shenzhen), Shenzhen, China 2 Huawei Cloud, Huawei Technologies Ltd. [email protected] , [email protected] , {lidongfang,hubaotian,zhangmin2021}@hit.ed
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-09958

Browse the full library · About the Tower of Babel