Ultra-Low-Dimensional Prompt Tuning via Random Projection

Zijun Wu, Yongchang Hao, Lili Mou * · 2025

Read Ultra-Low-Dimensional Prompt Tuning via Random Projection by Zijun Wu, Yongchang Hao, Lili Mou * (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-04501.txt.

Description

Zijun Wu, Yongchang Hao, and Lili Mou propose, in a 2025 paper, a way to tune language models with prompts that live in as few as two dimensions. Prompt tuning learns small prompt vectors instead of retraining a giant model. Yet standard prompts match the model's wide inner size, which caps the savings.

Their method learns the prompt in a tiny space, then lifts it with a frozen random matrix into the model's space. That cut training parameters by 98 percent against plain prompt tuning, with performance held. Tests spanned more than 20 language tasks. The tiny prompts beat recent lean tuning rivals while using far fewer stored values. Shops that host many custom models could store each variant for almost nothing.

Catalog record

CREATORZijun Wu, Yongchang Hao, Lili Mou *
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-04501

Browse the full library · About the Tower of Babel