Soft Injection of Task Embeddings Outperforms Prompt-Based In-Context Learning

Jungwon Park, Wonjong Rhee · 2025

Read Soft Injection of Task Embeddings Outperforms Prompt-Based In-Context Learning by Jungwon Park, Wonjong Rhee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-20906.txt.

Description

Jungwon Park and co-authors present a method to inject task knowledge into a language model by picking the right attention heads, in work from 2025. One way to adapt a model is to draw a task embedding from a few examples and inject it at run time. Past versions of that idea gave uneven gains over standard tuning or in context learning.

Their method uses gradients to find which attention heads matter for a task. Injection then targets those heads. Across open ended writing, reasoning, and language understanding tasks, the method beat prior embedding methods and few shot learning. It also used far fewer trainable parameters than standard fine tuning.

Catalog record

CREATORJungwon Park, Wonjong Rhee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE94 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-20906

Browse the full library · About the Tower of Babel