Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language Models

Kyeonghyun Kim, Jinhee Jang, Juhwan Choi, Yoonji Lee, Kyohoon Jin, YoungBin Kim · 2025

Read Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language Models by Kyeonghyun Kim, Jinhee Jang, Juhwan Choi, Yoonji Lee, Kyohoon Jin, YoungBin Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07424.txt.

Description

Kyeonghyun Kim and co-authors present in 2025 a way to lend a small language model some of a big model's skill. Big models know more and generalize better, but they cost too much for tight settings. Small models are cheap yet narrower. Their frame, PiFi, plugs one frozen layer from a big model into a small one.

Only the combined model is then fine-tuned for the task at hand. Scores rose across a range of language tasks, both understanding and generation, with little added cost. The borrowed layer also helped the small model handle topics it had never seen, which shows that big-model knowledge can transfer through a single layer.

Catalog record

CREATORKyeonghyun Kim, Jinhee Jang, Juhwan Choi, Yoonji Lee, Kyohoon Jin, YoungBin Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-07424

Browse the full library · About the Tower of Babel