Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language Models
Kyeonghyun Kim, Jinhee Jang, Juhwan Choi, Yoonji Lee, Kyohoon Jin, YoungBin Kim · 2025
Read Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language Models by Kyeonghyun Kim, Jinhee Jang, Juhwan Choi, Yoonji Lee, Kyohoon Jin, YoungBin Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07424.txt.
Description
Kyeonghyun Kim and co-authors present in 2025 a way to lend a small language model some of a big model's skill. Big models know more and generalize better, but they cost too much for tight settings. Small models are cheap yet narrower. Their frame, PiFi, plugs one frozen layer from a big model into a small one.
Only the combined model is then fine-tuned for the task at hand. Scores rose across a range of language tasks, both understanding and generation, with little added cost. The borrowed layer also helped the small model handle topics it had never seen, which shows that big-model knowledge can transfer through a single layer.
Catalog record
| CREATOR | Kyeonghyun Kim, Jinhee Jang, Juhwan Choi, Yoonji Lee, Kyohoon Jin, YoungBin Kim |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-07424