Enhancing Knowledge Distillation for LLMs with Response-Priming Prompting

Vijay Goyal Mustafa Khan Aprameya Tirupati Harveer Saini · 2024

Read Enhancing Knowledge Distillation for LLMs with Response-Priming Prompting by Vijay Goyal Mustafa Khan Aprameya Tirupati Harveer Saini (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17846.txt.

Description

Vijay Goyal and co-authors prime a small language model with better prompts while it learns from a giant one. Their 2024 paper works on knowledge distillation, where a big teacher model's skill is poured into a small student. Past work watched the teacher's answers but paid little mind to the prompts used.

Prompts that draw out reasoning steps were mixed into the training of an 8-billion student, taught by a far larger teacher. On a grade-school math benchmark, one prompt style lifted scores by over half against plain training. Peeks inside the student showed attention patterns that tracked the gains. The team posts its code for others to try on their own small models.

Catalog record

CREATORVijay Goyal Mustafa Khan Aprameya Tirupati Harveer Saini
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-17846

Browse the full library · About the Tower of Babel