Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs

Aldo Pareja, Nikhil Shivakumar Nayak, Hao Wang, Krishnateja Killamsetty, Shivchander Sudalairaj, Wenlong Zhao, Seungwook Han, Abhishek Bhandwaldar, Guangxuan Xu, Kai Xu, Ligong Han, Luke Inglis, Akash Srivastava · 2024

Read Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs by Aldo Pareja, Nikhil Shivakumar Nayak, Hao Wang, Krishnateja Killamsetty, Shivchander Sudalairaj, Wenlong Zhao, Seungwook Han, Abhishek Bhandwaldar, Guangxuan Xu, Kai Xu, Ligong Han, Luke Inglis, Akash Srivastava (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13337.txt.

Description

Aldo Pareja and Nikhil Shivakumar Nayak lead a large team on this guide from two thousand twenty-four. It lays out how to fine-tune small language models well, so lone builders and small shops can match what big labs do.

The study runs many set-ups across four open models in the small size range, on instruction data from many fields. Some stock tips fail: phased training from a known recipe adds cost with no gain, while stacked training is plainer and cheaper. Big batches with low learning rates win. Early training marks, such as gradient size and loss, foretell the end score well, so bad runs can be cut early.

Catalog record

CREATORAldo Pareja, Nikhil Shivakumar Nayak, Hao Wang, Krishnateja Killamsetty, Shivchander Sudalairaj, Wenlong Zhao, Seungwook Han, Abhishek Bhandwaldar, Guangxuan Xu, Kai Xu, Ligong Han, Luke Inglis, Akash Srivastava
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE84 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-13337

Browse the full library · About the Tower of Babel