Llama 3 Meets MoE: Efficient Upcycling
Aditya Vavre, Ethan He, Dennis Liu, Zijie Yan, June Yang, Nima Tajbakhsh, Ashwath Aithal · 2024
Read Llama 3 Meets MoE: Efficient Upcycling by Aditya Vavre, Ethan He, Dennis Liu, Zijie Yan, June Yang, Nima Tajbakhsh, Ashwath Aithal (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-09952.txt.
Description
Aditya Vavre and co-authors convert a trained language model in 2024 into a leaner expert-mixture model at a sliver of normal cost. Mixture models route each word to a few expert sub-nets, gaining size without matching compute. Training them fresh is unstable and prone to overfit, so the team starts from a finished standard model instead.
Their build grows an eight-expert model from a well-known 8-billion-parameter base, using under one percent of typical training compute, they report. On a broad knowledge test the converted model gained two points over its base in zero-shot use. Training also ran fast for its class. The trick, called upcycling, is now built into a major toolkit, so labs can grow expert models from models they already own.
Catalog record
| CREATOR | Aditya Vavre, Ethan He, Dennis Liu, Zijie Yan, June Yang, Nima Tajbakhsh, Ashwath Aithal |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-09952