Llama 3 Meets MoE: Efficient Upcycling

Aditya Vavre, Ethan He, Dennis Liu, Zijie Yan, June Yang, Nima Tajbakhsh, Ashwath Aithal · 2024

Read Llama 3 Meets MoE: Efficient Upcycling by Aditya Vavre, Ethan He, Dennis Liu, Zijie Yan, June Yang, Nima Tajbakhsh, Ashwath Aithal (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-09952.txt.

Description

Aditya Vavre and co-authors convert a trained language model in 2024 into a leaner expert-mixture model at a sliver of normal cost. Mixture models route each word to a few expert sub-nets, gaining size without matching compute. Training them fresh is unstable and prone to overfit, so the team starts from a finished standard model instead.

Their build grows an eight-expert model from a well-known 8-billion-parameter base, using under one percent of typical training compute, they report. On a broad knowledge test the converted model gained two points over its base in zero-shot use. Training also ran fast for its class. The trick, called upcycling, is now built into a major toolkit, so labs can grow expert models from models they already own.

Catalog record

CREATORAditya Vavre, Ethan He, Dennis Liu, Zijie Yan, June Yang, Nima Tajbakhsh, Ashwath Aithal
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-09952

Browse the full library · About the Tower of Babel