Return of the Encoder: Maximizing Parameter Efficiency for SLMs

Mohamed Elfeki, Rui Liu, Chad Voegele · 2025

Read Return of the Encoder: Maximizing Parameter Efficiency for SLMs by Mohamed Elfeki, Rui Liu, Chad Voegele (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-16273.txt.

Description

Mohamed Elfeki, Rui Liu, and Chad Voegele make the case for bringing back the two-part form of language model for small models. Big one-part models rule the field, yet they spend much at each step. The two-part form reads the input once, then writes, which should cost less. The team timed both forms on three kinds of chips used in small gear.

For models of a billion parts or less, the two-part form cut the wait for the first word by near half. It also pushed through more than four times the work on edge gear. A new way to pour skill from big one-part teachers into small two-part students won back about six score points on the whole. With fresh tricks added, the old form looks like the sane path for models that must run on the gear itself.

Catalog record

CREATORMohamed Elfeki, Rui Liu, Chad Voegele
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-16273

Browse the full library · About the Tower of Babel