Return of the Encoder: Maximizing Parameter Efficiency for SLMs
Mohamed Elfeki, Rui Liu, Chad Voegele · 2025
Read Return of the Encoder: Maximizing Parameter Efficiency for SLMs by Mohamed Elfeki, Rui Liu, Chad Voegele (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-16273.txt.
Description
Mohamed Elfeki, Rui Liu, and Chad Voegele make the case for bringing back the two-part form of language model for small models. Big one-part models rule the field, yet they spend much at each step. The two-part form reads the input once, then writes, which should cost less. The team timed both forms on three kinds of chips used in small gear.
For models of a billion parts or less, the two-part form cut the wait for the first word by near half. It also pushed through more than four times the work on edge gear. A new way to pour skill from big one-part teachers into small two-part students won back about six score points on the whole. With fresh tricks added, the old form looks like the sane path for models that must run on the gear itself.
Catalog record
| CREATOR | Mohamed Elfeki, Rui Liu, Chad Voegele |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-16273