FlexOlmo : Open Language Models for Flexible Data Use
Weijia Shi Akshita Bhagia Kevin Farhat Niklas Muennighoff Pete Walsh, Jacob Morrison Dustin Schwenk Shayne Longpre Jake Poznanski, Allyson Ettinger Daogao Liu Margaret Li Dirk Groeneveld Mike Lewis, Wen-tau Yih Luca Soldaini Kyle Lo Noah A. Smith Luke Zettlemoyer, Pang Wei Koh Hannaneh Hajishirzi Al · 2025
Read FlexOlmo : Open Language Models for Flexible Data Use by Weijia Shi Akshita Bhagia Kevin Farhat Niklas Muennighoff Pete Walsh, Jacob Morrison Dustin Schwenk Shayne Longpre Jake Poznanski, Allyson Ettinger Daogao Liu Margaret Li Dirk Groeneveld Mike Lewis, Wen-tau Yih Luca Soldaini Kyle Lo Noah A. Smith Luke Zettlemoyer, Pang Wei Koh Hannaneh Hajishirzi Al (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-07024.txt.
Description
Weijia Shi and co-authors build language models whose data can be switched in and out in a 2025 paper. Owners of private data often cannot share it for joint training, yet everyone wants the gains of pooling. FlexOlmo trains separate expert modules, each on its own closed data set. It then joins them in a mixture-of-experts model. A router, taught which expert fits which domain, needs no joint training on the raw data.
At use time, any expert, and with it its data, can be included or dropped with no retraining, honoring licenses and permissions. Tested at up to thirty-seven billion parameters across thirty-one tasks, adding closed-data experts lifted scores by about forty-one percent on average over the public-data expert alone. The method also beat model merging and matched unrestricted training.
Catalog record
| CREATOR | Weijia Shi Akshita Bhagia Kevin Farhat Niklas Muennighoff Pete Walsh, Jacob Morrison Dustin Schwenk Shayne Longpre Jake Poznanski, Allyson Ettinger Daogao Liu Margaret Li Dirk Groeneveld Mike Lewis, Wen-tau Yih Luca Soldaini Kyle Lo Noah A. Smith Luke Zettlemoyer, Pang Wei Koh Hannaneh Hajishirzi Al |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-07024