Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs

Microsoft · 2025

Read Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs by Microsoft (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01743.txt.

Description

A large Microsoft team introduces compact language models in a 2025 technical report. The text model holds under four billion parameters yet beats similar-sized open models and matches models twice its size on math and coding tasks. The secret is a curated diet of high-quality web and synthetic data, heavy on math and code. A much larger vocabulary aids many languages, and a grouped attention scheme speeds long outputs.

Its sibling adds vision and speech to the same core. Small adapter modules with routers let the modes combine without interfering. The report details the recipes and benchmarks for both models.

Catalog record

CREATORMicrosoft
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-01743

Browse the full library · About the Tower of Babel