Phi-4 Technical Report
Marah Abdin Jyoti Aneja Harkirat Behl Sébastien Bubeck Ronen Eldan Suriya Gunasekar Michael Harrison Russell J. Hewett Mojan Javaheripi Piero Kauffmann James R. Lee Yin Tat Lee Yuanzhi Li Weishung Liu Caio C. T. Mendes Anh Nguyen Eric Price Gustavo de Rosa Olli Saarikivi Adil Salim Shital Shah Xin W · 2024
Read Phi-4 Technical Report by Marah Abdin Jyoti Aneja Harkirat Behl Sébastien Bubeck Ronen Eldan Suriya Gunasekar Michael Harrison Russell J. Hewett Mojan Javaheripi Piero Kauffmann James R. Lee Yin Tat Lee Yuanzhi Li Weishung Liu Caio C. T. Mendes Anh Nguyen Eric Price Gustavo de Rosa Olli Saarikivi Adil Salim Shital Shah Xin W (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08905.txt.
Introduction
Carefully made fake data, more than raw web scale, powers a compact language model in the Phi-4 technical report from 2024. The model holds fourteen billion parameters and comes from a large author team. Synthetic lessons run through every training stage rather than serving as a top-up.
Earlier Phi models mostly distilled a big teacher's skill. Phi-4 beats its teacher on science and math question sets, which hints the recipe adds more than copying. Architecture barely changed from its predecessor, so gains trace to data and training order. The report makes a case that data quality can substitute for size in reasoning models.
Catalog record
| CREATOR | Marah Abdin Jyoti Aneja Harkirat Behl Sébastien Bubeck Ronen Eldan Suriya Gunasekar Michael Harrison Russell J. Hewett Mojan Javaheripi Piero Kauffmann James R. Lee Yin Tat Lee Yuanzhi Li Weishung Liu Caio C. T. Mendes Anh Nguyen Eric Price Gustavo de Rosa Olli Saarikivi Adil Salim Shital Shah Xin W |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 88 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-08905
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Phi-4 Technical Report Marah Abdin Jyoti Aneja Harkirat Behl Sébastien Bubeck Ronen Eldan Suriya Gunasekar Michael Harrison Russell J. Hewett Mojan Javaheripi Piero Kauffmann James R. Lee Yin Tat Lee Yuanzhi Li Weishung Liu Caio C. T. Mendes Anh Nguyen Eric Price Gustavo de Rosa Olli Saarikivi Adil Salim Shital Shah Xin Wang Rachel Ward Yue Wu Dingli Yu Cyril Zhang Yi Zhang Abstract We present phi-4 , a 14-billion parameter language model developed with a training recipe that is centrally focused on data quality. Unlike most language models, where pre-training is based primarily on organic data sources such as web content or code, phi-4 strategically incorporates synthetic data throughout the training process. While previous models in the Phi family largely distill the capabilities of a teacher model (specifically GPT-4), phi-4 substantially surpasses its teacher model on STEM-focused QA capabilities, giving evidence that our data-generation and post-training techniques go beyond distillation. Despite minimal changes to the phi-3 architecture, phi-4 achieves strong performance relative to its size – especially on reasoning-focused benchmarks – due to improved data, training curriculum, and innovations in the post-training scheme. 1 Introduction Recent advancements in Large Language Models (LLMs) have shown that significant improvements in data quality can rival, and sometimes surpass, the performance gains traditionally achieved by scaling compute with model and dataset size. Building on the success of the Phi family [ 10 , 18 , 14 , 1 ] , we introduce phi-4 , a 14-billion parameter model that further advances performance of small language models by introducing innovative synthetic data generation methods for reasoning-focused tasks, by optimizing the training curriculum and data mixture, and by introducing new techniques in post-training. Synthetic data constitutes the bulk of the training data for phi-4 and is generated using a diverse array of techniques, including multi-agent prompting, self-revision workflows, and instruction reversal. These methods enable the construction of datasets that induce stronger reasoning and problem-solving abilities in the model, addressing some of the weaknesses in traditional unsupervised datasets. Synthetic data in phi-4 also plays a crucial role in post-training, where techniques such as rejection sampling and a novel approach to Direct Preference Optimization (DPO) are employed to refine the model’s outputs. The development of phi-4 is guided by three core pillars: 1. Synthetic Data for Pretraining and Midtraining: High-quality synthetic datasets are designed to prioritize reasoning and problem-solving , carefully generated to ensure diversity and relevance. We change our training curriculum and create new pretraining and midtraining data mixtures to increase the allocation of synthetic tokens, compared to older generations of phi . 2. Curation and Filtering of High-Quality Organic Data: We meticulously curate and filter organic 2 2 2 We use organic to refer to human-generated or otherwise non-synthetic data. data sources, including web content, licensed books, and code repositories to extract seeds for the synthetic data pipeline that encourage high-depth reasoning and prioritize educational value (to the model). These seeds form the foundation of the synthetic generation pipeline. To complement these synthetic datasets, we also filter the web for high-quality data (in terms of knowledge and reasoning) to use directly in pretraining. 3. Post-Training: We further advance the post-training recipe in phi-4 by creating new refined versions of SFT datasets, as well as by developing a new technique to create DPO pairs, based on pivotal token search . With these innovations, the performance of phi-4 on reasoning-related tasks is comparable to or surpasses much larger models. For example, its performance on many widely used reasoning-related benchmarks meets or exceeds that of Llama-3.1-405B . In Table 1 we compare the performance of our model on academic benchmarks to several contemporary foundation models. We find that phi-4 significantly exceeds its teacher GPT-4o on the GPQA (graduate-level STEM Q&A) and MATH (math competition) benchmarks. Small models Large models phi-4 14b phi-3 14b Qwen 2.5 14b instruct GPT 4o-mini Llama-3.3 70b instruct Qwen 2.5 72b instruct GPT 4o simple-evals MMLU 84.8 77.9 79.9 81.8 86.3 85.3 88.1 GPQA 56.1 31.2 42.9 40.9 49.1 49.0 50.6 MATH 80.4 44.6 75.6 73.0 66.3 1 1 1 These scores are lower than those reported by Meta, perhaps because simple-evals has a strict formatting requirement that Llama models have particular trouble following. We use the simple-evals framework because it is reproducible, but Meta reports 77 for MATH and 88 for HumanEval on Llama-3.3. 80.0 74.6 HumanEval 82.6 67.8 72.1 86.2 78.9 1 1 footnotemark: 1 80.4 90.6 MGSM 80.6 53.5 79.6 86.5 89.1 87.3 90.4 SimpleQA 3.0 7.6 5.4 9.9 20.9 10.2 39.4 DROP 75.5 68.3 85.5 79.3 90.2 76.7 80.9 MMLUPro 70.4 51.3 63.2 63.4 64.4 69.6 73.0 HumanEval+ 82.8 69.2 79.1 82.0 77.9 78.4 88.0 ArenaHard 75.4 45.8 70.2 76.2 65.5 78.4 75.6 LiveBench 47.6 28.1 46.6 48.1 57.6 55.3 57.6 IFEval 63.0 57.9 78.7 80.0 89.3 85.0 84.8 PhiBench (internal) 56.2 43.9 49.8 58.7 57.1 64.6 72.4 Table 1: Performance of phi-4 on a set of standard benchmarks. The first set of benchmarks uses OpenAI’s simple-evals framework [ 24 ] , specifying the prompts/extraction/temperature=0.5. We compare to small models of similar inference cost, as well as to larger models. 1.1 Addressing Overfitting and Data Contamination Decontamination: One pitfall of foundation models is overfitting to benchmarks, such as through the leakage of benchmark test sets via the web corpus. We improved the data decontamination process for phi-4 compared to previous Phi models to ensure no unfair influence on evaluation results. More details of the decontamination method are given in Appendix B . AMC Benchmark: The surest way to guard against overfitting to the test set is to test on fresh data. We tested our model on the November 2024 AMC-10 and AMC-12 math competitions [ 6 ] , which occurred after all our training data was collected, and we only measured our performance after choosing all the hyperparameters in training our final model. These contests are the entry points to the Math Olympiad track in the United States and over 150,000 students take the tests each year. Figure 1 : Average performance of different models on the November 2024 AMC-10 and AMC-12 tests. This is the average score (with maximum score 150) over the four tests on 100 runs with temperature t=0.5 . We chose t=0.5 to follow simple-evals [ 24 ] . Error bars are 2\sigma of the estimate. On competition math, phi-4 scores well above its weight-class even compared to non–open-weight models. In Figure 1 we plot the average score over the four versions of the test, all of which have a maximum score of 150. phi-4 outperforms not only similar-size or open-weight models but also much larger frontier models. Such strong performance on a fresh test set suggests that phi-4 ’s top-tier performance on the MATH benchmark is not due to overfitting or contamination. We provide further details in Appendix C . Relying on Contamination-Proof Benchmarks: We give significant weight to benchmarks which were designed in such a way that the questions are original and do not appear on the web, such as GPQA [ 25 ] . While optimizing our model, we relied on an internal benchmark composed primarily of original prompts written by the team (see Section 5 for further details). Long Chain-of-Thought Models: A