Apple Intelligence Foundation Language Models
Apple · 2025
Read Apple Intelligence Foundation Language Models by Apple (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-13575.txt.
Introduction
Apple's twenty twenty-five technical report documents the foundation language models behind Apple Intelligence, the company's AI feature set. The hosted text opens with the title and a byline of well over a hundred named contributors, reflecting an industrial-scale model program rather than a small research team. The report is Apple's primary public record of its foundation models for that year. Readers should note the catalog text begins with front matter; detailed content sits deeper in the full report.
Catalog record
| CREATOR | Apple |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-13575
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Apple Intelligence Foundation Language Models Apple Abstract \vskip-20.0pt We introduce two multilingual, multimodal foundation language models that power Apple Intelligence features across Apple devices and services: (i) a ∼ \sim 3B-parameter on-device model optimized for Apple silicon through architectural innovations such as KV-cache sharing and 2-bit quantization-aware training; and (ii) a scalable server model built on a novel Parallel-Track Mixture-of-Experts (PT-MoE) transformer that combines track parallelism, mixture-of-experts sparse computation, and interleaved global–local attention to deliver high quality with competitive cost on Apple’s Private Cloud Compute platform. Both models are trained on large-scale multilingual and multimodal datasets sourced via responsible web crawling, licensed corpora, and high-quality synthetic data, then further refined with supervised fine-tuning and reinforcement learning on a new asynchronous platform. The resulting models support several additional languages while understanding images and executing tool calls. In public benchmarks and human evaluations, both the server model and the on-device model match or surpass comparably sized open baselines. A new Swift-centric Foundation Models framework exposes guided generation, constrained tool calling, and LoRA adapter fine-tuning, allowing developers to integrate these capabilities with a few lines of code. The latest advancements in Apple Intelligence models are grounded in our Responsible AI approach with safeguards like content filtering and locale-specific evaluation, as well as our commitment to protecting our users’ privacy with innovations like Private Cloud Compute. 1 Introduction Apple Intelligence integrates powerful generative AI right into the apps and experiences users turn to everyday, all while protecting their privacy. At the 2025 Worldwide Developers Conference we introduced a wide range of new Apple Intelligence features that help users communicate, express themselves, and achieve their goals. To power these features, we have created a new generation of language foundation models specifically developed to enhance the Apple Intelligence features in our latest software releases. We also introduced the new Foundation Models framework, which gives app developers direct access to the on-device language foundation model at the core of Apple Intelligence. We crafted these generative models to power the wide range of intelligent features integrated across our platforms. The models have improved tool-use and reasoning capabilities, understand image and text inputs, are faster and more efficient, and are designed to support 16 different languages. Our latest foundation models include a compact, approximately 3-billion-parameter model optimized to run efficiently on Apple silicon, alongside a mixture-of-expert server-based model with a novel architecture tailored for Private Cloud Compute [ Apple 2024 ] . These two foundation models are part of a larger family of generative models created by Apple to support our users. In this overview, we detail the data we used for training, the architectures of the models we designed, the training recipes we employed, the techniques we used to optimize inference, and our evaluation results when compared to similar models. Throughout, we highlight how we achieved an expansion of capabilities and quality improvements for the benefits of our users while also increasing the speed and efficiency of on-device and Private Cloud Compute. Finally, in our continued commitment to uphold our core values, we illustrate how Responsible AI principles are integrated throughout the entire model development process. Figure 1: Modeling overview for the Apple foundation models. 2 Model Architectures We developed both the on-device and server models to meet a wide range of performance and deployment needs. The on-device model is optimized for efficiency, enabling low-latency inference with minimal resource usage, while the server model is designed to deliver high accuracy and scalability for more complex tasks. Together, they form a complementary suite of solutions adaptable to diverse application scenarios. We first describe the general decoder architectures for both models, then explain how we extended them to support image inputs with vision encoders. 2.1 On-Device Model We incorporated the following improvements into the on-device model, resulting in a significant reduction in inference latency and improved efficiency. KV Cache Sharing. To improve the efficiency of our on-device architecture, we divided the full model into two blocks. Block 1 contained 62.5% of the total transformer layers, while Block 2 contained the remaining 37.5% of the transformer layers, but had the key and value projections removed. Every key and value cache (KV cache) of Block 2 was directly shared with those generated by Block 1 [ Snowflake 2025 ] , reducing KV cache memory usage by 37.5%. Additionally, because Block 2 does not produce any keys or values, the prefill stage is able to bypass all of its computation [ Sun et al. 2024 ] and reduce time-to-first-token (TTFT) by \sim 37.5%. 2.2 Server Model We also made several major changes to improve the capacity and efficiency of the server model. Parallel Track (PT) Transformer. We introduced a new architecture, the Parallel Track (PT) Transformer. Unlike the standard decoder-only transformer, which consists of a single sequential stack of layers, the PT-transformer partitions the model into multiple smaller transformers, referred to as tracks. Each track consists of multiple stacked track blocks, each of which is a Transformer layer stack. The track blocks process tokens independently, with synchronization across tracks applied only at the input and output boundaries of each track block. This isolated design enables straightforward parallel execution across tracks and reduces the synchronization overhead found in conventional transformer decoders, such as those using tensor parallelism. We refer to this approach as track parallelism. This improved training and inference latency without compromising model quality. Parallel Track Mixture of Experts (PT-MoE). To further scale the server-side model, we introduced Mixture-of-Experts (MoE) layers [ Shazeer et al. 2017 ; Du et al. 2022 ; Lepikhin et al. 2021 ; Zoph et al. 2022 ] within each track block, leading to the PT-MoE architecture. Since the experts in each MoE layer are local to their respective tracks, the communication overhead can be more effectively overlapped with the computation to improve training efficiency. Combined with track-level independence enabled by track parallelism, this design allowed the model to scale efficiently while maintaining low latency due to increased sparsity. Specifically, we replaced the dense feed-forward network in every other transformer layer with a MoE layer where the top-k routing is implemented via the grouped general matrix multiplication (GEMM) [ Gale et al. 2022 ] , ensuring that no tokens are dropped during routing. Figure 2 provides an overview of the PT-MoE architecture. Figure 2: Diagram of the PT-MoE architecture. Each track is composed of multiple track blocks, and each track block contains a fixed number of transformer/MoE layers. Assume that we have a total of L layers and track block of depth D , then we reduce the synchronization overhead from 2L (tensor parallelism) to L/D (track parallelism). For example, if D=4 , the PT reduces 87.5\% of the synchronization overhead. Interleaving Global and Local Attention Layers. We designed an interleaved attention architecture that alternates between local and global attention layers to support long sequences efficiently [ Beltagy et al. 2020 ] . Each repeating transformer block consists of three local attention layers using a sliding window of size 4096 and rotational positional embeddings (RoPE), followed by a global attention layer without positional embeddings (NoPE) [ Yang et al. 2025 ] . Omitting positional embeddings in the global attention layer allows better length generalization by avoiding out-of-distribution position issues