Intern-S1-Pro: Scientific Multimodal Foundation Model at Trillion Scale

Intern-S1-Pro Team, Shanghai AI Laboratory · 2026

Read Intern-S1-Pro: Scientific Multimodal Foundation Model at Trillion Scale by Intern-S1-Pro Team, Shanghai AI Laboratory (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-25040.txt.

Introduction

Zou and co-authors introduce Intern-S1-Pro, a scientific foundation model with one trillion parameters. The model reads both images and text, and it is tuned for science work. Its makers say it handles over 100 expert tasks across chemistry, materials, life science, and Earth science. It also gains stronger general reasoning and agent skills. Training at this scale rests on the XTuner and LMDeploy toolkits, which keep reinforcement learning fast and exact. The team calls it a specializable generalist.

Catalog record

CREATORIntern-S1-Pro Team, Shanghai AI Laboratory
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50941
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.25040

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Intern-S1-Pro: Scientific Multimodal Foundation Model at Trillion Scale Intern-S1-Pro Team, Shanghai AI Laboratory Abstract We introduce Intern-S1-Pro, the first one-trillion-parameter scientific multimodal foundation model. Scaling to this unprecedented size, the model delivers a comprehensive enhancement across both general and scientific domains. Beyond stronger reasoning and image-text understanding capabilities, its intelligence is augmented with advanced agent capabilities. Simultaneously, its scientific expertise has been vastly expanded to master over 100 specialized tasks across critical science fields, including chemistry, materials, life sciences, and earth sciences. Achieving this massive scale is made possible by the robust infrastructure support of XTuner and LMDeploy, which facilitates highly efficient Reinforcement Learning (RL) training at the 1-trillion parameter level while ensuring strict precision consistency between training and inference. By seamlessly integrating these advancements, Intern-S1-Pro further fortifies the fusion of general and specialized intelligence, working as a Specializable Generalist, demonstrating its position in the top tier of open-source models for general capabilities, while outperforming proprietary models in the depth of specialized scientific tasks. † † * Model is available at https://huggingface.co/internlm/Intern-S1-Pro 1 Introduction The advent of Large Language Models (LLMs) and Visual Language Models (VLMs) has fundamentally transformed the landscape of artificial intelligence, offering unprecedented capabilities in reasoning, generation, and multimodal understanding [ 1 , 18 ] . In the domain of AI for Science (AI4S), these foundation models have emerged as critical tools for accelerating scientific discovery, enabling researchers to tackle complex problems ranging from protein structure prediction to materials design [ 51 , 38 , 26 ] . Large Models serve as a unified interface for processing vast amounts of scientific literature, experimental data, and domain-specific knowledge, thereby bridging the gap between disparate scientific disciplines [ 35 ] . To build an effective scientific foundation model, scaling model size is imperative due to the immense diversity inherent in scientific domains. Compared to natural language, science encompasses much more specialized fields, such as chemistry, biology, physics, and earth sciences, each with its own unique “language", including domain-specific notations, knowledge, and reasoning patterns. This diversity grows with diving into the frontier science since they often involve long-tailed knowledge and specialized skills. Previous works in multilingual machine translation has founded that a single model requires more parameters when asking it to translate more language pairs, such as the model size for hundreds of language pairs is 90x larger compared to a bilingual model [ 30 ] . A scientific foundation model should possess sufficient capacity to master a wide array of scientific tasks while retaining general text and vision capabilities. In this work, we introduce Intern-S1-Pro , the first one-trillion-parameter scientific multimodal foundation model. Scaling to this unprecedented size, Intern-S1-Pro delivers a comprehensive enhancement across both general and scientific domains. Beyond stronger reasoning and image-text understanding capabilities, its intelligence is augmented with advanced agent capabilities, enabling it to autonomously plan and execute complex scientific workflows. Simultaneously, its scientific expertise has been vastly expanded to master over 100 specialized tasks across critical science fields, including chemistry, materials, life sciences, and earth sciences. Figure 1 : The SAGE (Synergistic Architecture for Generalizable Experts, including three layers, Foundation, Fusion, and Evolution) framework used in Intern-S1-Pro development, illustrating the core capabilities and the integrated learning process that enables synergistic improvements across domains. Following the three layers design in SAGE framework (shown in Figure 1 ), we demonstrate that Intern-S1-Pro, through joint training on general and specific tasks, can outperform specialized models in several scientific tasks. Contrary to the common belief that specialized models are superior for niche tasks, our findings reveal that a sufficiently large generalist model, when trained jointly, can achieve superior performance. In Section 5.5 , we will show in detail that even when using similar training data to specialized models, utilizing a larger model architecture and a joint training strategy yields significant performance gains, validating the effectiveness of our approach. Scaling model parameters introduces new challenges, and we list two architecture-related problems here: training instability due to load imbalance among a massive number of experts, and the difficulty in sufficiently optimizing router embeddings. Extreme imbalance of experts can cause memory spikes, and the conventional solution is to adopt a robust but slower parallelism strategy. To preserve both the stability and the efficiency, we propose a group routing mechanism that enforces a lower bound on expert load balance. Additionally, while we initialize experts from Intern-S1 to ensure a strong starting point of experts’ (the Fully Forward Network part) weights, the router embeddings require efficient learning to handle the expanded expert pool; thus, we introduce a gradient estimation scheme to accelerate their update frequency. On the engineering front, maintaining high training throughput is critical. Through the co-design of algorithms and infrastructure, we achieve deep optimization between the XTuner training framework and the LMDeploy inference engine. This synergy allows Intern-S1-Pro to scale to 4\times the size of its predecessor (Intern-S1) while incurring only a \sim 20\% reduction in training efficiency. This demonstrates that with careful system-level optimizations, massive-scale training can remain highly efficient. Meanwhile, the robust and optimized infrastructure facilitates highly efficient RL training at the 1-trillion parameter level while ensuring strict precision consistency between training and inference. 2 Architecture Figure 2 : Left: Illustration of the expert expansion process from Intern-S1 to Intern-S1-Pro. The grouped routing strategy ensures well-trained Top-1/Top-2 experts are distributed across groups to maintain training stability. Right: The illustration of Straight-Through Estimator for enabling updating all router embeddings in every pass. Intern-S1-Pro is derived from Intern-S1 through expert expansion, as illustrated in Figure 2 . In this expansion process, we incorporate the Grouped Routing design, where experts are distributed into groups. We ensure that the experts activated within each group correspond to the Top-1 or Top-2 experts prior to expansion. While this approach results in some homogenization of expert activation during the initialization phase, the experts naturally differentiate after a few step training, and this design significantly enhances training stability. In contrast, assigning differentiated experts corresponding to the pre-expansion Top-1 to Top-8 across groups leads to training instability and performance degradation. For example, we tested two initialization methods on a 30BA3 model and 2000 training steps. The first method can slight outperform the model prior to expansion while the second method have a performance drop over 20pts. Our hypothesis is that the experts that often activated as top-1 selection suggest they are well-trained and important modules, so keeping each group has well-trained experts are essential to the initialization. 2.1 Group Routing Figure 3 : Training with Grouped Router can achieve absolute load balancing across devices for MoE models with a Top-k configuration of k=8 under the EP8 training strategy. For the training of ultra-large-scale MoE models (e.g., Intern-S1-Pro), Expert Parallelism (EP) serves as the core technical approach to mitigate GPU memory and communication overheads. However, the expert load imbalance caused by the traditional Top- k routing strategy will lead to cross-device load imbalance during the expert parallel training process. Although a lower degree of expert parallelism (e.g., EP8) and the MoE Balance Loss can alleviate this issue, the phenomenon still persists, and is particularly severe in the post-training phase of large models. This not only significantly degrades the training efficiency of expert parallelism but also causes an Out-of-Memory (OOM) risk

Read the complete text