Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning

NVIDIA · 2025

Read Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning by NVIDIA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15558.txt.

Introduction

A large team from NVIDIA presents models that reason about the physical world and pick next steps for robots in plain words, in a paper from 2025. Body bound smart systems must see a scene, grasp how it works, and act in long chains. The models are trained to think in long word chains that end in a clear move call.

The team first lays out what such thought must hold, split into world sense and body choice. World sense is ranked in a tree of space, time, and force facts. Body choice is laid on two axes that span tasks from grip to route. Trained models score well on tests of both kinds. The authors share the build as a base for robot thought.

Catalog record

CREATORNVIDIA
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE105 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-15558

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning NVIDIA Abstract Physical AI systems need to perceive, understand, and perform complex actions in the physical world. In this paper, we present the Cosmos-Reason1 models that can understand the physical world and generate appropriate embodied decisions ( \eg , next step action) in natural language through long chain-of-thought reasoning processes. We begin by defining key capabilities for Physical AI reasoning, with a focus on physical common sense and embodied reasoning. To represent physical common sense, we use a hierarchical ontology that captures fundamental knowledge about space, time, and physics. For embodied reasoning, we rely on a two-dimensional ontology that generalizes across different physical embodiments. Building on these capabilities, we develop two multimodal large language models, Cosmos-Reason1-7B and Cosmos-Reason1-56B. We curate data and train our models in two stages: Physical AI supervised fine-tuning (SFT) and Physical AI reinforcement learning (RL). To evaluate our models, we build comprehensive benchmarks for physical common sense and embodied reasoning according to our ontologies. Evaluation results show that Physical AI SFT and RL bring significant improvements. To facilitate the development of Physical AI, we make our code and pre-trained models available under the NVIDIA Open Model License at https://github.com/nvidia-cosmos/cosmos-reason1 . \abscontent 1 Introduction Physical AI systems are designed to interact with the physical world. To effectively follow instructions and take appropriate actions to achieve a desired goal, they need first to perceive, understand, and reason about the physical world. Recently, with breakthroughs of post-training and test-time scaling via long chain-of-thought (CoT) processes, large language models (LLMs) have demonstrated remarkable general reasoning capabilities in tackling complex problems across domains such as coding and mathematics ( OpenAI, 2024 ; DeepSeek-AI, 2025 ) . However, a key limitation of these models lies in their ability to ground their knowledge in the physical world. While LLMs trained on vast amounts of textual data from the Internet may acquire essential knowledge to reason about the physical world, they often struggle to establish connections between that knowledge and real-world interactions and dynamics. In this paper, we begin by defining the fundamental capabilities essential for Physical AI systems. Unlike designing models that excel at solving coding and mathematical problems, our focus is on empowering models with physical common sense knowledge and embodied reasoning capabilities grounded in the real world. To establish a shared framework and help measure progress, we propose two ontologies. First, a hierarchical ontology that organizes physical common sense into three major categories — Space, Time, and Fundamental Physics — which are further divided into 16 fine-grained subcategories. This ontology encapsulates knowledge about how the physical world operates under the laws of physics and how it responds to interactions with embodied agents. Second, we introduce a two-dimensional ontology for embodied reasoning, which encompasses four key reasoning capabilities across five types of embodied agents. Effective embodied reasoning based on physical common sense is crucial for understanding and planning actions that achieve specific objectives in the real world. The details are described in Section 2 . We introduce Cosmos-Reason1 as a step toward enabling multimodal LLMs to generate more physically grounded responses. We focus on the visual world, where the observations of the world are represented as videos. Cosmos-Reason1 perceives the physical world through video input, understands it, and reasons about it through long chain-of-thought thinking processes before generating responses. These responses, expressed in natural language, include both explanatory insights and embodied decisions, such as determining the next action to take. We employ a decoder-only multimodal LLM architecture where the input video is processed by a vision encoder followed by a projector to align with the text token embeddings before feeding into the LLM. We experiment with both dense Transformer and hybrid Mamba-MLP-Transformer architecture as the LLM backbone. Cosmos-Reason1 comes with two model sizes: Cosmos-Reason1-7B and Cosmos-Reason1-56B. We describe details about the model architecture in Sec. 3 . Constructing rule-based, verifiable rewards at scale has been critical for the reasoning LLMs’ success in solving math and coding problems. Can we design rule-based, verifiable rewards for training Physical AI reasoning models with reinforcement learning? In this work, we explore two types of rewards based on answering multiple-choice questions (MCQs). The first type of MCQs are designed based on human annotations. With inspiration from video self-supervised learning, we automatically generate a second type of MCQ based on the structure of video data itself, such as solving puzzles with shuffled spatiotemporal video patches or predicting the arrow of time on whether a video is playing forward or backward. All these rewards are rule-based, verifiable, and highly relevant to Physical AI capabilities. We discuss details about the design of RL training algorithm and infrastructure in Sec. 4 . Data determines our model’s ceiling. To strengthen our models’ physical common sense and embodied reasoning capabilities, we carefully curate around 4M annotations of video and text pairs, including captions for physical understanding, multiple choice questions, and long chain-of-thought reasoning traces. We develop two pipelines to curate these physical common sense and embodied reasoning data according to our ontologies. The data are curated based on human annotations and model distillation from DeepSeek-R1 ( DeepSeek-AI, 2025 ) for Physical AI supervised fine-tuning. Details about data are discussed in Sec. 5 . Figure 1 : An overview of Cosmos-Reason1. Cosmos-Reason1 contains two multimodal large language models of 7B and 56B, trained in two stages, including Physical AI SFT and Physical AI RL. We also define two ontologies for physical common sense and embodied reasoning, and build two benchmarks to evaluate models’ Physical AI reasoning capabilities. In order to evaluate our models, we build new benchmarks for evaluating Physical AI capabilities in Sec. 6 . For physical common sense in Sec. 6.1 , we build three benchmarks (Space, Time, and Fundamental Physics) containing 604 questions from 426 videos. For embodied reasoning in Sec. 6.2 , we build six benchmarks containing 610 questions from 600 videos, covering a wide range of tasks across different physical embodiments, including humans, robot arms, humanoid robots, and autonomous vehicles. Sec. 7 presents the evaluation results of Cosmos-Reason1 and comparisons with existing models. In Sec. 7.1 , we introduce the experiment setup, including training details for Physical AI SFT, and evaluation results of SFT models on our benchmarks. In Sec. 7.2 , we present the evaluation results for Physical AI RL. Training using our rule-based, verifiable rewards in RL post-training leads to improvements in all of our benchmarks. Fig. 1 shows an overview of Cosmos-Reason1. In summary, we introduce two multimodal large language models, Cosmos-Reason1-7B and Cosmos-Reason1-56B. The models are trained in two stages: Physical AI SFT and Physical AI RL. We define ontologies for physical common sense and embodied reasoning and build benchmarks to evaluate models’ Physical AI reasoning capabilities. To facilitate the advancement of Physical AI, we make our code and pre-trained models available under the NVIDIA Open Model License at https://github.com/nvidia-cosmos/cosmos-reason1 . Building reasoning models for Physical AI is an open problem that is far from being solved, and we hope our paper contributes to the advancement of this field. 2 Physical AI Reasoning We identify two important capabilities for Physical AI reasoning models — physical common sense reasoning and embodied reasoning. First, Physical AI models should possess

Read the complete text