NVIDIA Nemotron Nano V2 VL

NVIDIA · 2025

Read NVIDIA Nemotron Nano V2 VL by NVIDIA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-03929.txt.

Introduction

NVIDIA, credited as the collaboration name in the arXiv record, introduces Nemotron Nano V2 VL, a vision-language model aimed at document understanding, long video comprehension, and reasoning. It succeeds an earlier eight-billion-parameter Nemotron vision-language release and claims gains across vision and text domains.

The model builds on a hybrid Mamba-Transformer language model and adds token reduction methods that raise throughput on long documents and video. Architecture, datasets, and training recipes all changed from the prior version. Checkpoints are released in three numeric formats, and large parts of the datasets, recipes, and training code are shared, making the release unusually open for a corporate model report.

Catalog record

CREATORNVIDIA
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-03929

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NVIDIA Nemotron Nano V2 VL NVIDIA Abstract Abstract. We introduce Nemotron Nano V2 VL, the latest model of the Nemotron vision-language series designed for strong real-world document understanding, long video comprehension, and reasoning tasks. Nemotron Nano V2 VL delivers significant improvements over our previous model, Llama-3.1-Nemotron-Nano-VL-8B, across all vision and text domains through major enhancements in model architecture, datasets, and training recipes. Nemotron Nano V2 VL builds on Nemotron Nano V2, a hybrid Mamba-Transformer LLM, and innovative token reduction techniques to achieve higher inference throughput in long document and video scenarios. We are releasing model checkpoints in BF16, FP8, and FP4 formats and sharing large parts of our datasets, recipes and training code. Model (BF16 ) | Model (FP8) | Model (NVFP4-QAD) | Dataset 1 Introduction We introduce Nemotron Nano V2 VL, an efficient 12B vision–language model that achieves leading accuracy on OCRBench v2 ( 23 ) private data leaderboard 1 1 1 https://99franklin.github.io/ocrbench_v2/ , along with strong performance in reasoning, document understanding, long-video comprehension, visual question answering, and STEM reasoning. Nemotron Nano V2 VL delivers substantial improvements over our previous model, Llama-3.1-Nemotron-Nano-VL-8B 2 2 2 https://huggingface.co/nvidia/Llama-3.1-Nemotron-Nano-VL-8B-V1 , driven by enhancements in model architecture, dataset composition, and training methodology. These gains stem from the inclusion of higher-quality reasoning data, expanded OCR datasets, and additional long-context datasets. In addition to overall benchmark improvements, we extend the model’s context length from 16K to 128K, enabling better handling of long videos and complex reasoning tasks. Consistent with our prior open-source efforts, we release the model weights, along with substantial portions of the training datasets, recipes, and codebase, to support continued research and development. Nemotron Nano V2 VL builds on top of the Nemotron Nano V2 ( 93 ) 12B reasoning LLM and RADIOv2.5 vision encoder ( 34 ) . In addition, Nemotron Nano V2 VL adopts the multimodal fusion architecture, training recipe as well as data strategy similar to Eagle 2 and 2.5 ( 64 ; 12 ) . We use the open-source Megatron ( 104 ) framework to train the model in FP8 precision using Supervised Finetuning (SFT) across several vision and text domains, followed by additional SFT stages to further improve video understanding, long context performance, and recover text-only capabilities to achieve competitive results across many vision and text benchmarks. Compared to Llama-3.1-Nemotron-Nano-VL-8B, the hybrid Mamba-Transformer architecture of the LLM offers 35% higher throughput in long multi-page document understanding scenarios. Additionally, we employ Efficient Video Sampling (EVS) ( 5 ) to accelerate throughput in video understanding use cases by 2x or more with minimal or no impact on accuracy. Furthermore, Nemotron Nano V2 VL supports both reasoning-on and reasoning-off modes, with the former enabling extended reasoning for tasks that require more complex problem-solving. This design enables a balanced trade-off between computational efficiency and task performance. We are releasing our model weights on HuggingFace in BF16, FP8 and FP4 formats: • Nemotron-Nano-12B-v2-VL : The final model weights after our multi-stage training recipe. • Nemotron-Nano-12B-v2-VL-FP8 : Quantized model weights in FP8 format • Nemotron-Nano-12B-v2-VL-NVFP4-QAD Quantized model weights in FP4 format using Quantization-aware Distillation (QAD) Additionally, we release a large portion of our SFT dataset and tooling: • Nemotron VLM Dataset V2 : A collection of over 8 million training samples. • NVPDFTex : A custom LaTeX compiler toolchain to generate annotated OCR ground truth. The remainder of this report is organized as follows: Section 2 describes the model architecture and input processing pipeline; Section 3 outlines the training recipe, datasets, and hyperparameters; and Section 4 presents comprehensive evaluations of the model across both multimodal and pure-text tasks. 2 Model Architecture Figure 1 : Visualization of our VLM architecture. For images, we extract a dynamic number of tiles based on the image aspect ratio. For videos, we uniformly extract frames. Tiles and frames are resized to 512\times 512 pixels, and go through the RADIO vision encoder and an MLP connector. The image and text embeddings are interleaved, and fed to the Nemotron-Nano-12B-V2 LLM. As illustrated in Figure 1 , Nemotron Nano V2 VL consists of three modules: a vision encoder, an MLP projector, and a language model. We initialize the vision encoder using the c-RADIOv2-VLM-H version of the RADIOv2 vision encoder ( 34 ) and the language model with Nemotron-Nano-12B-V2 ( 93 ) . Inspired by InternVL ( 17 ) , LLaVA-1.5 ( 70 ) and Eagle ( 64 ; 12 ) , we adopt a tiling strategy to handle varying image resolutions. First, each image is resized following the aspect ratio matching strategy employed by InternVL ( 17 ) so that its width and height are multiples of s . Then it is divided into non-overlapping tiles of size s\times s . In this work, we set s=512 . With a patch size of 16, this results in 1024 visual tokens per tile. For scalability, we employ pixel shuffle with 2x downsampling to reduce the token count further to 256. During training, we set the maximum number of tiles to 12. Additionally, we use a single-tile thumbnail of the image to capture global image context. For video inputs, we limit each input frame to a single tile. 3 Training Recipe & Datasets To preserve the base language model’s text comprehension and reasoning abilities while improving its visual understanding, we adopt a multi-stage training approach as detailed below. 3.1 Stage 0 In this stage, we aim to warm up the MLP connector to establish cross-modal alignment between the language and vision domains. To this end, we freeze the vision encoder and language model weights and train only the MLP connector on a diverse multimodal subset of the Stage 1 SFT dataset (see Section 3.2 ), consisting of approximately 2.2 million samples (up to 36 billion tokens) spanning multiple tasks, including captioning, visual question answering, visual grounding, OCR, and document extraction. 3.2 SFT Stage 1: 16K context length In this and all subsequent stages, we unfreeze all model components for training. In SFT Stage 1, the maximum sequence length is set to 16,384 tokens. This stage is trained on approximately 32.5 million samples (about 112.5 billion tokens). To preserve the text comprehension capabilities of the Nemotron-Nano-V2 ( 93 ) LLM backbone, we incorporate a subset of the text reasoning data used in its Stage 1 SFT training, comprising approximately 6.5 million samples (around 40 billion tokens) spanning diverse domains and tasks such as mathematics, science, code, multilingual understanding, multi-turn dialogue, tool-use, and safety. In addition, we include multimodal datasets totaling 26 million samples (approximately 72 billion tokens) drawn from various tasks and sources, including: 1. Image Captioning : OpenImages ( 54 ) , TextCaps ( 106 ) , TextVQA ( 107 ) , PixMo-cap ( 20 ) . 2. Video Captioning : Localized Narratives ( 97 ) , YouCook2 ( 153 ) , VaTeX ( 124 ) . 3. General Visual QA : TextVQA ( 107 ) , VQAv2 ( 27 ) , OK-VQA ( 84 ) , GQA ( 42 ) , CLEVR ( 47 ) , CLEVR-Math ( 66 ) , TallyQA ( 1 ) , Dolly-15K ( 19 ) , ScreenQA ( 37 ) , VizWiz ( 32 ) , MapQA ( 10 ) , ScienceQA ( 79 ) , PMC-VQA (

Read the complete text