Position: Foundation Models Need Digital Twin Representations
Yiqing Shen, Hao Ding, Lalithkumar Seenivasan, Tianmin Shu, Mathias Unberath · 2025
Read Position: Foundation Models Need Digital Twin Representations by Yiqing Shen, Hao Ding, Lalithkumar Seenivasan, Tianmin Shu, Mathias Unberath (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03798.txt.
Introduction
Yiqing Shen and four co-authors argue a position in a 2025 paper: foundation models need a new kind of input. Today's large AI models chop the world into tokens, small fixed pieces of text, image, or sound. The authors say this chopping forces models to learn the world from surface patterns alone.
They propose digital twin representations instead: digital copies of real processes, built to track outcomes and grounded in domain knowledge. Such copies keep the smooth, unbroken nature of real data and state links between causes plainly. The authors claim this would help models stay coherent across data types, track fine changes in space and time, and reason about causes.
Catalog record
| CREATOR | Yiqing Shen, Hao Ding, Lalithkumar Seenivasan, Tianmin Shu, Mathias Unberath |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-03798
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Position: Foundation Models Need Digital Twin Representations Yiqing Shen, Hao Ding, Lalithkumar Seenivasan, Tianmin Shu, Mathias Unberath Abstract Current foundation models (FMs) rely on token representations that directly fragment continuous real-world multimodal data into discrete tokens. They limit FMs to learning real-world knowledge and relationships purely through statistical correlation rather than leveraging explicit domain knowledge. Consequently, current FMs struggle with maintaining semantic coherence across modalities, capturing fine-grained spatial-temporal dynamics, and performing causal reasoning. These limitations cannot be overcome by simply scaling up model size or expanding datasets. This position paper argues that the machine learning community should consider digital twin (DT) representations, which are outcome-driven digital representations that serve as building blocks for creating virtual replicas of physical processes, as an alternative to the token representation for building FMs. Finally, we discuss how DT representations can address these challenges by providing physically grounded representations that explicitly encode domain knowledge and preserve the continuous nature of real-world processes. 1 Introduction Current multimodal foundation models (FMs), such as multimodal large language models (LLMs), have already demonstrated their capabilities in processing and understanding diverse modalities like text, images, and videos [ 93 , 87 ] . These FMs typically rely on token representations [ 80 ] , which discretize continuous multimodal inputs into tokens representing image patches, or video frames. It therefore introduces various limitations that can constrain FMs’ ability to achieve genuine multimodal understanding and reasoning [ 47 , 89 ] . The primary issue lies in how token representations fragment the natural continuity of real-world knowledge and relationships [ 102 ] . Specifically, when processing multimodal data, these representations create artificial boundaries between modalities, sacrificing important physical relationships and semantic connections [ 96 ] . For instance, in video analysis, discretizing continuous visual and temporal information into fixed-length token sequences compromises the fine-grained spatial relationships and temporal dynamics essential for understanding complex scenarios [ 98 ] . This discretization forces FMs to learn fundamental physical principles and relationships purely through statistical correlations, rather than leveraging explicit physical constraints and domain knowledge. Moreover, token representations struggle to maintain semantic coherence across modalities [ 97 ] . Without an explicit framework for encoding real-world knowledge such as physical laws and domain constraints that govern real-world interactions, current FMs suffer from capturing complex causal relationships [ 43 ] . This limitation particularly affects their ability to perform counterfactual reasoning and generate physically plausible predictions, making them susceptible to spurious correlations and limiting their generalization capabilities [ 104 ] . While previous efforts to address these limitations have focused on scaling up model size and expanding training datasets [ 14 ] , they face inherent constraints due to data availability and computational costs. Even synthetic data generation methods can struggle to produce training examples that adequately reflect real-world physics and domain-specific constraints [ 32 ] . This misalignment occurs because token representations lack explicit modeling of physical laws and domain rules, resulting in generated data that inadequately captures the complexities of real-world scenarios. These challenges indicate the need for an alternative representation paradigm for FMs that better captures the inherent structure and relationships in multimodal data [ 1 , 19 ] . This paradigm is expected to preserve rich semantic relationships across modalities while enabling more efficient and interpretable model operations that maintain alignment with physical constraints and domain knowledge [ 57 ] . Digital twins (DTs) are outcome-driven replicas of physical processes, capturing and modeling task-specific entities and interactions to analyze and optimize the physical counterpart. While traditional DTs mostly focus on industrial applications like manufacturing, smart cities, and healthcare [ 85 ] , we propose extending this concept to create universal semantic representations for FMs. DT representations are the building blocks of DTs, comprising outcome-driven digital representations extracted from raw data. Unlike token representations that fragment continuous information through direct vectorization, DT representations preserve the semantic and physical relationships inherent in multimodal data while maintaining domain-specific constraints [ 21 , 99 ] . The DT paradigm also introduces an architectural separation between low-level representation extraction and high-level FM analysis. Real-world data undergoes outcome-driven processing to extract meaningful representations that capture task-specific entities and interactions, which maps diverse multimodal inputs into unified, interpretable formats while preserving essential semantic relationships. This preservation therefore enables FMs to operate on representations that directly reflect real-world properties and relationships. Our Position The machine learning community should more seriously consider alternative representations of multimodal data beyond tokens. Token representations have been demonstrated to exhibit inherent limitations that cannot be overcome by simply scaling up model size or expanding datasets. We posit that alternative representations, such as DT representations, offer a promising paradigm shift. These representations will enable multimodal FMs to capture the continuous nature of real-world information better, preserve physical constraints, and facilitate meaningful cross-modal interactions . This shift from statistical pattern matching to physically grounded representations will enhance model capabilities across multiple dimensions i . e ., from improved generalization and interpretability to more efficient scaling. Alternative Views Several plausible counterarguments contrast our position on adopting DT representations for FMs. First, proponents of token representations argue that the success of current FMs demonstrates the effectiveness of learned statistical representations [ 86 ] . They contend that token representations, despite their limitations, have proven capable of capturing complex patterns and relationships across modalities without requiring explicit physical modeling [ 10 ] . The historic improvements in performance through architectural innovations and scaling suggest that token representations may eventually overcome their current limitations [ 6 , 92 ] . Another perspective criticizes that the computational overhead of maintaining and processing DT representations could outweigh their benefits. These challenges may be most pronounced in diverse real-world scenarios, where token representations retain their simplicity. They suggest that improving existing token representations through better architectures and training methods might be more cost-effective than developing entirely new representation paradigms [ 50 ] . Some researchers also question whether explicit physical modeling is necessary for achieving robust multimodal understanding [ 84 ] . They argue that human cognition often operates on abstract representations rather than detailed physical models, and FMs might similarly develop effective reasoning capabilities through learned abstractions rather than explicit physical constraints [ 17 ] . This view suggests that the limitations of current token representation might be addressed through improved training objectives and architectural designs rather than fundamental changes to the representation strategy [ 70 , 15 ] . These alternative perspectives highlight important considerations about the practicality, efficiency, and necessity of transitioning to DT representations. 2 Current Landscape of Foundation Models Foundation Models Containing billions or trillions of parameters, FMs usually learn generalizable representations through a self-supervised manner on massive datasets, enabling them to transcend their initial training domains [ 61 , 3 ] . Furthermore, FMs generally offer the flexibility to be adapted to specific datasets and novel downstream tasks in a few-shot ( e . g ., fine-tuning) or a zero-shot ( e . g ., prompt engineering) manner [ 52 ] . Such transformative potential of FMs emerged first in the domain of natural language processing, where Transformer-based large language models (LLMs) demonstrated unprecedented zero-shot capabilities ( i . e ., the ability to handle novel tasks without