CLIPTime: Time-Aware Multimodal Representation Learning from Images and Text
Anju Rani, Daniel Ortiz-Arroyo, Petar Durdevic · 2025
Read CLIPTime: Time-Aware Multimodal Representation Learning from Images and Text by Anju Rani, Daniel Ortiz-Arroyo, Petar Durdevic (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00447.txt.
Introduction
Anju Rani and co-authors teach a vision-language model to sense growth over time in a 2025 paper. Growth timing matters in microbiology, farming, and the study of materials breaking down. Standard joint image-text models read scenes well but track change over time poorly.
Their model, CLIPTime, learns from images and text together to judge fungal growth. It predicts both the growth stage and the time stamp of a sample. At test time it needs no explicit time input to make that call. A synthetic fungal dataset, built and labeled for this task, trains and tests the system. Time-aware reading could spread to crops, cultures, and decay studies.
Catalog record
| CREATOR | Anju Rani, Daniel Ortiz-Arroyo, Petar Durdevic |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-00447
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CLIPTime: Time-Aware Multimodal Representation Learning from Images and Text Anju Rani, Daniel Ortiz-Arroyo, Petar Durdevic Abstract Understanding the temporal dynamics of biological growth is critical across diverse fields such as microbiology, agriculture, and biodegradation research. Although vision-language models like Contrastive Language Image Pretraining (CLIP) have shown strong capabilities in joint visual-textual reasoning, their effectiveness in capturing temporal progression remains limited. To address this, we propose CLIPTime, a multimodal, multitask framework designed to predict both the developmental stage and the corresponding timestamp of fungal growth from image and text inputs. Built upon the CLIP architecture, our model learns joint visual-textual embeddings and enables time-aware inference without requiring explicit temporal input during testing. To facilitate training and evaluation, we introduce a synthetic fungal growth dataset annotated with aligned timestamps and categorical stage labels. CLIPTime jointly performs classification and regression, predicting discrete growth stages alongside continuous timestamps. We also propose custom evaluation metrics, including temporal accuracy and regression error, to assess the precision of time-aware predictions. Experimental results demonstrate that CLIPTime effectively models biological progression and produces interpretable, temporally grounded outputs, highlighting the potential of vision-language models in real-world biological monitoring applications. The dataset and implementation are available in our open-source GitHub repository: github.com/PetarDurdevic/Funghi Keywords Deep learning \cdot CLIP \cdot Synthetic Dataset \cdot Classification \cdot Segmentation \cdot Biological processes 1 Introduction Vision-language models (VLMs) minaee2024large , such as Contrastive Language–Image Pretraining (CLIP) radford2021learning , have significantly advanced deep learning (DL) by unifying computer vision (CV) and natural language processing (NLP) through a shared multimodal embedding space. These models enable a broad range of tasks, including zero-shot classification and image-text retrieval, by encoding images and texts independently and aligning them in a common semantic space. Despite their strengths in spatial and semantic reasoning, a major limitation of current VLMs is their inability to natively process temporal information. This restricts their applicability in domains where understanding progression or sequential patterns is essential, such as biological process modeling sarch2024vlm , cooper2025rethinking , forecasting xue2025regression , zhao2025images , ouyang2018multi , and time-sensitive monitoring applications hong2024cogvlm2 , mahmud2025integrating . Recent efforts have aimed to extend VLMs by incorporating temporal dynamics, for example through temporal attention or sequence alignment in video-text tasks. Nevertheless, fully integrating spatial, semantic, and temporal reasoning within a unified vision-language framework remains a core challenge and a promising direction for future research. Recent studies have begun to incorporate temporal reasoning into multimodal architectures. iTransformer liu2023itransformer introduces a novel transformer-based model tailored for time-series forecasting. Unlike conventional transformers that apply attention across sequential time steps, iTransformer inverts this approach by applying attention over the feature dimension. This allows the model to capture inter-variable dependencies more effectively, which is particularly beneficial when handling multivariate time-series data. Similarly, PV-VLM lin2025pv proposes a unified architecture for intra-hour photovoltaic (PV) power forecasting that integrates temporal, textual, and visual modalities. It comprises three specialized modules: a time-aware component based on PatchTST for modeling both local and global dependencies in PV time-series; a prompt-aware module that uses a large language model (LLM) to incorporate textual weather and domain-specific knowledge; and a vision-aware module that employs pretrained VLMs to extract semantic features from sky imagery. Expanding on this multimodal framework, Time-VLM zhong2025time enhances time-series forecasting by converting temporal data into images, generating contextual textual descriptions, and integrating them via retrieval-augmented memory. This approach enriches multimodal reasoning and leads to notable gains in forecasting accuracy. In parallel, several efforts have aimed to adapt CLIP-based architectures for temporal tasks, capitalizing on their strong vision-language grounding to enable time-aware reasoning. There is growing interest in extending CLIP to handle applications such as forecasting, temporal reasoning, and time-series analysis. Approaches like TC-CLIP kim2024leveraging , C-TPT yoon2024c , and others singh2024pattern , he2024driver , feng2023diverse explore how temporal information can be integrated into vision-language models, either implicitly through visual progression or explicitly via temporal embeddings. These works underscore the potential of multimodal models to reason not only about what is observed, but also when it occurs. TC-CLIP kim2024leveraging enhances video action recognition by incorporating temporal context into CLIP’s architecture. Rather than analyzing each frame in isolation, it introduces specialized context tokens that aggregate information across frames, enabling the model to better understand temporal dependencies, particularly in low-label scenarios. C-TPT yoon2024c improves the reliability and interpretability of CLIP’s predictions by modifying the text prompts through text-feature dispersion. This encourages greater diversity among prompts, which increases prediction robustness and enhances model transparency regarding uncertainty. Other approaches further demonstrate CLIP’s temporal versatility. For instance, singh2024pattern converts time-series data into visual representations to leverage CLIP for retrieving visually similar patterns from a reference set, subsequently using the top-k matches for forecasting. In he2024driver , CLIP is employed to predict driver intentions, such as turning or lane changes, by encoding both in-cabin and external video data alongside textual prompts. This method captures temporal dynamics like motion patterns and sequential behaviors, illustrating CLIP’s ability to model time-dependent phenomena and laying the groundwork for its application in time prediction tasks. In this work, we extend the CLIP framework to reason jointly across visual, textual, and temporal modalities, aiming to predict both the stage of fungal growth and its associated timestamp. Fungal development follows a biologically ordered sequence, beginning with spores, transitioning through hyphae, and culminating in the formation of mycelium. To model this progression, we propose a multimodal architecture capable of capturing time-aware patterns without requiring explicit temporal input during inference. Our main contributions are summarized as follows: 1. We extend CLIP’s joint vision-language embedding space to support continuous time prediction, enabling the model to infer temporal dynamics from multimodal semantics. 2. We introduce a transformer-based temporal prediction head that employs self-attention to learn temporal dependencies within the fused CLIP embeddings. 3. We employ a multi-task learning framework that jointly performs fungal stage classification and timestamp regression, enhancing both representation learning and inference robustness. 4. We fine-tune the proposed model on a synthetically generated, time-annotated fungal growth dataset, showcasing CLIP’s adaptability to scientific domains that demand temporally structured predictions beyond typical vision-language tasks. The remainder of this paper is organized as follows: Section 2 presents the proposed methodology, including the creation of the synthetic fungal growth dataset and the design of the multimodal architecture for processing visual, textual, and temporal inputs. Section 3 describes the implementation details and reports the experimental results. Section 4 concludes the paper by summarizing the key findings and discussing their broader implications. Figure 1: Illustration of the proposed framework. 2 Methodology This section describes the architecture of the proposed CLIPTime model, which is designed to jointly process and reason across multimodal inputs encompassing visual, textual, and temporal information. We begin by detailing our multi-task learning framework that simultaneously performs fungal growth stage classification and timestamp regression. This is followed by an overview of the synthetic fungal growth dataset used to train and evaluate the model. 2.1 Multi-Task Fusion of CLIP Unlike traditional applications of CLIP that focus on zero-shot classification, CLIPTime repurposes the joint vision-language embedding space for time-aware inference. In this framework, images are first encoded using CLIP’s vision encoder, while corresponding textual prompts describing fungal stages are processed by the text encoder rani2025fungalzsl . These embeddings are fused to form a shared multimodal representation that captures the semantic