MTPChat: A Multimodal Time-Aware Persona Dataset for Conversational Agents

Wanqi Yang Yanda Li Meng Fang Ling Chen · 2025

Read MTPChat: A Multimodal Time-Aware Persona Dataset for Conversational Agents by Wanqi Yang Yanda Li Meng Fang Ling Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-05887.txt.

Introduction

Wanqi Yang, Yanda Li, Meng Fang, and Ling Chen built a dataset for chat agents that track time in a 2025 paper. Good agents must grasp how talks change over time to reply well. Yet few datasets mix persona, images, and time cues in one place. The team made MTPChat to fill that gap.

MTPChat joins text, images, and time data in persona-based chats. The team set two tasks to test time skill. One asks for the next reply with time in mind. The other asks which past memory fits a time cue. They also built a model with a module that adapts to time signals. Tests show the tasks are hard, and their model does best on them so far.

Catalog record

CREATORWanqi Yang Yanda Li Meng Fang Ling Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-05887

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MTPChat: A Multimodal Time-Aware Persona Dataset for Conversational Agents Wanqi Yang Yanda Li Meng Fang Ling Chen Abstract Understanding temporal dynamics is critical for conversational agents, enabling effective content analysis and informed decision-making. However, time-aware datasets, particularly for persona-grounded conversations, are still limited, which narrows their scope and diminishes their complexity. To address this gap, we introduce MTPChat, a multimodal, time-aware persona dialogue dataset that integrates linguistic, visual, and temporal elements within dialogue and persona memory. Leveraging MTPChat, we propose two time-sensitive tasks: Temporal Next Response Prediction (TNRP) and Temporal Grounding Memory Prediction (TGMP), both designed to assess a model’s ability to understand implicit temporal cues and dynamic interactions. Additionally, we present an innovative framework featuring an adaptive temporal module to effectively integrate multimodal streams and capture temporal dependencies. Experimental results validate the challenges posed by MTPChat and demonstrate the effectiveness of our framework in multimodal time-sensitive scenarios. 1 Introduction Temporal awareness has garnered significant attention in AI research, particularly following Min et al.’s work Min et al. (2020) , which highlighted the inherent temporal dynamics in question-answering systems. Understanding time-sensitive information is crucial across various domains, including financial decision-making, event prediction, multimedia content analysis, and conversational AI. To explore temporal reasoning in large language models (LLMs), multiple time-sensitive datasets have been developed. TimeQA Chen et al. (2021) and SituatedQA Zhang and Choi (2021) provide temporally grounded questions with free-text contexts extracted from WikiData Vrandečić and Krötzsch (2014) . Similarly, TEMPLAMA Dhingra et al. (2022) builds on temporal knowledge bases, while StreamingQA Liska et al. (2022) compiles time-sensitive question-answering (QA) data from English news articles spanning 2007 to 2020. Figure 1: An example of a multimodal, time-sensitive, persona-grounded scenario, showcasing how the user’s dialogue responses evolve over time based on the temporal dynamics of dialogue and episodic memories. Dataset Knowledge Corpus # Samples Time-Sensitive Task has Images TempLama Dhingra et al. (2022) CustomNews 50.0k YES Question Answering NO TimeQA Chen et al. (2021) Wikipedia 41.2k YES Question Answering NO StreamingQA Liska et al. (2022) WMT07-20 138.0k YES Question Answering NO TempReason-L2L3 Tan et al. (2023) Wikipedia 49.0k YES Question Answering NO PhotoChat Zang et al. (2021) OpenImage V4 12.3k NO Dialogue YES MMDialog Feng et al. (2022) SocialMedia 1.1M NO Dialogue YES MTPChat Reddit 28.7k YES Dialogue YES Table 1: Related datasets overview, including free-text time-sensitive datasets and multimodal dialogue datasets. However, these datasets primarily focus on text-based QA tasks, limiting their applicability to real-world conversational AI. They lack the multimodal components (e.g., images) that are essential for capturing rich temporal contexts and do not account for persona-grounded dialogues, where responses evolve based on a user’s dynamic memory and past interactions. Although TimeIT Ren et al. (2023) introduces time-sensitive multimodal tasks for long-video understanding, it has several limitations: (1) its focus on QA tasks restricts broader conversational applications, (2) the use of explicit temporal markers in videos reduces the challenge of reasoning over implicit temporal cues, and (3) its rigid response format (e.g., “<timestamp_start> to <timestamp_end> seconds: <event_description>”) simplifies the task, minimizing complex temporal reasoning. Similarly, MPChat Ahn et al. (2023) provides persona-grounded dialogues with multimodal memory, but it lacks an explicit temporal dimension. To overcome these limitations, we introduce MTPChat, a multimodal, time-aware persona dialogue dataset built upon MPChat Ahn et al. (2023) , a comprehensive multimodal persona-grounded dialogue dataset. Rather than relying on explicit timestamps, MTPChat leverages the natural progression of dialogues and memories to simulate real-world temporal shifts in human cognition. Figure 1 illustrates an example of a multimodal time-sensitive scenario. Our dataset integrates linguistic, visual, and temporal elements, making it the first of its kind to model persona-driven temporal changes in dialogues and memory. Unlike existing time-sensitive datasets, MTPChat incorporates dialogue, persona memory, and visual elements, enhancing its realism and complexity. In addition, we propose two novel tasks—Temporal Next Response Prediction (TNRP) and Temporal Grounding Memory Prediction (TGMP)—that challenge models to infer implicit temporal cues and track evolving responses over time. Beyond dataset creation, we introduce an adaptive temporal module designed to enhance the temporal reasoning capabilities of multimodal models. This framework dynamically integrates linguistic, visual, and temporal streams, allowing for more effective reasoning over time-sensitive interactions. Specifically, the module dynamically merges features based on their temporal relevance, improving coherence in multimodal integration. To evaluate MTPChat, we conducted experiments using SBERT Reimers and Gurevych (2019) and CLIP Radford et al. (2021) . Our results demonstrate that MTPChat introduces new challenges in multimodal, time-sensitive scenarios, requiring models to track temporal changes effectively. Our adaptive temporal module outperforms other feature integration methods, significantly enhancing a model’s ability to reason over multimodal time-aware dialogue. The main contributions of this work are as follows: • We introduce the first multimodal, time-aware persona dialogue dataset, which contains numerous instances where both dialogue responses and grounding memories evolve significantly over time. • We define Temporal Next Response Prediction (TNRP) and Temporal Grounding Memory Prediction (TGMP) to advance research in time-aware conversational AI. • We propose a framework with an adaptive temporal module that enhances a model’s ability to integrate multimodal streams while maintaining temporal awareness. • Experimental results validate the novel challenges posed by MTPChat and demonstrate that our framework outperforms existing methods in multimodal temporal reasoning. 2 Comparison with Existing Datasets We start with a brief comparison of existing datasets, emphasizing multi-modal and time-aware strategies (see Table 1 for an overview). Time-Sensitive QA Datasets Time-Sensitive Question Answering (TSQA) involves interpreting and responding to questions that are dependent on specific time points or intervals. We analyse a set of TSQA datasets Dhingra et al. (2022) ; Chen et al. (2021) ; Liska et al. (2022) ; Tan et al. (2023) , as shown in the upper part of Table 1 . Currently, TSQA datasets typically use free-text form or knowledge graphs (KGs) and are structured as QA tasks. However, our work introduces the first multimodal time-aware dataset based on conversation. Similar to TSQA, we modify the time of dialogues, which affects the responses and the related grounding memory, thereby testing the model’s ability to understand time. MultiModal Dialogue Datasets Multimodal dialogue datasets generally comprise one or more images and multi-turn textual dialogues. As depicted in the lower half of Table 1 , we analyse two representative datasets Zang et al. (2021) ; Feng et al. (2022) . These datasets are designed for models to interpret images and utterances within a dialogue framework and generate coherent responses. Our MTPChat dataset, although drawing on the conversational structure and task, distinctively emphasizes the annotation and manipulation of time information. MTPChat allows the model to acknowledge the influence of temporal dynamics on dialogue interaction and memory processes, demonstrating temporal awareness. Time-Sensitive Video-Centric Dataset TimeIT Ren et al. (2023) is a novel dataset focused on video-based instructions, encompassing a collection of long-video datasets annotated with timestamps. It requires models to describe video content across specified time intervals. The description follows a structured format, such as “<timestamp_start> to <timestamp_end> seconds: <event_description>”. Ingeniously, our dataset integrates time of dialogues and memories, making model awareness of the time order of dialogue and memory significant influence on dialogue responses and memory recall. In contrast to TimeIT’s tasks that directly answer timestamp and associated content, MTPChat offers a more complex challenge with

Read the complete text