LiFT: Does Instruction Fine-Tuning Improve In-Context Learning for Longitudinal Modelling by Large Language Models?

Iqra Ali, Talia Tseriotou, Mahmud Elahi Akhter, Yuxiang Zhou, Maria Liakata · 2026

Read LiFT: Does Instruction Fine-Tuning Improve In-Context Learning for Longitudinal Modelling by Large Language Models? by Iqra Ali, Talia Tseriotou, Mahmud Elahi Akhter, Yuxiang Zhou, Maria Liakata (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16382.txt.

Introduction

Ali and colleagues introduce LiFT, a way to teach language models to follow text that changes over time, as in mental-health monitoring or shifting opinions. Standard in-context learning struggles with ordered sequences, rare events and sparse labels. LiFT unifies such tasks under one instruction format that uses sequence order, staged training and time-aware conditioning. Tested on five longitudinal datasets in zero, one and three-example settings, it beat comparable instruction-tuned models. Gains were larger for minority classes, and the model paid clear attention to recent examples.

Catalog record

CREATORIqra Ali, Talia Tseriotou, Mahmud Elahi Akhter, Yuxiang Zhou, Maria Liakata
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58594
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16382

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LiFT: Does Instruction Fine-Tuning Improve In-Context Learning for Longitudinal Modelling by Large Language Models? Iqra Ali, Talia Tseriotou, Mahmud Elahi Akhter, Yuxiang Zhou, Maria Liakata Abstract Longitudinal NLP tasks require reasoning over temporally ordered text to detect persistence and change in human behavior and opinions. However, in-context learning with large language models struggles on tasks where models must integrate historical context, track evolving interactions, and handle rare change events. We introduce LiFT, a longitudinal instruction fine-tuning framework that unifies diverse longitudinal modeling tasks under a shared instruction schema. LiFT uses a curriculum that progressively increases temporal difficulty while incorporating few-shot structure and temporal conditioning to encourage effective use of past context. We evaluate LiFT across five datasets. Models trained on longitudinal tasks with different levels of temporal granularity are tested for generalisability on two separate datasets. Across models with different parameter sizes (OLMo (1B/7B), LLaMA-8B, and Qwen-14B), LiFT consistently outperforms base-model ICL, with strong gains on out-of-distribution data and minority change events. 1 Introduction Many real-world applications require understanding how language, behaviour, and latent user states evolve over time. Such problems fall under longitudinal modeling 59 ; 15 , where observations are analyzed as temporally ordered sequences rather than independent instances 3 ; 53 ; 2 ; 44 . Unlike static NLP tasks that treat inputs independently, longitudinal tasks require modeling persistence, temporal dependencies, and state transitions across sequences of observations 55 ; 9 ; 29 . Longitudinal modeling arises in many NLP settings, including stance change detection 28 ; 31 , mental health monitoring 49 ; 48 ; 51 ; 52 ; 21 ; 22 , multi-session dialogue reasoning 35 ; 20 ; 17 ; 16 , timeline summarization 46 ; 64 , longitudinal clinical prediction and summarization 9 ; 29 ; 27 ; 6 ; 5 , and behavioral forecasting from experiential timelines 75 ; 18 ; 19 . Recent datasets and models increasingly focus on capturing temporal dependencies and evolving user behavior 71 ; 68 ; 54 ; 25 . Despite strong general capabilities, Large Language Models (LLMs) often struggle with longitudinal prediction in zero-shot or few-shot prompting settings 7 ; 47 ; 14 . Most approaches treat prediction as a static task by concatenating historical observations into a single prompt without explicitly modeling temporal structure 45 ; 68 ; 66 ; 36 ; 30 . Recent work highlights the limitations of prompting-based approaches. Importantly, improving prompting strategies alone does not improve performance on longitudinal tasks 4 ; 69 ; 43 . Moreover, longitudinal tasks present structural difficulties for prompting and in-context learning (ICL) . Histories may span long time horizons and exceed practical context limits 67 ; 34 . Predictive signals are often distributed across multiple observations 55 ; 9 , observations are sequentially interdependent 51 ; 17 , and many tasks require detecting behavioral trajectories rather than static patterns 48 ; 25 ; 73 . In light of these challenges, we introduce LiFT , a framework for temporally structured NLP tasks. LiFT improves zero- and few-shot performance, agnostic to the task or domain, through a multi-domain instruction curriculum, stage-wise few-shot training, and temporal conditioning. We make the following contributions: • We introduce LiFT , a curriculum-based instruction fine-tuning framework that progressively increases temporal difficulty and incorporates few-shot demonstrations to encourage modeling over temporally structured histories. • We propose temporal–label conditioning with history-aware auxiliary objectives to encourage models to extract predictive signals from temporally distributed context. • Through comprehensive ablations and experiments we show that LiFT improves zero- and few-shot performance across models of different sizes (OLMo (1B/7B), LLaMA-8B, and Qwen-14B) and transfers to unseen domains and tasks. • Probing and activation analysis explains why the framework works, showing it induces increased reliance on temporally structured history. 2 Related Work LLMs for Longitudinal Tasks Recent work explores LLMs for longitudinal tasks involving clinical records, mental health monitoring, timeline summarization, and multi-session dialogue 33 ; 46 ; 64 ; 16 ; 50 ; 59 . LLMs have been applied to longitudinal electronic health records (EHRs) for tasks such as patient trajectory summarization and outcome prediction 29 ; 27 ; 1 ; 13 . While some approaches use instruction fine-tuning or task-specific prompting, they often treat longitudinal data as concatenated text, limiting the modeling of temporal progression and evolving states 45 ; 51 . Similar challenges arise in mental health tasks, where psychological signals often emerge across timelines rather than individual posts 49 ; 59 . Longitudinal Classification Modeling Approaches model trajectories using sequential architectures such as recurrent networks and time-aware transformers to capture temporal dependencies across observations 55 ; 2 ; 44 . These methods have been applied to stance change detection, dialogue classification, and clinical outcome prediction from longitudinal records 28 ; 31 ; 35 ; 29 ; 9 . In mental health settings, longitudinal classification is used to detect behavioral or psychological changes across user timelines 50 ; 59 . Applying large language models (LLMs) to these tasks remains challenging. Prompting-based approaches typically concatenate historical observations into a single input, requiring models to implicitly infer temporal dependencies 45 ; 51 . Recent work shows that such methods often under-perform smaller supervised models on longitudinal classification tasks 23 ; 63 ; 70 ; 68 . Figure 1: LiFT framework: The IFT Builder constructs histories and a staged curriculum from AnnoMI, LRS, and TalkLife . The IFT Trainer injects temporal and label embeddings and fine-tunes a base model with LoRA for improved ICL . 2.1 Prompt-Based Strategies To address limitations of in-context learning (ICL), recent work has explored improved prompt design and instruction-based prompting strategies 37 ; 40 ; 11 ; 26 . Techniques such as demonstration selection, task decomposition, and chain-of-thought prompting improve performance on complex reasoning tasks by making task structure explicit in the prompt 40 ; 62 ; 56 ; 76 . However, prompt-based methods remain sensitive to prompt formulation, example ordering, and demonstration formatting 12 ; 39 ; 74 . This brittleness is problematic in longitudinal settings, where models integrate temporally distributed information across inputs, limiting prompting-based approaches. 2.2 Instruction Fine-Tuning Strategies Instruction fine-tuning (IFT) adapts large language models (LLMs) to follow natural language instructions using curated instruction–response datasets and complements prompting and in-context learning by aligning models to better interpret task instructions 61 ; 38 ; 42 . Large-scale frameworks such as FLAN and FLAN-T5 demonstrate that training on diverse instruction-based tasks improves zero-shot and cross-task generalization 60 ; 8 . Datasets such as Super-NaturalInstructions and methods like Self-Instruct further expand instruction diversity through automatically generated instruction–task pairs 58 ; 57 . Multi-task instruction mixtures enhance generalization across tasks 60 ; 8 , while parameter-efficient approaches such as LoRA and QLoRA enable scalable tuning with minimal parameter updates 24 ; 10 . Despite these advances, instruction-tuned models typically treat inputs as independent text segments and rarely model temporal dependencies across sequences 32 , limiting their ability to reason over longitudinal trajectories. 3 Methodology We introduce LiFT (Longitudinal Instruction Fine-Tuning), a framework for improving language models’ ability to reason over temporally structured sequences. Standard instruction fine-tuning and in-context learning treat examples largely as independent instances and therefore fail to model temporal dependencies across events. LiFT addresses this by combining several mechanisms for temporal

Read the complete text