Large Language Models for Sequential Decision-Making: Improving In-Context Learning via Supervised Fine-Tuning

Minmin Zhang, Sina Aghaei, Soroush Saghafian · 2026

Read Large Language Models for Sequential Decision-Making: Improving In-Context Learning via Supervised Fine-Tuning by Minmin Zhang, Sina Aghaei, Soroush Saghafian (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-09009.txt.

Introduction

Minmin Zhang, Sina Aghaei, and Soroush Saghafian ask whether large language models can learn to make good choices step by step, not just answer questions. They study settings where an agent sees full, partial, or unclear information about its world, and they train models on examples labeled by an expert.

Fine-tuning, they show, lets a model copy good policies from past records rather than rely only on examples shown in the prompt. Their theory links part of the trained model to estimates of the best long-term value of each action. In tests, fine-tuned models came much closer to the best possible results than prompt-only ones, above all on long and partly hidden tasks.

Catalog record

CREATORMinmin Zhang, Sina Aghaei, Soroush Saghafian
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.09009

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Large Language Models for Sequential Decision-Making: Improving In-Context Learning via Supervised Fine-Tuning Minmin Zhang, Sina Aghaei, Soroush Saghafian Abstract Large language models (LLMs) have shown remarkable in-context learning (ICL) capabilities, yet their potential for sequential decision-making remains underexplored. In this paper, we study the ICL capabilities of LLMs in sequential decision-making settings, including Markov Decision Processes (MDPs), Partially Observable MDPs (POMDPs), and Ambiguous POMDPs (APOMDPs). We fine-tune pretrained LLMs to perform few-shot decision-making directly from offline, oracle-labeled trajectories. Our framework enables flexible imitation of policies through supervised fine-tuning (SFT). Theoretically, we focus on linear MDPs and interpret a fine-tuned attention layer as implicitly estimating optimal Q-functions from in-context data. Building on this interpretation, we derive an end-to-end suboptimality bound for the induced policy that separates the in-context estimation error from the training-length bias. Empirically, across synthetic MDP, POMDP, and APOMDP settings, we find that fine-tuned LLMs achieve substantially smaller optimality gaps than in-context-only and random baselines, with especially large gains in longer-horizon, partially observed, and model-ambiguous environments. Together, these results show that supervised fine-tuning provides an effective route to endowing pretrained LLMs with sequential decision-making capabilities from offline data, which is an important advantage in domains such as healthcare where offline data are abundant. 1 Introduction Sequential decision-making under uncertainty is central to many high-stakes domains, from clinical treatment planning to resource management. In these settings, an agent must choose a sequence of actions over time to maximize cumulative rewards, using only observational data collected under some historical policy. Dynamic Treatment Regimes (DTRs) provide a principled framework for discovering effective policies from such data [ 21 , 19 ] . However, standard DTR approaches rely on the strong assumption that all confounders are observable or that their effects can be ignored (sequential ignorability), which is often violated in practice. To address this challenge, Ambiguous Dynamic Treatment Regimes (ADTRs) account for unobservable confounders by evaluating policies against a set of plausible data-generating models (model ambiguity) rather than a single one [ 23 , 24 , 25 ] . Furthermore, previous studies such as [ 24 ] connect ADTRs to APOMDPs and develop reinforcement learning (RL) approaches to find the best sequence of actions. Compared with MDPs or POMDPs, APOMDPs provide a more realistic framework for learning sequential decision-making from observational data in the presence of unobservable confounders by directly considering the underlying model ambiguity. Large language models have shown remarkable in-context learning capabilities. Given a small set of demonstrations as the input prompt, a pretrained LLM can make predictions on new instances without updating its parameters [ 1 ] . This ability has been studied in supervised learning [ 9 , 30 ] and, more recently, in sequential decision-making settings [ 3 , 13 , 14 ] . However, prior work on in-context learning for sequential decision-making mainly trains transformer architectures from scratch, focuses on fully observable MDPs, and does not consider the real-world challenges caused by partial observability or model ambiguity. In this study, we adapt supervised fine-tuning of pretrained LLMs to sequential decision-making in MDPs, POMDPs, and APOMDPs. Rather than training a transformer from scratch, we fine-tune an open-source pretrained LLM so that it can first read a few demonstration trajectories and then act in new, unseen tasks without any further parameter updates. Building on pretrained LLMs leverages the broad representational capacity acquired during pretraining, which may improve generalization across diverse tasks. Starting from pretrained LLMs also provides an avenue for integrating diverse inputs and human intuition inclusion in real-world sequential decision-making applications, augmenting the capabilities of human-AI centaurs [ 25 , 22 ] . Our main contributions in this work are as follows. • SFT framework for in-context sequential decision-making. We fine-tune a pretrained LLM on offline trajectories so that it can make sequential decisions in new tasks given only a few demonstrations at test time. The framework uses parameter-efficient adapters and trains on serialized trajectories. At test time, this approach allows the model to read few-shot demonstrations and act in new tasks without any further parameter updates. • Offline data use with oracle supervision. Our framework leverages offline datasets along with an optimal (or near-optimal) action oracle to synthesize high-quality trajectories for fine-tuning. This practice is particularly appealing in domains such as healthcare, where experimentation is costly (or unethical) and model-based or expert-derived supervision can often be constructed from logged observational data. • Theoretical analysis in linear MDPs. Adopting the in-context linear-regression framework of [ 30 ] , we interpret a trained linear self-attention layer as implicitly estimating the parameters of the optimal Q-function for a new task. Our theoretical contribution (a) makes use of this prediction-level interpretation to create an end-to-end suboptimality bound for the induced policy and (b) decomposes the bound into an in-context estimation error (decreasing in the number of support trajectories) and a training-length bias (decreasing in the number of training trajectories). • Empirical evaluation across MDPs, POMDPs, and APOMDPs. We show that fine-tuned LLMs achieve low optimality gaps and consistently outperform both random policies and ICL-only baselines. Specifically, we find that fine-tuning roughly halves the optimality gap in long-horizon MDPs. In POMDPs and APOMDPs, fine-tuned LLMs achieve large gains even under partial observability and model ambiguity, suggesting robust performance for our approach across different settings. 2 Related work In-context prediction. Existing research has investigated the ICL capabilities of LLMs in classification and regression tasks and shown that transformer-based architectures can learn general function classes when trained on in-context prompts [ 9 , 30 , 18 , 32 ] . Some studies (see, e.g., [ 30 ] ) provide theoretical guarantees for learning linear regression in-context. LLMs have also been applied to time series analysis (see, e.g., [ 31 ] for a detailed survey). LLMs as decision-making agents. Some studies have explored using LLMs directly for decision-making through prompting. [ 28 ] generates both reasoning traces and task-specific actions in an interleaved manner, and [ 10 ] uses language feedback for embodied planning. [ 20 ] trains a single generalist transformer across vision, language, and control tasks. These approaches either use pretrained models directly or train models from scratch. In contrast, we fine-tune pretrained LLMs for few-shot sequential decision-making. In-context decision-making. Decision Transformer was among the first to cast RL as sequence modeling, training a transformer to predict actions conditioned on past states, actions, and a target return [ 3 ] . Algorithm Distillation instead trains transformers to imitate the behavior of RL algorithms from their learning histories [ 13 ] . Decision-Pretrained Transformer (DPT) builds on these ideas and trains a transformer to imitate an optimal action oracle given in-context data [ 14 ] . Pretrained LLM representations are also used as input to added layers for decision-making under partial observability [ 16 ] , with partial observability referring to limited visual input rather than noisy observations of latent states. A survey of related studies in this stream can be found in [ 2 ] . Our study differs from these approaches in two key aspects. First, we fine-tune a pretrained LLM rather than training a transformer from scratch, preserving the representational capacity acquired during pretraining. Second, we extend the evaluation from MDPs to POMDPs and APOMDPs, more general settings with partial

Read the complete text