CoFineLLM: Conformal Finetuning of Large Language Models for Language-Instructed Robot Planning

Jun Wang, Yevgeniy Vorobeychik, Yiannis Kantaros · 2025

Read CoFineLLM: Conformal Finetuning of Large Language Models for Language-Instructed Robot Planning by Jun Wang, Yevgeniy Vorobeychik, Yiannis Kantaros (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06575.txt.

Introduction

Jun Wang, Yevgeniy Vorobeychik, and Yiannis Kantaros make language-guided robot planners ask for help only when truly unsure. Language models can plan action sequences from instructions, but on long tasks they often sound certain while being wrong. Conformal prediction wraps their answers in sets guaranteed to hold the right action at a chosen confidence, at the cost of frequent human rescue calls.

CoFineLLM fine-tunes the planner with those prediction sets in view, shrinking the sets and cutting interventions. Earlier fine-tuning ignored uncertainty and produced oversized sets, especially at high confidence levels. Robot planning tests showed smaller sets and fewer help requests than rival methods, and hardware trials held up on out-of-distribution tasks.

Catalog record

CREATORJun Wang, Yevgeniy Vorobeychik, Yiannis Kantaros
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-06575

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CoFineLLM: Conformal Finetuning of Large Language Models for Language-Instructed Robot Planning Jun Wang, Yevgeniy Vorobeychik, Yiannis Kantaros Abstract Large Language Models (LLMs) have recently emerged as planners for language-instructed agents, generating sequences of actions to accomplish natural language tasks. However, their reliability remains a challenge, especially in long-horizon tasks, since they often produce overconfident yet wrong outputs. Conformal Prediction (CP) has been leveraged to address this issue by wrapping LLM outputs into prediction sets that contain the correct action with a user-defined confidence. When the prediction set is a singleton, the planner executes that action; otherwise, it requests help from a user. This has led to LLM-based planners that can ensure plan correctness with a user-defined probability. However, as LLMs are trained in an uncertainty-agnostic manner, without awareness of prediction sets, they tend to produce unnecessarily large sets, particularly at higher confidence levels, resulting in frequent human interventions limiting autonomous deployment. To address this, we introduce CoFineLLM (Conformal Finetuning for LLMs), the first CP-aware finetuning framework for LLM-based planners that explicitly reduces prediction-set size and, in turn, the need for user interventions. We evaluate our approach on multiple language-instructed robot planning problems and show consistent improvements over uncertainty-aware and uncertainty-agnostic finetuning baselines in terms of prediction-set size, and help rates. Finally, we demonstrate robustness of our method to out-of-distribution scenarios in hardware experiments. \coltauthor Washington University in St. Louis, 1 Brookings Drive St. Louis, MO 63130 keywords Finetuning of LLMs, Conformal Prediction, Language-instructed Robot Planning 1 Introduction Task planning is a fundamental capability in robotics, enabling autonomous agents to design sequences of actions to accomplish high-level missions. Such missions are typically specified using formal languages ( 30 ; 16 ; 52 ; 15 ; 42 ) or optimization-based formulations ( 6 ; 17 ; 55 ; 29 ; 43 ) . These approaches, while powerful, require significant user expertise and manual effort to craft correct task specifications. 1 1 1 Our software implementation of CoFineLLM is available at https://cofinellm.github.io . In contrast, natural language (NL) offers a more intuitive interface for specifying robot missions. Motivated by the strong generalization abilities of large language models (LLMs), recent work has introduced a new paradigm of high-level planning from NL instructions ( 1 ; 35 ; 33 ; 11 ; 23 ; 51 ; 26 ; 12 ; 47 ; 19 ; 54 ; 34 ; 27 ; 31 ; 36 ; 49 ) . Despite impressive empirical results, LLM-based planners often hallucinate and generate overconfident yet wrong actions. To mitigate this, recent works have leveraged Conformal Prediction (CP) ( 2 ; 32 ) to quantify uncertainty in LLM-based planners ( 28 ; 44 ) . The planning problem is cast as a sequence of multiple-choice question–answering (MCQA) tasks, where the LLM is queried to select an action from a finite set. Instead of selecting the action with the highest model’s confidence score, these works apply CP to produce a prediction set guaranteed, with user-specified confidence, to contain the correct action. When the set is a singleton, the planner selects the action included in that set; otherwise, it queries a human to disambiguate. These approaches ensure that the designed plans are correct with a user-defined probability. However, the larger this probability is, the larger the prediction sets are resulting in frequent human queries. The resulting help rate, i.e., the fraction of steps requiring user input, can become prohibitively high as LLMs are trained in an uncertainty-agnostic manner, with CP applied only post hoc ( 14 ; 7 ; 53 ) . We address this limitation by introducing CoFineLLM (Conformal Finetuning for LLMs), the first uncertainty-aware fine-tuning framework for LLM-based planners that integrates CP during training. Our method aims to reduce prediction set sizes while preserving CP’s coverage guarantees. The key idea is to “simulate” conformalization during training by incorporating both calibration and prediction steps within mini-batches. We propose a novel loss function that combines a cross-entropy term with a CP-based term penalizing non-singleton prediction sets. To optimize this objective, we employ Low-Rank Adaptation (LoRA) ( 13 ) with a curriculum-based training scheme; however, other fine-tuning methods are compatible. During deployment, CP is applied to the fine-tuned model as in prior works to enable probabilistically correct decision-making ( 28 ; 44 ) . We show empirically that this approach significantly outperforms baselines in terms of help rates. Related Works: (i) Conformal LLM-based Decision Making: Prior works have applied CP to quantify uncertainty of LLMs used to address planning, translation, and general MCQA problems ( 28 ; 18 ; 38 ; 8 ; 41 ; 40 ; 20 ; 4 ; 21 ; 44 ; 45 ; 46 ) . However, CP in these methods is used exclusively at test time, leading to frequent human interventions when prediction sets are non-singleton. A comprehensive survey on how CP has been used to quantify uncertainty of various AI-enabled components can be found in ( 22 ) . (ii): Conformal Training: Several CP-aware training methods have been proposed for neural network classifiers ( 10 ; 3 ; 37 ; 24 ; 50 ) . Efforts towards developing conformal finetuning methods for graph neural networks are presented in ( 48 ) . These works design differentiable loss functions to minimize prediction set size while preserving the CP coverage guarantees. However, unlike this paper, they focus on models addressing single-step tasks (e.g., classification) and, therefore, target single-step prediction-set efficiency. Moreover, our empirical analysis suggests that existing CP-based loss functions, such as those proposed in ( 37 ) , can be sensitive to the quality of the initial model and may not transfer effectively to LLMs. In particular, we observed empirically that when a pre-trained LLM (e.g., Gemma3-1B) is both incorrect and overconfident, these losses can produce near-zero gradients, effectively halting learning. As a result, such approaches may be less suitable for fine-tuning large models that begin far from convergence. Building on these insights, we introduce a new CP-aware fine-tuning framework for sequential, multi-step decision-making tasks with LLMs. Our formulation explicitly mitigates the gradient sensitivity issue by defining a loss function that combines the standard cross-entropy objective with a novel CP regularizer. Contribution: The contribution of this paper can be summarized as follows: (i) We propose CoFineLLM, the first CP-aware fine-tuning framework for LLMs in sequential decision-making tasks. (ii) We instantiate this framework for robot planning and demonstrate substantial reductions in help rates from users compared to both uncertainty-agnostic and conformal-aware fine-tuning methods. (iii) We empirically show that the proposed framework is robust to the performance of the initial model, making it well suited for cases where the model is far from convergence. (iv) We validate the method on hardware experiments, demonstrating robustness in out-of-distribution settings where test-time tasks and environments differ from those used during finetuning and calibration. 2 Problem Formulation Robot Agent and Environment. We consider a single robot equipped with a known skill set \mathcal{S}=\{s_{1},\ldots,s_{S}\} (e.g., ‘move’, ‘turn’, ‘pick up’). The robot operates in a known environment \Omega\subset\mathbb{R}^{d} , d\in\{2,3\} , containing M>0 semantic objects or regions o_{m} , m\in\{1,\dots,M\} . Each object o_{m} is defined by its location and a semantic label (e.g., ‘box’, ‘key’). Applying a skill s_{j}\in\mathcal{S} to an object o_{m} yields an action a(s_{j},o_{m})

Read the complete text