TAPS: Tool-Augmented Personalisation via Structured Tagging
Ekaterina Taktasheva and Jeff Dalton · 2025
Read TAPS: Tool-Augmented Personalisation via Structured Tagging by Ekaterina Taktasheva and Jeff Dalton (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-20409.txt.
Introduction
Chat agents that book and buy for users rarely ask what the user actually prefers, and Ekaterina Taktasheva and Jeff Dalton show how much that costs. Their 2025 paper finds language models fumbling personal preferences when choosing tools. Their fix tags user tastes in a structured record the agent can consult.
An uncertainty-based detector decides when the tags are needed, so the agent checks preferences exactly when it is unsure. On a standard benchmark of preference-laden dialogues, the method set a new best among open models. The lesson is that personalization in tool use needs explicit structure, not just a longer prompt.
Catalog record
| CREATOR | Ekaterina Taktasheva and Jeff Dalton |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-20409
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TAPS: Tool-Augmented Personalisation via Structured Tagging Ekaterina Taktasheva and Jeff Dalton Abstract Recent advancements in tool-augmented large language models have enabled them to interact with external tools, enhancing their ability to perform complex user tasks. However, existing approaches overlook the role of personalisation in guiding tool use. This work investigates how user preferences can be effectively integrated into goal-oriented dialogue agents. Through extensive analysis, we identify key weaknesses in the ability of LLMs to personalise tool use. To this end, we introduce TAPS, a novel solution that enhances personalised tool use by leveraging a structured tagging tool and an uncertainty-based tool detector. TAPS significantly improves the ability of LLMs to incorporate user preferences, achieving the new state-of-the-art for open source models on the NLSI task 1 1 1 The code is available at github.com/grill-lab/taps . . 1 Introduction Successfully completing complex user tasks through conversation remains a fundamental challenge for goal-oriented dialogue agents. Consider a user interacting with a task assistant to book a last-minute flight. To effectively assist the user, the system must (i) retrieve real-time flight availability, (ii) find the flight that fits user constraints, including airline, layover, and time preferences, (iii) and execute the booking seamlessly, possibly across multiple platforms. Despite their success in many areas, Large Language Models (LLMs) are still unable to fulfil these requirements on their own, and there have been many attempts to address these challenges throughout the years ( Goel et al. 2018 ; Muise et al. 2019 ; Agarwal et al. 2022 ). Recently, a growing number of studies have emerged on tool-augmented language models (TALMs), allowing LLMs to access real-world APIs to perform a wide range of tasks ( Parisi et al. 2022 ; Schick et al. 2023 ) . Tool use has enabled the development of autonomous goal-oriented agents capable of interacting with real-world environments and accessing external data to seamlessly plan and execute complex user tasks ( Mialon et al. 2023 ; Qin et al. 2023 ; Liu et al. 2024a ) . Although there have been efforts to incorporate tool use into conversational agents ( Farn and Shin 2023 ; Li et al. 2023 ; Lu et al. 2024 ) , most of the research in the area neglects conversational history and user preferences. Recognising these can enhance the user experience by tailoring the responses to individual users and improving the relevance and efficiency of task execution, especially in complex and dynamic environments. Moghe et al. 2024 attempt to bridge this gap by introducing the Natural Language Standing Instructions dataset (NLSI). To the best of our knowledge, it is the first work that addresses the problem of personalisation in TALMs, enabling more coherent and context-aware tool use through standing instructions , phrases that prescribe model behaviour based on the specific scenario. While the work provides a strong basis for further research on tool use personalisation, it focuses on dataset construction and provides only simple baselines. In this work, we ask how we can effectively leverage user preferences to personalise and enhance user-agent interactions . We conduct an extensive behavioural analysis of commonly used LLMs on the NLSI dataset and demonstrate their limited ability to accurately infer tool calls in the presence of user preferences, leading to semantic errors, missing arguments, and hallucinations. We hypothesise that introducing a high-quality intermediate representation between natural language and code can significantly enhance model performance and minimise said errors. To this end, we propose TAPS – T ool- A ugmented P ersonalisation via S tructured Tagging, the first solution that leverages a structured tagging tool for data augmentation as well as an internal tool detection mechanism for personalised tool-use in a dialogue setting. Our contributions are: (i) we analyse common LLMs’ performance on the personalised tool-use task and identify their current weaknesses; (ii) we propose structured tagging, an annotation scheme that bridges natural language and API calls by hierarchically marking functions and their arguments within user preferences, (iii) we introduce TAPS, a tuning-free approach that uses a structured tagging tool and an uncertainty-based tool detector to facilitate integration of user preferences into tool-augmented goal-oriented dialogue agents; (iv) we demonstrate that our method improves the effectiveness of LLMs on the task, achieving state-of-the-art results for open-source models on the interpretation subtask of NLSI with an increase of +16.5% in exact match (EM) and +16.9% in F1. Our findings suggest TAPS’s potential for generalisation to other goal-oriented tasks, where reductions in errors such as hallucinations and missing arguments could improve system reliability and user experience. With this work, we hope to inspire future research on tool-use personalisation. 2 Task Setup 2.1 Task Definition Figure 1: Example of the NLSI task. Given a user query and user-specific list of preferences, and API documentation, the model has to parse the input into structured output. The model has to (i) select, which preferences are relevant for the current query and (ii) interpret the utterance into one or several API calls. The diagram is a replica of Figure 1 from Moghe et al. 2024 . The NLSI task is defined as follows. Given a user query, standing instructions, and API documentation, an agent must generate up to three API calls to fulfil the user request ( Figure 1 ). The standing instructions constitute the user profile – their preferences regarding different aspects, e.g., favourite cuisine, preferred airline, or music taste. The task requires complex reasoning to integrate query details with user preferences to generate appropriate API calls. Ultimately, the task consists of two subtasks: selection , identifying the subset of instructions relevant to the current query; and interpretation , generation of API calls to perform the user task using the user query, user profile, and API documentation. This work focuses on the interpretation subtask, which is crucial for improving LLMs’ ability to handle contextualised tool use – a key challenge in real-world applications. Successful interpretation requires an agent to understand the user intent, reason over the conversation and user profile, and identify the appropriate APIs, necessary arguments, and their values. To ensure a controlled evaluation, we provide LLMs with the correct selected standing instructions, allowing them to access only the relevant user profile information. 2.2 Evaluation We follow the evaluation setup, described in Moghe et al. 2024 to assess model performance. We convert each API call into (function name, argument name, value) triplets, or slots, to compute the metrics and report exact match (EM), slot-wise F1, precision, and recall. 2.3 Behaviour Analysis Model Source Size Instr.-Tuned Tools CodeLlama Rozière et al. 2024 7B ✗ ✗ CodeLlama-Inst 7B ✓ ✗ Llama-2 Touvron et al. 2023 7B ✗ ✗ Llama-2-Chat 7B ✓ ✗ Llama-3 Dubey et al. 2024 8B ✗ ✗ Llama-3-Inst 8B ✓ ✗ Mistral-3 Jiang et al. 2023 7B ✗ ✓ Mistral-3-Inst 7B ✓ ✓ OLMo-2-7B-Inst OLMo et al. 2024 7B ✓ ✗ GPT4o OpenAI et al. 2024 unk ✓ ✓ Table 1: LLMs used in our work. The challenge of NLSI is incorporating several aspects: models must not only accurately identify the users’ intended task but also select relevant information from both the current user query and the user profile, and effectively utilise it to generate the appropriate API call. An additional complexity arises from the limited