Model Behavior Specification by Leveraging LLM Self-Playing and Self-Improving DOI: XXXXXXX.XXXXXXX Conference: -; ; - ISBN: -

Soya Park, J.D. Zamfirescu-Pereira, Chinmay Kulkarni · 2025

Read Model Behavior Specification by Leveraging LLM Self-Playing and Self-Improving DOI: XXXXXXX.XXXXXXX Conference: -; ; - ISBN: - by Soya Park, J.D. Zamfirescu-Pereira, Chinmay Kulkarni (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-03967.txt.

Introduction

Soya Park, Zamfirescu-Pereira, and Chinmay Kulkarni present Visionary Tuning, a system that helps people write clear behavior rules for chatbots. Telling a model how to act in plain words sounds easy, yet people struggle to state what they want. Their system lets the chatbot play-act talks with itself, which surfaces patterns a user can then name.

In a study with twelve users, self-play helped people spot more behavior patterns and write better prompts than working alone. A larger crowd check with sixty people found the tuned chatbot followed the rules without losing quality. A second test on movie ratings showed the method can also learn one critic's taste, from low to high scores.

Catalog record

CREATORSoya Park, J.D. Zamfirescu-Pereira, Chinmay Kulkarni
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-03967

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Model Behavior Specification by Leveraging LLM Self-Playing and Self-Improving DOI: XXXXXXX.XXXXXXX Conference: -; ; - ISBN: - Soya Park, J.D. Zamfirescu-Pereira, Chinmay Kulkarni Abstract . Training AI models is challenging, particularly when crafting behavior instructions. Traditional methods rely on machines (supervised learning) or manual pattern discovery, which results in not interpretable models or time sink. While Large Language Models (LLMs) simplify instruction writing through natural language, articulating intended model behavior still remains difficult. We introduce Visionary Tuning , a human-in-the-loop self-playing followed by automatic self-refinement to improve behavior specification. Our system helps users clarify desired behavior through self-playing and generates prompts through self-improving, Our first evaluation involves user study conducted on a system implementation of Visionary Tuning within the context of chatbot behavior. Our system self-play itself by simulating user interactions to identify patterns and create effective prompts based on the pattern. In a within-subject study (N=12), participants pinpointed more patterns through self-playing and crafted better prompts. Surprisingly, users felt more or less success level in specifying the model behavior. Follow-up crowd studies (N=60) confirmed that the chatbot adhered to instructions without sacrificing quality. Our second evaluation is a case study on a real-world implementation using a movie rating dataset with Visionary Tuning, demonstrating its effectiveness and robustness in modeling a critic’s preferences across the spectrum of low to highly rated movies. Together, these results suggest how AI improves the design process of interactive AI systems. Furthermore, they suggest how the benefits of these tools may be non-obvious to end-users. We reflect on these findings and suggest future directions. Figure 1 . Visionary Tuning allows developers to craft model instructions by letting them focus on the high-level behavior articulation through two components: Self-playing phase : a target model interacts with itself under the monitoring and supervision of the model developer. In this phase, the goal is to surface diverse criteria to probe users’ training intentions. Self-improving phase : Based on the user feedback, we use LLM to articulate instructions for the target model. 1. Introduction Training Artificial Intelligence (AI) models in open-ended domains is challenging, primarily due to the difficulty in crafting instructions that precisely describe the model behavior ( Pitis et al. 2024 ) . In supervised learning, developers often rely on machines to discover behavior patterns within the dataset, though the results may not be easily interpretable. Outside of supervised learning, developers work with smaller datasets and manually identify patterns. For example, model developers frequently spend extensive time in meetings, analyzing intricate and subtle patterns by focusing on a handful of examples, so-called example-driven conversation, before implementing the model ( Park et al. 2021 ) . This manual pattern discovery process is both labor-intensive and time-consuming ( Piorkowski et al. 2021 ) . Prompt engineering simplifies the process of creating LLM instructions by enabling developers to craft prompts that guide its behavior. This approach leverages the natural language understanding capabilities of AI models, allowing for a more intuitive setup of agent functionalities ( Bommasani et al. 2021 ) . Developers can shape the model behavior and responses to fit a wide range of use cases, from customer service and support to personalized tutoring and beyond ( Park and Kulkarni 2023 ) . Furthermore, prompt engineering reduces technical barriers traditionally associated with training machine-learning models, making such opportunities accessible for individuals without programming or machine-learning skills. Despite its potential to streamline development, prompt engineering continues to face the same challenges: articulation and implementation. Expressing an idea clearly is difficult, and translating that articulation into an effective prompt is even harder. For example, personal preference in movies is hard to describe. Even within a preferred genre, subtle factors—such as directing style, casting choices, and thematic elements—play a significant role (even multi-decade movie critics often struggle to pinpoint exactly what makes a film resonate with them ( Ebert 2024 ) ). Even when articulation is successful, implementing prompts that reliably capture intended behavior remains challenging. Some such unexpected behaviors can lead to extremely large real-world consequences, such as promising some benefits to customers that are not aligned with a company’s policy ( air 2024 ) . Consequently, engineers often engage in a process of brute-force trial and error, refining prompts based on feedback loops from (expensive) real-world interactions ( Zamfirescu-Pereira et al. 2023a ; Bai et al. 2022 ) . For example, LLMs often struggle with interpreting and adhering to negative instructions, so-called anti-behaviors that specify what not to do or topics not to discuss ( OpenAI 2024 ) . The inability of the model to deliberately exclude certain topics or information from its responses prevents engineers from tailoring content to specific user needs, or even enhancing content safety, or ensuring compliance with privacy guidelines or content restrictions. In this paper, we introduce Visionary Tuning, a model behavior specification method inspired by the metaphor of Visionary leadership. Visionary Tuning positions the model developer as a visionary leader—akin to a leader within an organization—who has a vision that their team (the model) executes to bring that vision to life. Unlike typical reinforcement learning, which relies on abundant data and an explicit environment for self-playing to iteratively update behaviors, Visionary Tuning operates effectively in scenarios where these training data are sparse or unavailable. For example, when dealing with personal preferences, such as a user’s taste in movies—an area often limited by a lack of comprehensive datasets or shaped by implicit signals ( Andukuri et al. 2024 ; Pitis et al. 2024 ) – Visionary Tuning first presents the user with curated instances through self-playing . Users share their preferences (e.g., ratings), and Visionary Tuning crafts model instructions based on this input ( self-improving phase ), tailoring the model’s behavior to align with the user’s intent. We first developed VisionForge based on Visionary Tuning. As a challenging test for Visionary Tuning, we instantiated prompt engineering for anti-behavior. Avoiding specific outputs via prompts has historically challenged LLMs ( Zamfirescu-Pereira et al. 2023b ) . Before RLHF-based instruction tuning ( Ouyang et al. 2022 ) , LLMs operated as “primed language models”, where input tokens strongly influenced output, even with negative instructions. While RLHF has improved negative prompting, effective avoidance remains difficult ( Park and Kulkarni 2023 ) . VisionForge iteratively enhances specifications through LLM self-play , enabling users to identify diverse conversation flows that may trigger existing model biases and recommend refinements to user instructions based on these discoveries, leveraging self-improvement for crafting prompts. Our findings from within-subjects study (N=12) on VisionForge indicate that participants were able to effectively utilize self-play and explore the domain of the given tasks approximately two times as broadly when using VisionForge ( p =.001). Additionally, participants were able to use VisionForge self-improvement and write more reliable and robust prompts; chatbots successfully avoided the user-defined anti-behavior 22% more effectively ( p =.02) with significantly low variability in performance. Despite the improved performance, however, participants did not perceive a benefit from the use of VisionForge , and believed they were equally successful at accomplishing the task when using VisionForge . Our supplementary study (N=60) confirms that the VisionForge chatbot was robust in topic avoidance ( p < .001) without comprising the chatbot conversation quality (e.g. helpfulness, accuracy). To further show technical

Read the complete text