LLMartini: Seamless and Interactive Leveraging of Multiple LLMs through Comparison and Composition Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY CCS: Human-centered computing Human computer interaction (HCI) CCS: Huma

Yingtian Shi, Jinda Yang, Yuhan Wang, Yiwen Yin, Haoyu Li, Kunyu Gao, Chun Yu · 2025

Read LLMartini: Seamless and Interactive Leveraging of Multiple LLMs through Comparison and Composition Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY CCS: Human-centered computing Human computer interaction (HCI) CCS: Huma by Yingtian Shi, Jinda Yang, Yuhan Wang, Yiwen Yin, Haoyu Li, Kunyu Gao, Chun Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19252.txt.

Introduction

Yingtian Shi and six co-authors present LLMartini, an interactive system for using several large language models together without juggling separate windows. Users often compare models or stitch their answers by hand, which is slow and mentally taxing, and good material from each model can get lost in the process.

The system splits responses into aligned segments, merges points where models agree, and color-codes differences while keeping unique contributions visible. In a study with eighteen users, it clearly beat manual workflows. The design treats model diversity as a resource to compose, not a burden to manage.

Catalog record

CREATORYingtian Shi, Jinda Yang, Yuhan Wang, Yiwen Yin, Haoyu Li, Kunyu Gao, Chun Yu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-19252

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLMartini: Seamless and Interactive Leveraging of Multiple LLMs through Comparison and Composition Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY CCS: Human-centered computing Human computer interaction (HCI) CCS: Human-centered computing Interactive systems and tools CCS: Human-centered computing User models Yingtian Shi, Jinda Yang, Yuhan Wang, Yiwen Yin, Haoyu Li, Kunyu Gao, Chun Yu Abstract . The growing diversity of large language models (LLMs) means users often need to compare and combine outputs from different models to obtain higher-quality or comprehensive responses. However, switching between separate interfaces and manually integrating outputs is inherently inefficient, leading to a high cognitive burden and fragmented workflows. To address this, we present LLMartini, a novel interactive system that supports seamless comparison, selection, and intuitive cross-model composition tools. The system decomposes responses into semantically-aligned segments based on task-specific criteria. It automatically merges consensus content and highlights model differences through color coding, all the while preserving unique contributions. In a user study (N=18), LLMartini significantly outperformed conventional manual methods across all measured metrics, including task completion time, cognitive load, and user satisfaction. Our work highlights the importance of human-centered design in enhancing the efficiency and creativity of multi-LLM interactions and offers practical implications for leveraging the complementary strengths of various language models. Figure 1 . LLMartini is a novel interactive system that supports seamless comparison, selection, and intuitive cross-model composition. The three key features of LLMartini are: Task-based purpose recognition and model results fusion; two enhanced interfaces for results comparison and fusion; and flexible fusion tools and functions. 1. Introduction The rapid advancement of powerful large language models (LLMs) has fundamentally transformed how users access information and accomplish tasks ( Bommasani et al. 2022 ; ope 2024 ) . Today, a growing diversity of models is available, each exhibiting unique strengths in areas such as factual accuracy, creative generation, or stylistic variation ( Liang et al. 2023 ; Bommasani et al. 2023 ) . Faced with this expanding ecosystem, users often lack a clear understanding of each model’s specific capabilities. This uncertainty has prompted users to adopt a new strategy: querying multiple LLMs with the same prompt, then comparing and combining their outputs to produce higher-quality or more comprehensive responses. Empirical observations ( Jakesch et al. 2023 ) and user studies (Section 3 ) indicate that this is not a niche activity. In fact, 93% of surveyed users actively compare outputs across different models and often engage in manual fusion, such as copying a paragraph from one model and combining it with a table from another, to construct a unified response. Despite its practical benefits, this process remains highly inefficient. It requires users to constantly switch between interfaces, mentally track sources and quality of responses, and manually integrate outputs. This fragmented process imposes a significant cognitive load, creates operational friction, and makes the integration of outputs tedious and error-prone ( Sweller 2011 ) . Prior work has established valuable insight for Human-LLM Interaction, but remains largely confined to single-model scenarios ( Rezwana and Maher 2022 ; Zamfirescu-Pereira et al. 2023 ; Virvou et al. 2024 ) . These insights can not be directly used to support the more complex scenario of multi-LLM interaction. Although multi-agent frameworks ( Wu et al. 2023 ; Hong et al. 2023 ) employ multiple models, they typically constrain user interaction within predefined, automated workflows that prioritize system-level optimization over user agency. Similarly, LLM ensemble systems ( Sagi and Rokach 2018 ; Mienye and Swart 2025 ) are designed to produce a single, optimized output through algorithmic fusion. The ensemble system often suppresses the diversity and differences that users seek when manually exploring responses across models. To bridge this gap, we first conducted user studies to identify core needs and derive design principles for multi-LLM interaction. Informed by these findings, we present LLMartini, a novel interactive system designed to support seamless and intuitive comparison, selection, and composition of outputs from multiple LLMs. LLMartini replaces the fragmented multi-tab workflow with a unified interface for accessing diverse models. It introduces a dual-mode interaction paradigm: a visual comparison view for divergent responses and a fusion interface for similar ones. The system automatically segments responses into semantically aligned units, merges consensus content, highlights discrepancies via color coding, and preserves unique contributions. Through this system, users can efficiently select, combine, and refine content while significantly reducing cognitive load. In a user study (N=18), LLMartini significantly outperformed manual workflows across all measured metrics, including task completion time, cognitive load, and user satisfaction. Our work underscores the importance of human-centered design in facilitating efficient and creative multi-LLM interactions and offers practical insights for harnessing the complementary strengths of diverse models. This work makes the following contributions: • We identify key user needs, behaviors, and pain points in multi-LLM interaction and derive actionable design guidelines for future multi-model systems. • We present LLMartini, a novel system centered on a visual comparison and user-controlled fusion paradigm, effectively reducing cognitive load and interaction friction. • We conduct a user study demonstrating that LLMartini significantly improves task efficiency, increases user satisfaction, and promotes model exploration compared to conventional multi-tab workflows. 2. Related Work 2.1. Human-LLM Interaction With the growing capabilities of AI systems such as Large Language Models (LLMs), user interaction with AI has become more frequent and widespread. Human-LLM interaction has emerged as a critical research area in human-centered artificial intelligence. Early studies ( Amershi et al. 2019 ; Xu et al. 2021 ; Bingley et al. 2022 ) provided foundational principles and guidelines for designing interactive AI systems, emphasizing that building effective human-AI partnerships requires enhancing transparency, controllability, and human agency. However, the uncertain capabilities and complex outputs from LLMs pose significant challenges for users ( Yang et al. 2020 ; Li et al. 2023 ; Xu et al. 2022 ) . Researchers have attempted to improve user interaction experiences through various approaches, including enhancing the interpretability and controllability of AI outputs ( Li et al. 2021 ; Virvou et al. 2024 ; Ali et al. 2023 ; Liao et al. 2020 ; Shao et al. 2024 ) , developing collaborative frameworks ( Rezwana and Maher 2022 ) , and optimizing prompt design methods ( Zamfirescu-Pereira et al. 2023 ; Wei et al. 2022 ) . Extensive exploration has also been conducted on user-LLM collaboration in specific contexts ( Guo et al. 2024b ) such as healthcare ( Sharma et al. 2023 ; Lu et al. 2024a ; Wang et al. 2023 ) , education ( Pataranutaporn et al. 2021 ; Järvelä et al. 2023 ; Holstein and Aleven 2021 ) , management ( Raisch and Fomina 2024 ) , writing ( Lee et al. 2022 ) , and programming ( Shi et al. 2024 ; Yang et al. 2025 ) . However, the interaction paradigms proposed in these studies still focus on interaction with a single LLM. Although current research on powerful models like GPT-4 highlights their broad capabilities, it also underscores that no single model is universally suitable for all tasks ( Bubeck et al. 2023 ; Reicher et al. 2024 ) . Interacting with only one model limits users’ ability to leverage the strengths of different models and

Read the complete text