Collaborative QA using Interacting LLMs. Impact of Network Structure, Node Capability and Distributed Data.

Adit Jain [email protected], Vikram Krishnamurthy [email protected], Yiming Zhang [email protected] · 2025

Read Collaborative QA using Interacting LLMs. Impact of Network Structure, Node Capability and Distributed Data. by Adit Jain [email protected], Vikram Krishnamurthy [email protected], Yiming Zhang [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-14098.txt.

Introduction

Adit Jain, Vikram Krishnamurthy, and Yiming Zhang model teams of language models answering questions together. Each model in a network holds part of the documents needed, so they must share findings to estimate one ground truth. The risk is contagion: when evidence is thin, one model's hallucination can spread and flip previously correct partners.

The authors build a generative model that mixes mean-field dynamics from network science with a randomized utility model from economics. Each model carries a latent truthful-or-not state, and the team derives conditions for a unique stable outcome. They analyze how that outcome shifts with incentives such as extra test-time computing given to individual models.

Catalog record

CREATORAdit Jain [email protected], Vikram Krishnamurthy [email protected], Yiming Zhang [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-14098

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Collaborative QA using Interacting LLMs. Impact of Network Structure, Node Capability and Distributed Data. Adit Jain [email protected], Vikram Krishnamurthy [email protected], Yiming Zhang [email protected] Abstract In this paper, we model and analyze how a network of interacting LLMs performs collaborative question-answering (CQA) in order to estimate a ground truth given a distributed set of documents. This problem is interesting because LLMs often hallucinate when direct evidence to answer a question is lacking, and these effects become more pronounced in a network of interacting LLMs. The hallucination spreads, causing previously accurate LLMs to hallucinate. We study interacting LLMs and their hallucination by combining novel ideas of mean-field dynamics (MFD) from network science and the randomized utility model from economics to construct a useful generative model. We model the LLM with a latent state that indicates if it is truthful or not with respect to the ground truth, and extend a tractable analytical model considering an MFD to model the diffusion of information in a directed network of LLMs. To specify the probabilities that govern the dynamics of the MFD, we propose a randomized utility model. For a network of LLMs, where each LLM has two possible latent states, we posit sufficient conditions for the existence and uniqueness of a fixed point and analyze the behavior of the fixed point in terms of the incentive (e.g., test-time compute) given to individual LLMs. We experimentally study and analyze the behavior of a network of 100 100 open-source LLMs with respect to data heterogeneity, node capability, network structure, and sensitivity to framing on multiple semi-synthetic datasets. 1 Introduction In May 2025, roughly 50\% of internet articles were generated using the help of large language models (LLMs), up from 20\% in May 2023 according to 29 . The explosion of LLM-generated text leads to this text being used for training LLMs or using it as their context. Therefore, LLMs (explicitly or implicitly) interact with each other to generate content. Further LLMs have demonstrated improved performance when collaborating with each other in a network-like structure with other LLMs for question-answering, programming, and scientific research ( 25 ) . Given the quirks of LLMs (e.g., hallucination, sycophancy), it is crucial to investigate their emergent behavior when interacting with one another. In this paper, we model and analyze how a network of interacting LLMs performs collaborative question-answering (CQA) in order to estimate a ground truth given a distributed set of documents. These distributed documents constitute inputs (referred to as ‘context’) to individual LLMs. In estimating the ground truth, the LLMs are prone to hallucination 1 1 1 We define hallucination as reporting a state estimate which is not substantiated by the context and is not the ground truth. when they are provided with a limited non-informative context. Further, in a network of interacting LLMs, hallucination can be either amplified or mitigated depending on the network structure. We analyze the spread of information between the LLMs with respect to network structure, LLM capability and distributed data given their salient features including limited context window and hallucination. To achieve this, we study interacting LLMs by combining novel ideas of mean-field dynamics from network science and the randomized utility model from economics to construct a useful generative model. Figure 1 illustrates our theoretical approach, which we complemented with experimental results on networks of LLMs performing CQA on semi-synthetic real-world datasets. Figure 1 : This paper proposes an analytical model for a network of interacting LLMs performing state estimation - we model the information diffusion in the network using a mean-field dynamics (MFD) for directed networks and model the utility of an LLM as being sampled from a random utility model (RUM), which allows for a transition model which can be plugged into the MFD. We further empirically analyze the behavior of a network of 100 open-source interacting LLMs for CQA on different semi-synthetic datasets. 1.1 Main Results and Insights for interacting LLMs performing CollaborativeQA 1. Mean-Field Dynamics for Information Diffusion in a Network of LLMs. We study the mean-field dynamics, which a generative model where each LLM is represented as an agent with an estimate of the true underlying state, which evolves over time based on its local context and incentives to the case of directed networks. The agent uses its private observation, as well as information from its neighbors, to update its belief about the true state of the world and output a response. To handle the combinatorial complexity of modeling information diffusion in large networks, we derive an ordinary differential equation (ODE) that approximates the average behavior of agents using a mean-field approach in a directed network. We theoretically and experimentally demonstrate the analytical and predictive capabilities of this model. 2. Randomized Utility Model for Decision Making in LLMs. To specify the probabilities that govern the MFD, we propose a randomized utility model (RUM) to parameterize the choice probabilities of the LLMs. Such a representation enables us to model contextual information, including the authenticity of sources and the consensus formed by the neighbors of the LLMs. The RUM model provides interpretability on how sensitive LLMs are to changes in their beliefs. The parameters of the RUM can be estimated efficiently using a logistic regression. RUM from economics, operating at a lower level of abstraction, combines seamlessly with the mean-field dynamics from network science, operating at a higher level of abstraction, and provides a useful generative model for decision-making among interacting networked LLMs. 3. Empirical Study on Semi-Synthetic and Real Datasets Characterizing Interesting Behavior. Further we experimentally study information diffusion in network of 100 interacting LLMs (e.g. network of 100 LLaMa3-8B initialized with a power law degree distribution) on three widely used datasets, including Fiction Dataset (which constructed using 30 books of Project Gutenberg similar to the NarrativeXL dataset 26 ), Knowledge Cutoff (which we curate using updates on Wikipedia articles), and event-based QA benchmarks (created using news articles from BBC, Reuters and CNN). As analysts with knowledge of the ground truth, we can classify LLMs into three states: hallucination (H), truthful (T), and don’t know (D). Let \boldsymbol{\rho}_{T} be the fraction of LLMs that are in a truthful state after L interactions, and we refer to this as the truthful population state. We derive the following insights: Insight 1: The truthful population state \boldsymbol{\rho}_{T} in a network of LLMs is proportional to the computation used by LLMs (test-time compute) and the base model capabilities of the individual LLMs. Insight 2: The placement of the different types of data (correct, missing, incorrect) in a directed network of LLMs affects the \boldsymbol{\rho}_{T} , which increases when more influential nodes have correct data. Insight 3: The network structure affects the truthful population state \boldsymbol{\rho}_{T} , with power-law distribution showing a promising alternative to chain or tree based network structures widely used right now. The above insights open a new line of research and are instructive for designing networks of interacting LLMs. 1.2 Motivation. Networks of LLMs as a primitive for Collective Intelligence. Networks of LLMs now show up in real-world systems and are an example of collective intelligence. Human interactions lead to an emergent collective intelligent sensing behavior that individuals can not ( 18 ) . Similarly, ensembling in machine learning, and mixture

Read the complete text