Reproducibility Study of Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation
Jose L. García, Karolína Hájková, Maria Marchenko, Carlos Miguel Patiño · 2025
Read Reproducibility Study of Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation by Jose L. García, Karolína Hájková, Maria Marchenko, Carlos Miguel Patiño (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-16242.txt.
Introduction
Language models can play stakeholders haggling in negotiation games, and an earlier study charted how they cooperate, compete, or act badly. Jose Garcia and co-authors rerun and extend that study in this 2025 reproducibility paper, testing open models from small to large plus a commercial mini model.
Small models often broke format or rambled, while large open models came close to the commercial one. Surprisingly, a single agent working alone often matched full multi-agent talks, which questions how much the talking itself adds. The team also probes fairness, information leaks in replies, and the cost of running such talks, offering a fuller picture of model negotiation.
Catalog record
| CREATOR | Jose L. García, Karolína Hájková, Maria Marchenko, Carlos Miguel Patiño |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-16242
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reproducibility Study of Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation Jose L. García, Karolína Hájková, Maria Marchenko, Carlos Miguel Patiño Abstract This paper presents a reproducibility study and extension of "Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation." We validate the original findings using a range of open-weight models (1.5B-70B parameters) and GPT-4o Mini while introducing several novel contributions. We analyze the Pareto front of the games, propose a communication-free baseline to test whether successful negotiations are possible without agent interaction, evaluate recent small language models’ performance, analyze structural information leakage in model responses, and implement an inequality metric to assess negotiation fairness. Our results demonstrate that smaller models (<10B parameters) struggle with format adherence and coherent responses, but larger open-weight models can approach proprietary model performance. Additionally, in many scenarios, single-agent approaches can achieve comparable results to multi-agent negotiations, challenging assumptions about the necessity of agent communication to perform well on the benchmark. This work also provides insights into the accessibility, fairness, environmental impact, and privacy considerations of LLM-based negotiation systems. 1 Introduction The rapid advancement of Large Language Models (LLMs) has opened new opportunities for Artificial Intelligence (AI) applications, particularly in the form of autonomous AI agents. These agents, capable of interacting and communicating with one another, can be especially useful in negotiation systems. Negotiations involve complex multi-agent interactions that require resolving multi-issue scenarios with little to no supervision, so it is critical to evaluate whether these agents are reliably performing their intended tasks ( Shavit et al., 2023 ) . Within this context, reproducibility studies are valuable, as they help validate benchmarks the AI community can use to evaluate the performance of current and future models ( Reuel et al., 2024 ; Chang et al., 2023 ) . This paper aims to reproduce the results of an LLM testbed specially designed for negotiation games ( Abdelnabi et al., 2024 ) to contribute to developing robust benchmarks. Our reproducibility goal is to assess whether the benchmark effectively evaluates the models’ negotiation performance and whether the Chain-of-Thought (CoT) prompt configurations contribute to successful deal-making, especially in terms of accountability ( Chan et al., 2024 ) and confidentiality. To achieve this, we compared different open-weight models against a state-of-the-art (SOTA) closed-source model. Building on the original paper, we analyze the Pareto front of the proposed games and establish a strong baseline to test whether an agent can successfully propose a deal without communicating with other parties. We also present extensions that address the environmental impact, accessibility, fairness, and confidentiality. For environmental impact and accessibility, we use the benchmark to evaluate open-weight models and compare their performance to larger, more energy-consuming ones. Smaller models reduce the computational resource requirements ( Fu et al., 2024 ; Sinha et al., 2024 ) , and we compare the performance between models that range from 1.5B to 70B parameters. For fairness, we introduce an inequality metric to assess whether all parties benefit equally from the negotiation. For confidentiality, we compare open-weight models with a closed-source model that requires access through an external API and may expose attack surfaces that can be exploited by malicious actors ( Evertz et al., 2024 ; Dunn et al., 2024 ) . 2 Scope of Reproducibility We focus our reproduction efforts on investigating the following claims from the paper: 1. Open-weight models fall slightly behind SOTA closed-source models. 2. Small models—i.e., models with 8B parameters or less—are not suitable for the negotiation benchmark because they cannot follow the format of the game. 3. Agents have better outcomes when using CoT structures in their prompts. 4. The agents’ incentives in the benchmark result in non-zero-sum games. 5. The benchmark measures cooperation, communication, and negotiation skills. The code and instructions for reproducing the results of this work, along with our additions and the fixes mentioned in Section A.7 , are available on Github 1 1 1 https://anonymous.4open.science/r/llm_negotiation-E731/README.md . 3 Original Paper 3.1 Benchmark and Base Game The original paper ( Abdelnabi et al., 2024 ) introduces a benchmark to evaluate whether LLMs can successfully reach agreements in a multi-agent, multi-issue, semantically rich negotiation game over multiple rounds. The benchmark is designed to test agents in a dynamic, multi-turn scenario that requires advanced reasoning and strategic decision-making. Specifically, it assesses their ability to reach agreements through arithmetic calculations, inference, exploration, and planning, in addition to essential skills such as communication, collaboration, and instruction following. The benchmark’s base negotiation game is adapted from HarborCo ( Susskind, 1985 ) , a game traditionally used to teach negotiation skills involving six parties negotiating over five key issues related to constructing a sports complex. Each party is assigned a name and a role with different goals and preferences for each issue. That way, every issue is presented with three to five options, each assigned with a score for the evaluating agent. A deal proposal is agreeable for an agent if the sum of the chosen options across all issues meets or exceeds its minimum utility score threshold. Before the game begins, all agents are given a global context outlining the negotiation setup, including information about the issues and the other agents’ assigned names, roles, objectives, and preferences. The game then encourages agents to cooperate and compromise when necessary over the negotiation rounds to achieve a successful outcome, as a negotiation is only considered successful if at least five out of the six players surpass their scores threshold and accept the final deal, including Player 1 ( p_{1} ) and Player 2 ( p_{2} ). Player 1 and Player 2 are specially designated roles with veto power: for the final deal to be accepted, it must receive approval from both. p_{1} , referred to as the main negotiator , is also responsible for proposing the negotiation’s first and final deals. As the main negotiator, p_{1} receives a bonus of 10 points upon suggesting a final deal accepted by everyone. The initial deal is provided to p_{1} at the start of the game, serving as the foundation for the follow-up discussions, and the remaining five agents aim to negotiate better deals for themselves. If the parties fail to reach a valid agreement, each agent receives their Best Alternative To a Negotiated Agreement (BATNA) score. BATNA typically corresponds to the minimum utility score threshold of each party but may vary depending on the game variation. 3.2 Game Variations To further expand the benchmark, the authors introduced variations of the base game where a single agent exhibits greedy or adversarial behaviors, influencing interactions and outcomes. In these variations, an agent can be designated as greedy, focusing solely on maximizing its own benefit, or adversarial, actively targeting another agent to sabotage the deal. The adversarial behavior is also divided into two variations: targeted, where the adversarial agent is explicitly assigned a specific target, and untargeted, where the agent independently selects which opponent to sabotage. LLMs are evaluated even further across four difficulty levels: Base, Game 1, Game 2, and Game 3 . These variations adjust the base game’s difficulty, with Games 1 and 3 being more challenging and Game 2 presenting the highest level of difficulty. Each variation introduces a new and unique setup with modified negotiation roles, objectives, and minimum utility thresholds