Profit is the Red Team: Stress-Testing Agents in Strategic Economic Interactions

Shouqiao Wang, Marcello Politi, Samuele Marro, Davide Crapis · 2026

Read Profit is the Red Team: Stress-Testing Agents in Strategic Economic Interactions by Shouqiao Wang, Marcello Politi, Samuele Marro, Davide Crapis (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20925.txt.

Introduction

Wang and co-authors propose profit driven red teaming to stress test AI agents in economic settings. Agents now act on outside inputs that rivals can shape for gain. Fixed prompt attack lists miss such adaptive pressure. Their protocol trains an opponent to maximize profit from scalar outcome feedback alone. It needs no judge model or attack labels. They test it on four classic economic interactions. Agents that look strong against static attacks prove easy to exploit. Rules distilled from the failures then block most of them.

Catalog record

CREATORShouqiao Wang, Marcello Politi, Samuele Marro, Davide Crapis
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60376
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.20925

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Profit is the Red Team: Stress-Testing Agents in Strategic Economic Interactions Shouqiao Wang, Marcello Politi, Samuele Marro, Davide Crapis Abstract As agentic systems move into real-world deployments, their decisions increasingly depend on external inputs such as retrieved content, tool outputs, and information provided by other actors. When these inputs can be strategically shaped by adversaries, the relevant security risk extends beyond a fixed library of prompt attacks to adaptive strategies that steer agents toward unfavorable outcomes. We propose profit-driven red teaming , a stress-testing protocol that replaces handcrafted attacks with a learned opponent trained to maximize its profit using only scalar outcome feedback. The protocol requires no LLM-as-judge scoring, attack labels, or attack taxonomy, and is designed for structured settings with auditable outcomes. We instantiate it in a lean arena of four canonical economic interactions, which provide a controlled testbed for adaptive exploitability. In controlled experiments, agents that appear strong against static baselines become consistently exploitable under profit-optimized pressure, and the learned opponent discovers probing, anchoring, and deceptive commitments without explicit instruction. We then distill exploit episodes into concise prompt rules for the agent, which make most previously observed failures ineffective and substantially improve target performance. These results suggest that profit-driven red-team data can provide a practical route to improving robustness in structured agent settings with auditable outcomes. 1 Introduction Agentic systems are moving beyond single-turn prompting into deployments where an agent’s behavior unfolds over multiple steps and depends on external inputs such as retrieved content, tool outputs, web observations, and other environment signals [ 19 ] . In practice, agents browse websites, retrieve documents, call APIs, use tools, coordinate with other agents, negotiate terms, compete for resources, and carry out workflows on behalf of users. Recent benchmarks reflect this shift by evaluating end-to-end behavior in realistic environments with delayed feedback rather than isolated prompt-response pairs [ 10 , 24 , 22 ] . As a result, the security problem also changes. In the wild, the main risk is often not that an agent fails on a known test prompt, but that strategically motivated parties learn how to shape its effective environment and steer it toward harmful or attacker-beneficial outcomes. This raises a central question: how vulnerable are agents to adaptive, profit-seeking pressure? Rather than treating this as a checklist-style attack problem, we study it as strategic exploitation in a structured environment and ask whether a motivated adversary or counterparty can reliably learn to exploit a target agent over time. Such pressure does not arise only through direct dialogue. It can also operate through retrieved context, manipulated web content, tool-visible state, selective disclosure, or other inputs that affect the agent’s decisions. This perspective motivates profit-driven red teaming , a protocol for measuring systematic exploitability under adaptive pressure and for converting discovered failures into practical hardening signals. This framing highlights a mismatch with common LLM security practice. Most evaluations emphasize specific attack patterns, such as prompt injection and jailbreak-style inputs [ 12 ] . These evaluations are valuable, but they can be an incomplete proxy for deployment risk in strategically influenceable settings. A real adversary or counterparty is adaptive and self-interested: it does not commit to a single trigger string, but probes, adapts, and opportunistically exploits whatever the agent reveals. If framing, anchoring, feigned constraints, or selective disclosure improve outcomes, those tactics will be used. If the agent can be induced to reveal private information, accept unfavorable terms, or follow untrusted instructions, adaptive pressure will search for a way to trigger it. Even interactions that would not typically be labeled malicious, such as bargaining aggressively for a better outcome, are admissible strategies. What matters, then, is not whether an agent passes a fixed checklist of attacks, but whether it remains robust when an adaptive opponent searches for any strategy that improves its return. This aligns with recent LLM security work emphasizing that defenses must be evaluated under adaptive attacks, not only static prompt sets [ 13 , 4 , 11 , 15 ] . To close this evaluation gap, we seek stress tests that (i) operate under an explicit environment contract and (ii) return an auditable outcome signal, so that adaptive pressure can be modeled and iterated on transparently. This setup naturally covers not only the economic interactions studied here, but also broader agent settings, such as web or coding tasks, in which an adversary can strategically shape the agent’s observations and derive value from the resulting behavior. We therefore propose profit-driven red teaming . Instead of curating attack prompts or relying on an external judge to label transcripts as injection, deception, or policy violations, we optimize an opponent whose only training signal is the realized interaction outcome. Concretely, the opponent is improved by automated strategy search over repeated play, guided solely by the scalar payoff returned by the environment. The protocol uses no attack taxonomy and requires no transcript annotation. As in black-box automated attack search methods such as Tree of Attacks with Pruning (TAP) and AutoDAN [ 11 , 15 ] , iterative search can surface effective prompting and interaction strategies, but here selection is driven purely by profit: any behavior that reliably improves the opponent’s outcome is retained, whether it resembles prompt injection, targeted elicitation, deceptive commitments, or other opportunistic manipulation. The profit signal effectively replaces the judge, and the learned opponent replaces a handcrafted attack library. Using this protocol, we instantiate profit-driven red teaming in a lean, controlled arena that runs repeated multi-player games, logs full transcripts, and scores outcomes in a machine-checkable fashion. To keep the evidence minimal yet concrete, we study four canonical economic interactions: ultimatum bargaining, first-price auctions, bilateral trade, and a provision-point public goods game. These games are intentionally simple, but they expose in especially clean form the ingredients we care about: strategic influence, private information, repeated adaptation, and clear auditable outcome metrics. In small-scale experiments, we observe a consistent pattern. Agents that appear strong against static baselines become reliably exploitable when facing an adaptive, profit-optimized opponent. The opponent often discovers probing, anchoring, and deceptive commitments without being explicitly instructed to do so. We then show that the data produced by this protocol is directly useful for hardening: attacker episodes that reveal exploitable behavior in the target agent can be distilled into prompt rules, which disable many previously successful attacks and deliver large target-side gains. Our contributions are: • We frame a central deployment risk for agents in structured settings with auditable outcomes as adaptive exploitability under profit-seeking pressure, rather than success against a fixed set of predefined attacks. • We propose an opponent-based stress-testing protocol where an automatically optimized, profit-maximizing opponent searches for any strategy that improves its profit using only outcome feedback, without an LLM judge or handcrafted attack library [ 11 , 15 , 17 ] . • We implement a lightweight arena with four strategic economic games and provide empirical evidence that adaptive opponents can systematically exploit otherwise strong agents. • We show that episodes that reveal exploitable target agent behavior produced by profit-driven red teaming can be distilled into prompt rules: many attacks that previously exploited the target agent fail after hardening, with strong improvements in target-side outcomes. Overall, our work argues that security evaluation for agents in

Read the complete text