When Prompt Optimization Becomes Jailbreaking: Adaptive Red-Teaming of Large Language Models

Zafir Shamsi, Nikhil Chekuru, Zachary Guzman, Shivank Garg · 2026

Read When Prompt Optimization Becomes Jailbreaking: Adaptive Red-Teaming of Large Language Models by Zafir Shamsi, Nikhil Chekuru, Zachary Guzman, Shivank Garg (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19247.txt.

Description

Shamsi and co-authors show that tools built to tune prompts for good tasks can be turned into jailbreaks. They use three black-box optimizers from DSPy to refine harmful prompts step by step. A separate model grades each try with a danger score from 0 to 1. On one open model, the score jumps from 0.09 to 0.79 after tuning. Small open models show the weakest guardrails. The team argues that fixed test sets miss this moving risk. Moving checks, they say, must join safety tests.

Catalog record

CREATORZafir Shamsi, Nikhil Chekuru, Zachary Guzman, Shivank Garg
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20529
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.19247

Browse the full library · About the Tower of Babel