CanaryBench: Stress Testing Privacy Leakage in Cluster-Level Conversation Summaries

Deep Mehta · 2026

Read CanaryBench: Stress Testing Privacy Leakage in Cluster-Level Conversation Summaries by Deep Mehta (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18834.txt.

Description

Deep Mehta builds CanaryBench, a test for leaks in the short takes that firms make from groups of user chats. Firms group chats by theme and share a short take on each group. The chats stay hid, but the take is shared wide. A take can still hold a rare string from one chat: a name, a code, a place. Such a string can point back to one user like a bird's song in a mine.

The test plants such strings in fake chats and checks if they show in the takes. A leak shows the take tool is not safe to share. Firms need that check before they trust group takes with real chats. The test gives them a way to grade the risk and to pick guards that work. Names and codes of real users are at stake.

Catalog record

CREATORDeep Mehta
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-18834

Browse the full library · About the Tower of Babel