Segment-Based Attention Masking for GPTs
Shahar Katz Liran Ringel Yaniv Romano Lior Wolf · 2024
Read Segment-Based Attention Masking for GPTs by Shahar Katz Liran Ringel Yaniv Romano Lior Wolf (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18487.txt.
Description
Shahar Katz, Liran Ringel, Yaniv Romano, and Lior Wolf free a transformer's prompt processing from a constraint it never needed. Causal masking forces each token to attend only backward, mimicking generation. During prefill, though, the whole prompt already exists; pretending otherwise wastes structure. Their segment scheme masks by block instead: a chat's system prompt forms one block, the user's message the next.
Within a block, early tokens may look ahead at later ones, since nothing is being generated yet. Generation after prefill stays conventionally causal. The scheme costs no extra computation. Wired into two leading open model families, it consistently reached state-of-the-art performance in their tests. The insight is bookkeeping, not architecture: the prompt arrives as documents, not as a stream of guesses. Treating input as input unlocks free accuracy.
Catalog record
| CREATOR | Shahar Katz Liran Ringel Yaniv Romano Lior Wolf |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-18487