Segment-Based Attention Masking for GPTs

Shahar Katz Liran Ringel Yaniv Romano Lior Wolf · 2024

Read Segment-Based Attention Masking for GPTs by Shahar Katz Liran Ringel Yaniv Romano Lior Wolf (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18487.txt.

Description

Shahar Katz, Liran Ringel, Yaniv Romano, and Lior Wolf free a transformer's prompt processing from a constraint it never needed. Causal masking forces each token to attend only backward, mimicking generation. During prefill, though, the whole prompt already exists; pretending otherwise wastes structure. Their segment scheme masks by block instead: a chat's system prompt forms one block, the user's message the next.

Within a block, early tokens may look ahead at later ones, since nothing is being generated yet. Generation after prefill stays conventionally causal. The scheme costs no extra computation. Wired into two leading open model families, it consistently reached state-of-the-art performance in their tests. The insight is bookkeeping, not architecture: the prompt arrives as documents, not as a stream of guesses. Treating input as input unlocks free accuracy.

Catalog record

CREATORShahar Katz Liran Ringel Yaniv Romano Lior Wolf
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-18487

Browse the full library · About the Tower of Babel