Mechanistic Interpretability of GPT-like Models on Summarization Tasks

Anurag Mishra · 2025

Read Mechanistic Interpretability of GPT-like Models on Summarization Tasks by Anurag Mishra (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17073.txt.

Description

Anurag Mishra studies how GPT-like models handle summary tasks in a 2025 paper on the inner work of language models. Most work of this kind looks at sorting or text making, not at summaries. Mishra compares a base model with a fine tuned one to see what changes inside when a model learns to sum up text.

He tracks attention patterns and inner activity across layers and heads. Middle layers changed the most, and many heads grew more focused, which points to tighter selection of key information. He calls the changed set a summarization circuit. Tuning only that circuit with a light method beat standard light tuning and needed fewer training rounds. The work links black box scores to clear inner change.

Catalog record

CREATORAnurag Mishra
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17073

Browse the full library · About the Tower of Babel