Mechanistic Interpretability of GPT-like Models on Summarization Tasks
Anurag Mishra · 2025
Read Mechanistic Interpretability of GPT-like Models on Summarization Tasks by Anurag Mishra (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17073.txt.
Description
Anurag Mishra studies how GPT-like models handle summary tasks in a 2025 paper on the inner work of language models. Most work of this kind looks at sorting or text making, not at summaries. Mishra compares a base model with a fine tuned one to see what changes inside when a model learns to sum up text.
He tracks attention patterns and inner activity across layers and heads. Middle layers changed the most, and many heads grew more focused, which points to tighter selection of key information. He calls the changed set a summarization circuit. Tuning only that circuit with a light method beat standard light tuning and needed fewer training rounds. The work links black box scores to clear inner change.
Catalog record
| CREATOR | Anurag Mishra |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-17073