Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers
Adam Karvonen, James Chua, [1em] Clément Dumas, Kit Fraser-Taliente, Subhash Kantamneni, Julian Minder, Euan Ong, Arnab Sen Sharma, Daniel Wen, [1em] Owain Evans, Samuel Marks · 2025
Read Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers by Adam Karvonen, James Chua, [1em] Clément Dumas, Kit Fraser-Taliente, Subhash Kantamneni, Julian Minder, Euan Ong, Arnab Sen Sharma, Daniel Wen, [1em] Owain Evans, Samuel Marks (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-15674.txt.
Description
Adam Karvonen and co authors train language models to explain the internal activations of other language models in a 2025 paper. Activations are hard to read, and most interpretation tools are complex and narrow. An earlier approach, LatentQA, trained models to answer questions about activations in plain language, but only in narrow settings.
The team takes a generalist path and calls the resulting models Activation Oracles. Tested far outside their training setup, these oracles recover facts fine tuned into a model, such as biographical knowledge or hidden propensities, that never appear in the input text. The best oracles match or beat white box baselines on all four downstream tasks studied.
Catalog record
| CREATOR | Adam Karvonen, James Chua, [1em] Clément Dumas, Kit Fraser-Taliente, Subhash Kantamneni, Julian Minder, Euan Ong, Arnab Sen Sharma, Daniel Wen, [1em] Owain Evans, Samuel Marks |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 83 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-15674