How Do Transformers Model Physics? Investigating the Simple Harmonic Oscillator

Subhash Kantamneni, Ziming Liu, Max Tegmark, Jaroslaw Krzywanski, Marcin Sosnowski, Karolina Grabowska, Dorian Skrobek, Ghulam Moeen Uddin · 2024

Read How Do Transformers Model Physics? Investigating the Simple Harmonic Oscillator by Subhash Kantamneni, Ziming Liu, Max Tegmark, Jaroslaw Krzywanski, Marcin Sosnowski, Karolina Grabowska, Dorian Skrobek, Ghulam Moeen Uddin (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/pmc-11592621.txt.

Description

Subhash Kantamneni and co-authors probe how transformer models handle a basic physics system, the simple harmonic oscillator. Their 2024 study asks whether the models use methods people can recognize, or invent an alien physics of their own. The oscillator, a mass on a spring with damping, has exact known solutions for comparison.

The team set four tests for whether a candidate method truly runs inside the model. These ranged from reading the method's halfway values out of hidden states to changing those states and watching predictions shift. All signs pointed to the matrix exponential method, a standard numerical approach. The authors offer their test kit for wider use on harder systems.

Catalog record

CREATORSubhash Kantamneni, Ziming Liu, Max Tegmark, Jaroslaw Krzywanski, Marcin Sosnowski, Karolina Grabowska, Dorian Skrobek, Ghulam Moeen Uddin
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC11592621

Browse the full library · About the Tower of Babel