Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructs
Manuel Pita · 2026
Read Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructs by Manuel Pita (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-28574.txt.
Description
Manuel Pita, writing alone, separates two virtues a coding instrument can have. When a language model labels a theoretical construct in text exactly as a human would, it is a reliable coder. But it may reach the right label through a shallow correlate the theory behind the construct would reject, and then it is not measuring anything at all.
Pita proposes grain calibration. The construct is decomposed into clause-level components drawn from its theory, each component is tested against the text with extractive evidence, and an explicit theory-derived rule combines the verdicts. Because the rule is stated openly, its workings are themselves evidence. They show which components settled each code. When the code is wrong, they show whether a component was missed or a neighboring construct was mistaken for the target.
Catalog record
| CREATOR | Manuel Pita |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.28574