Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructs

Manuel Pita · 2026

Read Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructs by Manuel Pita (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-28574.txt.

Description

Manuel Pita, writing alone, separates two virtues a coding instrument can have. When a language model labels a theoretical construct in text exactly as a human would, it is a reliable coder. But it may reach the right label through a shallow correlate the theory behind the construct would reject, and then it is not measuring anything at all.

Pita proposes grain calibration. The construct is decomposed into clause-level components drawn from its theory, each component is tested against the text with extractive evidence, and an explicit theory-derived rule combines the verdicts. Because the rule is stated openly, its workings are themselves evidence. They show which components settled each code. When the code is wrong, they show whether a component was missed or a neighboring construct was mistaken for the target.

Catalog record

CREATORManuel Pita
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.28574

Browse the full library · About the Tower of Babel