SEMbeddings: how to evaluate model misfit before data collection using large-language models
Tommaso Feraco, Enrico Toffalini · 2025
Read SEMbeddings: how to evaluate model misfit before data collection using large-language models by Tommaso Feraco, Enrico Toffalini (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/pmc-11832509.txt.
Description
Tommaso Feraco and Enrico Toffalini offer a way to check a questionnaire's structure before collecting any answers. Their tool, SEMbeddings, uses a language model trained on personality text to guess how test items will correlate, from the items' wording alone. Their study, whose head dates it to 2024, tests the idea on a 96-item scale of character strengths answered by nearly 32,000 people.
Guessed and real correlations matched at a level of 0.67, close but not exact. Fitting factor models to the guessed matrix then flags items that may not fit their scale, through standard warning indices. The authors warn that the usual fit cutoffs mislead here, tending to judge too harshly. Used as a screening step, though, the warnings can help designers fix weak items early. They expect gains as language models sharpen.
Catalog record
| CREATOR | Tommaso Feraco, Enrico Toffalini |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | CC BY 4.0 |
| RIGHTS | CC BY |
| SOURCE | Europe PMC |