SEMbeddings: how to evaluate model misfit before data collection using large-language models

Tommaso Feraco, Enrico Toffalini · 2025

Read SEMbeddings: how to evaluate model misfit before data collection using large-language models by Tommaso Feraco, Enrico Toffalini (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/pmc-11832509.txt.

Description

Tommaso Feraco and Enrico Toffalini offer a way to check a questionnaire's structure before collecting any answers. Their tool, SEMbeddings, uses a language model trained on personality text to guess how test items will correlate, from the items' wording alone. Their study, whose head dates it to 2024, tests the idea on a 96-item scale of character strengths answered by nearly 32,000 people.

Guessed and real correlations matched at a level of 0.67, close but not exact. Fitting factor models to the guessed matrix then flags items that may not fit their scale, through standard warning indices. The authors warn that the usual fit cutoffs mislead here, tending to judge too harshly. Used as a screening step, though, the warnings can help designers fix weak items early. They expect gains as language models sharpen.

Catalog record

CREATORTommaso Feraco, Enrico Toffalini
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC11832509

Browse the full library · About the Tower of Babel