GenZ: Foundational models as latent variable generators within traditional statistical models
Marko Jojic, Nebojsa Jojic · 2025
Read GenZ: Foundational models as latent variable generators within traditional statistical models by Marko Jojic, Nebojsa Jojic (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-24834.txt.
Introduction
Marko Jojic and Nebojsa Jojic present GenZ, a hybrid that uses foundation models as generators of latent meaning traits inside standard data-based models. Large models hold broad knowledge but miss data set-certain patterns that drive prediction. GenZ finds feature accounts by contrasting groups of items that the data-based model gets wrong, iterating between the two parts.
The procedure is framed as a general expectation-maximization method that jointly improves feature descriptors and the data-based fit. Readers get interpretable meaning traits rather than opaque vectors. The hybrid aims to keep data-based discipline while borrowing the descriptive power of large language models.
Catalog record
| CREATOR | Marko Jojic, Nebojsa Jojic |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 86 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-24834
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GenZ: Foundational models as latent variable generators within traditional statistical models Marko Jojic, Nebojsa Jojic Abstract We present GenZ, a hybrid model that bridges foundational models and statistical modeling through interpretable semantic features. While large language models possess broad domain knowledge, they often fail to capture dataset-specific patterns critical for prediction tasks. Our approach addresses this by discovering semantic feature descriptions through an iterative process that contrasts groups of items identified via statistical modeling errors, rather than relying solely on the foundational model’s domain understanding. We formulate this as a generalized EM algorithm that jointly optimizes semantic feature descriptors and statistical model parameters. The method prompts a frozen foundational model to classify items based on discovered features, treating these judgments as noisy observations of latent binary features that predict real-valued targets through learned statistical relationships. We demonstrate the approach on two domains: house price prediction (hedonic regression) and cold-start collaborative filtering for movie recommendations. On house prices, our model achieves 12% median relative error using discovered semantic features from multimodal listing data, substantially outperforming a GPT-5 baseline (38% error) that relies on the LLM’s general domain knowledge. For Netflix movie embeddings, our model predicts collaborative filtering representations with 0.59 cosine similarity purely from semantic descriptions—matching the performance that would require approximately 4000 user ratings through traditional collaborative filtering. The discovered features reveal dataset-specific patterns (e.g., architectural details predicting local housing markets, franchise membership predicting user preferences) that diverge from the model’s domain knowledge alone. 1 Introduction Given an item’s semantics, such as a text description, image, or audio recording, large foundational models can answer specific questions about the content, and thus convert the item’s semantic representation into a canonical discrete representation. Such a representation is useful for traditional statistical models—such as graphical generative models, belief networks, or probabilistic circuits—which can capture statistics from external, possibly proprietary, datasets related to these items. Movies are an example of such items. We can use an LLM to featurize them as binary vectors based on answers to questions as illustrated in Fig. 1 . These binary vectors can help explain the patterns found in the movie-user ratings matrix. Collaborative filtering techniques reduce the observation matrix to embeddings, which capture usage statistics absent from the foundational models’ training data. The foundational models, however, can reason about the semantic content—such as the movie synopsis, critical response, credits, and other metadata—either from their training or by retrieving information from a database as part of a RAG system. We describe how appropriate prompts can be learned from such item statistics by jointly modeling the semantic features and the statistical patterns in the data. We study a general hybrid statistical-foundational model defined as follows. Let s be the semantic representation of an item, e.g., text with a movie title or its description. Given a collection of semantic feature descriptors \theta_{f} , such as ’historical war film’, a foundational model, which we refer to as function h , makes a judgment whether the feature applies to the item (Figure 1 ). Let {\bf z} be the true binary feature vector that allows for possible errors in either the model’s judgment or the wrongly worded feature descriptors. We model uncertainty by defining a conditional distribution p({\bf z}|s,\theta_{f},\theta_{e}) , where {\bf z}=[z_{1},z_{2},...,z_{n_{f}}]^{T} is a discrete feature vector (binary, in the example in the figure and in our experiments), \theta_{f} is a collection of semantic feature descriptors, such as ’historical war film’, and \theta_{e}=[p^{e}_{1},p^{e}_{2},...,p^{e}_{n_{f}}] is a set of estimated probabilities of feature classification errors. A feature vector is then linked to observation {\bf y} through a classical statistical model p({\bf y}|{\bf z},\theta_{y}) , possibly involving latent variables or some other way of modeling structure, and which has its own parameters \theta_{y} . By jointly optimizing for the parameters of the model p({\bf y}|s)=\sum_{{\bf z}}p({\bf y}|{\bf z})p({\bf z}|s) based on a large number of pairs ({\bf y},s) we can discover the feature descriptions, or the right questions to ask , in order to explain the statistics in the dataset. More generally, hybrid models could have more than one category of items with a (learnable) statistical model of their relationships, for example geographic regions and native plants. s = { aa "0. Fast and Furious", aa "1. Miss Congeniality", aa "2. Independence Day", … } \bf\theta_{f} = { aa "0. Fantastical world", aa "1. Historical war film", aa "2. Special effects", aa "3. Coming-of-age", … } e_prompt1 = f"For item ’{ s[t] }’ first describe its characteristics. Then consider this feature: ’{ \bf\theta_{f} [i] }’ Finally, output a JSON object with an answer determining if the feature applies to the item. {{ "applies": [0 or 1] }}" e_prompt2 = f"For item list ’{ s }’ first describe items characteristics. Then consider this feature: ’{ \bf\theta_{f} [i] }’ Finally, output a JSON object where the keys are the item numbers that satisfy the feature: {{ "matched_items": [list] }}" Figure 1: Feature classification h(s,\theta_{f}) with extraction prompt strings shown in Python syntax. Items s in the list are featurized using feature descriptions in \theta_{f} . The first prompt is used within a double loop ( t,i ) over items and features. The second is looped only over i (and batches of items {s}) as each call retrieves the indices of items for which z_{i}=1 (with z_{i}=0 for all other items). The idea of using interpretable intermediate representations to bridge inputs and predictions has a long history in machine learning. Concept bottleneck models (CBM) 9 ; 8 formalized this approach by training neural networks to predict pre-defined human-interpretable concepts from inputs, then using those concepts to make final predictions. Recent work has leveraged large language models (LLMs) to overcome the need for manually labeled concepts, e.g., 2 ; 5 ; 4 ; 12 ; 10 . These methods may or may not use the CBM formulation but share with it the goal of feature interpretability. In addition, they have a goal of feature discovery , which is also one of the goals here. They address the difficulty of modeling and jointly adjusting the semantic and statistical parts of the model—in our notation p({\bf z}|s) and p({\bf y}|{\bf z}) —in different ways. In most cases, the target {\bf y} is a discrete variable (the goal is a classification through intermediate concepts), and thus the classification model p({\bf y}|{\bf z}) is a simple logistic regressor or an exhaustive conditional table 4 ; 5 ; 12 . The optimization of the featurizer p({\bf z}|s) primarily relies on the LLM’s domain understanding (such as a task description), but some methods also provide examples of misclassified items, e.g., 12 ; 10 . When real-valued multi-dimensional targets {\bf y} are modeled, showing items with large prediction errors is much less informative because each item may have a different error vector, and thus item-driven discovery is usually abandoned. For example, in 2 multidimensional targets (fMRI recordings) are modeled, and the method relies on direct LLM prompting about potentially predictive features based on domain knowledge alone, followed by feature selection, rather than showing specific items to contrast. A fundamental limitation shared by these approaches is their reliance on the LLM’s domain understanding to propose discriminative features. This works well when the LLM has been trained on