User choice, hidden gems, and the quadratic horizon Authors: Flavio Chierichetti, Mirko Giacchini, Ravi Kumar, Alessandro Panconesi, Andrew Tomkins Published: 2026 ABSTRACT AbstractContent platforms typically engage with their users through small recommendation sets of items drawn from an extensive catalog. These sets are curated using machine-learned models optimized to present choices most likely to align with user preferences. We present surprising findings about such platforms. Even with complete information on user preferences within sets of up to k items, these models can only predict preferences within a “quadratic horizon” of k2 items and might fail to identify the best items in larger sets. To illustrate, we present striking examples where a platform interacting with users through small item sets, despite knowing that one item is favored by millions of users, cannot identify this item with better than random chance. Through both theoretical analysis and studies across various datasets, we demonstrate that “hidden gems,” items preferred by many users but invisible to platforms, exist in real-world datasets of moderate size, highlighting a significant gap in current recommendation platforms. Content recommendation systems typically display only a small set of items from huge catalogs. Our work demonstrates that even when a system perfectly knows users’ preferences within these limited sets, it may still fail to identify the very best items across the entire catalog. As a result, these systems miss “gems” that could be favored by millions. Our empirical and theoretical findings uncover a fundamental challenge in recommendation models, paving the way for new approaches that help platforms better highlight the highest-quality content. Introduction It is of paramount interest among platform providers to identify the best items from a catalog to recommend to users: what is the best song, the best currently-available movie, the best dystopian novel? What is the best tourist destination in the world? The great successes of content recommendation platforms are based largely on a simple recipe: build a machine-learned model to output small recommendation sets from a large catalog, optimized to contain the choices a user is most likely to prefer, and then refine the predictions of the model based on user engagement with these recommendations. Think here of the ten blue links of Web search, or a shelf of movies in Netflix, or of books in the Amazon Kindle Store. We present three surprising findings about such platforms. First, we show mathematically that “hidden gems” may exist that are preferred by many users, but are invisible to platforms like the ones above whose feedback comes from users’ choices within many small sets of recommendations. Second, the ability to detect such gems is governed by a law stating that, with complete information of user preferences within every possible set of k or fewer items (10 links, 15 movies in a carousel, etc), the best model training procedure may learn to predict preferences within sets of up to k2 items, but cannot reliably predict the best items within larger sets. Thus, the best items within large groupings beyond this “quadratic horizon,” such as the entire catalog, may remain provably invisible to any algorithm. Finally, these theoretical concerns about hidden gems do, in fact, occur in practice: across a wide range of moderate-size real-world datasets, the best predictions possible about quality of items are routinely off by large factors, and highly-preferred items appear less popular than other items. In this sense, the fact that users cannot realistically offer opinions about an entire catalog of possibly millions of items represents an invisible but information-theoretically challenging barrier to the ability of platforms to identify the most preferred items. These findings have implications for the quality of content recommendations. The model we study is the following. A content platform repeatedly offers a user a small slate of possible choices, and sees which option (if any) the user selects. To formalize our study, we assume the user makes choices according to a RUM model (1), the most widely-studied model of user behavior in these settings. In a RUM model users arrive in the system with a value for each item of the catalog, drawn from an arbitrary joint distribution, and select the highest-value item available to them within the slate.1 The selected item is also called the winner of the slate. Our focus on slates that are small relative to the size of the catalog reflects reality. Content platforms would love to have users with the patience to study each item in the catalog in order to select the best one, but catalogs are enormous, and user attention is limited. Instead, modern user interfaces are carefully designed to show a limited number of choices. If the user does not engage, the interface will typically offer another small set representing a different slice of the catalog: recently-watched movies, followed by most-popular, followed by “because you liked Ponyo,” and so forth. We will call an interface limited(k) if it shows no more than k items to the user at a time, with realistic values of k ranging from 3 to perhaps 100 depending on the platform. We will use n to denote the size of the overall catalog, with the understanding that n≫k. We obtain new theoretical results showing that one can construct RUMs with the following properties. Firstly, any algorithm given infinitely many observations of a limited(k) interface will see all catalog items as identical in quality. Nonetheless, there is a special item that is the overall catalog-wide favorite of an approximately 4/k2 fraction of the users. We call this type of item a hidden gem: that is, an item which is favored by a significant fraction of the whole population, and yet it is invisible to limited(k) interfaces. For example, with a catalog of 105 movies, and a limited(5) interface, all movies might be identically likely to be selected in the interface, but some movie might actually be the top choice of 104 times more users than any other movie (see Figure 1). Example of our construction. a) Consider a streaming platform with a catalog of 105 movies. b) The platform shows, to its 107 users, slates of at most 5 movies. The platform observes that, in all of these slates, each movie is preferred by the same fraction of the population. c) Given these observations, the platform concludes that, even in the complete catalog, each movie must be preferred by the same fraction of users. d) However, there is a hidden gem in the catalog, preferred by ≈1.1 million more users than all other movies. The platform has no way of identifying this gem by looking at its interactions with the users. e) This plot shows the CDF of the utilities of our RUM containing a hidden gem. The utility of the non-gem movies is chosen as a uniform-at-random (continuous) rating in [1,5]. The utility of the gem, parametrized by k, is chosen to make the slates of size at most k appear uniformly at random, while maintaining a significantly high probability of having rating exactly 5 (see the the Supplementary Information for a proof). The plot shows the gem utility for k∈{2,3,5,30}. The streaming platform scenario of a)–d) uses k=5. Our theoretical result holds very broadly in the context of limited(k) interfaces. When faced with the problem of finding items that are the favorite of many users, it is natural to consider approaches to identify such items by iteratively offering the most popular items, and continuing to refine this set until eventually converging to those that most users prefer above all. Our results show that, surprisingly, any such approach is doomed to failure: no matter how one tries to restrict to the most popular candidate items, there is simply not enough information in a limited(k) interface to identify the most preferred ones. This raises an immediate pragmatic question: do such instances of hidden gems arise in practice, or are practical settings fundamentally “easier” than the theoretical constructions described above? Given the theoretical results just described, it is a non-trivial task to state confidently that a hidden gem exists, since doing so requires information not available in the limited(k) interface. Thus, in order to study this question, one needs access to data representing the preferences of the users among smaller slates, and the preferences of the users in the entire catalog. We perform two types of studies. First, we study existing choice datasets of small catalogs, for which each user’s complete preference information is available. We identify seven datasets of this form, and show that hidden gems do, in fact, exist in all of them. It is typical in these datasets that the hidden gem is roughly two times more popular than would be estimated from a usable interface. Errors of this magnitude in estimating the popularity might have significant consequences in identifying the most compelling content on a platform. Additionally, the most popular item we identify from the evidence of limited(k) interfaces is often not the most favored overall: not only do we have factor-two errors in favorability, but we also see inversions in our ordering of the most-preferred items. Second, we perform a crowd-sourced study of this question on the much larger catalog of movies. In this enormous catalog, collecting complete user preferences is impossible. Therefore, we collect the preferences in two non-overlapping phases. In the first phase, users are asked to type in their overall favorite movie. In the second phase, users are presented with a random slate of three movies and they go through two or three screens, where they are asked to: (i) select their preferred movie from the slate, then (ii) select the slate’s movies they have watched, and (iii) pick their preferred movie among the slate’s movies that they watched. We observe that hidden gems also exist in this dataset. We highlight that, since we are not collecting complete user rankings, this dataset might not be compatible with a RUM (instead, the other seven datasets are indeed RUMs). To address this point, we repeated this last experiment by asking an LLM to generate the rankings, rather than human users—we observed that the results are qualitatively similar in the two settings (the results for the LLM-generated dataset can be found in the Supplementary Information). Theoretical results We use the standard notations [n]={1,…,n} and (Sk)={T⊆S∣|T|=k}. A RUM R over the catalog of items [n] is a probability distribution over Rn such that ties happen with probability 0. A slate is non-empty subset of the catalog and a k-slate has k items. Slate [n] is also called the full slate. For a slate ∅≠S⊆[n], the winning distribution on S is given by Pr[swins inS]=Pru∼R[us=maxs′∈S{us′}], where s∈S and u∈Rn is the utility vector drawn from R. A RUM is called independent when the utilities of the items are mutually independent. We formally define a hidden gem as follows: A RUM R over [n] contains a hidden gem G∈[n] for limited(k) interfaces if (i) for each slate S of size i≤k, G wins in S with probability ≤1i, and (ii) G wins in the full slate with probability >cn, for a constant c>1. Of course, the larger the winning probability in the full slate, the larger the value of the gem. As discussed above, our main theoretical result is as follows, which implies the existence of hidden gems having winning probability Ω(1) in the full slate when k is constant: For each k≥1, and for each n>k, there exists an independent RUM on [n] such that (i) the winner of each slate S of size at most k is chosen uniformly at random, and (ii) this RUM contains a special item i* such that if a slate S (of arbitrary size) contains i*, then the probability that i* wins in S is at least 1⌈k+12⌉⋅⌊k+12⌋. This result entails that, when only considering slates of size at most k, it is impossible to guess whether (i) the RUM assigns probability 1n to each item in the full slate (and, more generally, probability 1|S| to each item in the generic slate S), or whether (ii) it assigns probability at least ≈4k2 to some unknown special item. Consequently, we cannot tell whether any gems exist in slates containing more than k2 items. In contrast, in earlier work (2) we showed that observing slates of size at most O(k⋅log(1/ϵ)) suffices to reconstruct, up to an ϵ error, the winning distributions of slates of size k2 in a generic RUM. In other words, algorithms can discover gems within this quadratic horizon, but not beyond it. We mention that in (2) we also showed a construction of two RUMs inducing the same distributions on small slates—however, that construction does not contain hidden gems.2 We also show the near-optimality of Theorem 1: For each fixed k≥1, for each RUM on [n] that chooses the winner of each slate of size at most k uniformly at random, it holds that if 1≤s≤n, then ∀S∈([n]s) there exists no item of S that wins in S with probability larger than 1⌈k+12⌉⋅⌊k+12⌋+os(1).3 It is thus impossible to obtain a stronger construction: a RUM that is uniform on all slates of size at most k, cannot give to any one item a probability of winning in S asymptotically larger than that of Theorem 1. We recall that the literature often assumes that the choice model is an MNL (1, 3–6). An MNL is a type of RUM where each item is assigned a weight, and the probability of choosing an item is proportional to its weight. A uniform MNL has equal weights for all items. MNLs are efficiently learnable from data, readily leverage specialized hardware such as GPUs, and demonstrate effective empirical performances (7). These properties have made them ubiquitous in practice, making the study of the hidden gem phenomenon when learning MNLs particularly relevant. Our Theorem 1 implies that even assuming that the observations on slates of size at most k are compatible with a uniform MNL, it is impossible to distinguish between the following two alternatives: the RUM behaves like a uniform MNL everywhere (i.e. it chooses an item uniformly at random from each slate), or it gives an overwhelming probability to a gem in the full slate. In fact, Theorem 1 can be generalized to yield the following: Let w1,…,wn, and k, be positive integers. Moreover, let i⋆∈[n]. Then, there exists an independent RUM on [n] such that: ∀S⊆[n] such that ∑j∈Swj≤k, it holds Pr[iwins inS]=wi∑j∈Swj for all i∈S;∀S⊆[n] such that i⋆∈S, it holds Pr[i⋆wins inS]≥1⌈k+12⌉⋅⌊k+12⌋. ∀S⊆[n] such that ∑j∈Swj≤k, it holds Pr[iwins inS]=wi∑j∈Swj for all i∈S; ∀S⊆[n] such that ∑j∈Swj≤k, it holds Pr[iwins inS]=wi∑j∈Swj for all i∈S; ∀S⊆[n] such that i⋆∈S, it holds Pr[i⋆wins inS]≥1⌈k+12⌉⋅⌊k+12⌋. ∀S⊆[n] such that i⋆∈S, it holds Pr[i⋆wins inS]≥1⌈k+12⌉⋅⌊k+12⌋. This result shows that, even when the data collected from small slates is perfectly consistent with an MNL model with integer weights that does not significantly favor any item, one still cannot rule out the existence of a gem in the full slate, with winning probability as large as 1⌈k+12⌉⋅⌊k+12⌋. The proof of our main result uses Legendre polynomials and the Gauss–Radau and Gauss–Lobatto quadratures, for obtaining weight assignments that make it possible to construct RUMs with the desired properties. All proofs are in the Supplementary Information. Experimental results We now study whether hidden gems exist in real world data. We define the gem of a dataset as the item having the largest winning probability in the full slate. In most of our datasets, each such item has significantly larger winning probability than all the others, supporting our definition. We are interested in whether the algorithms make one or both of these errors: (i) inversion error, i.e. the estimated winning probability of the gem is smaller than that of other items, and (ii) estimation error, i.e. the winning probability of the gem is greatly misestimated. Algorithms As for the theoretical results, we first take an information-theoretic approach: by using a linear program, we find the maximum and minimum probability of the gem in a RUM that fits exactly the training data of the limited(k) interface. When the gap between these two probability values is large, no algorithm can guarantee a good estimate of the gem. We also take a computational approach by explicitly learning a RUM that minimizes the average ℓ1-error, employing the faster RumRunner algorithm of (8, 9). We also consider three standard algorithms to learn an MNL model: (i) Stochastic Gradient Descent (SGD) minimizing the KL-divergence, (ii) SGD minimizing the MSE error, and (iii) Iterative Luce Spectral Ranking (ILSR) to approximate the maximum likelihood estimate (3). We found these three algorithms to behave similarly in all tests. All algorithms can be used with either Max-Dist or Max-Sample oracle access to the dataset.4 For a given slate S, the Max-Dist oracle returns the exact winning distribution on S, while the Max-Sample oracle returns only a winner drawn from the winning distribution on S (when using Max-Sample, the same slate can appear multiple times in the dataset). The Mallows model is another popular special case of RUMs (10), however, MNLs are typically preferred in discrete choice settings and therefore we focus on the latter (1). Datasets The seven choice datasets that we considered are: course1 and course2 (11), sushi (12), breakfast (13), t-shirt (14, 15), students (15, 16), and jester (17). Each of these datasets consists of complete rankings of the catalog items (from 6 to 100) from several respondents. We test these datasets in four ways: for a slate size k, we either provide the algorithms with slates of size exactly k or with slates of size 2,…,k; moreover, we provide the algorithms with either the exact winning distributions of the slates (Max-Dist oracle), or with samples from it (Max-Sample oracle). Our findings are consistent across these four scenarios. Regarding the movies dataset, we use as catalog the 34k US movies listed on Wikipedia.5 Data collection proceeds in two phases: first, users report their overall favorite movie (to estimate the full slate distribution); second, they select their favorite among three options, and their favorite among those options restricted to the movies they have watched (to learn preferences on small slates). We collect 5k and 8k responses in the two phases, respectively. We conducted the study on Amazon Mechanical Turk and each respondent was paid 0.05 USD, for a few seconds work. In both phases, we did our best to filter duplicate accounts and correct user mistakes. Given the large catalog of movies, it is inevitable that some movies get tested only once or never in the second phase. We considered three increasingly restricting approaches to sparsity reduction, each one either filtered to movies we know the user has seen, or not filtered. All six showed similar patterns. We will focus on the results for the most conservative one, while we report the results for all other versions in the Supplementary Information. All details on data collection and data cleaning are provided in the Supplementary Information. Real and predicted value of the gem We first observed that, in datasets sushi, breakfast, jester, and movies, the gem has a winning probability at least 70% higher than the second largest item; while for the other four datasets, its winning probability is 70% higher than the third item. Therefore, gems exist in real-world datasets. However, we found that in all datasets,6 slates of size two are compatible with a RUM that assigns probability 0 to the gem. Therefore, in principle, any RUM-learning algorithm might incur both inversion and estimation errors. In fact, for k=2, we observe inversions in students, course2, t-shirt, jester, and movies when learning an MNL model with SGD, while when learning a RUM with RumRunner, we observe inversions in students, t-shirt, and jester. Moreover, for k=2, the gem is roughly 100% more popular than what predicted by an MNL in students, course2, breakfast, jester, and movies, and at least 30% more popular in course1, t-shirt and sushi. From this, we conclude that real-world datasets can contain hidden gems. We report in Figure 2 the predictions for movies; the plots of all other datasets can be found in the Supplementary Information. Predicted winning distributions for movies dataset. We show the first 100 items sorted by real winning probability. Among the six versions of movies, this is the one where MNL works best (see Supplementary Information for details). Note that the gem is 66% more popular than the second item and 500% more popular than predicted by the MNL. Moreover, the MNL incurs several inversion errors. In this plot, we have k=2. Quadratic horizon As the slate size increases, because of the quadratic horizon, the algorithms provide better estimates of the gem’s popularity (2). In particular, the gap between the maximum and minimum probability of the gem in a RUM compatible with the training data goes close to zero at roughly k≈n for breakfast, course1, course2, students, and t-shirt in accordance with the quadratic horizon, even for these small datasets. Moreover, for k≥3, RumRunner provides good estimates for all datasets7 and it consistently converges quicker to the real gem value than an MNL. We also observe that MNLs often do not converge to the gem value even with slates of size ≈n. This suggests that, as the slate size increases, a general RUM might be a better choice than an MNL when it comes to uncovering hidden gems. However, when the slate size is very small, general RUMs will also incur severe errors, as we showed in the previous section where k=2. We report in Figure 3 the predictions across increasing slate sizes for t-shirt; the plots of all other datasets can be found in the Supplementary Information. Quadratic horizon in t-shirt dataset. We show the predicted winning probability of the gem as k increases. The algorithms are given in input the winning distributions for slates of size 2,…,k. Observe that there exists RUMs that exactly fit the interactions at k=2 but assign probability 0 to the gem. However, as k increases, the gap diminishes according to the quadratic horizon. Therefore, RumRunner quickly converges to the correct probability. Note that the MNL does not necessarily lie between the maximum and minimum probability of a compatible RUM, because it might not be expressive enough to exactly fit the training data. Conclusion Our findings reveal a fundamental limitation of content recommendation platforms that are trained on user choices on small slates of items. Our theoretical analysis demonstrates that hidden gems—items highly favored by many users—cannot be identified using existing interfaces, in general; our empirical data confirm the existence of these gems across multiple real-world moderate-size datasets. These findings call into question the effectiveness of existing recommender systems in carrying out their tasks successfully and emphasize the need for alternative approaches. Future work could tackle this challenge by eliciting information beyond choices within small slates or carousels and this could significantly enhance recommender systems and improve user satisfaction across digital platforms. It would also be interesting to extend the analysis of hidden gems to more general models beyond RUMs. In particular, while Theorem 1 remains valid, the gem value of Theorem 2 may no longer be tight. Additional research directions include modeling other practical aspects of content recommendation; for example, ensuring that users receive multiple suggestions and that platforms can provide a diverse set of recommendations tailored to different user needs. Exploring other notions of “gem” is also a valuable direction for future work. In particular, our definition focuses on population-wide preference and does not capture items that are extremely favored only by a specific sub-population. Likewise, we consider a closed catalog of items without features; similar questions could also be considered in the presence of features, which can be important for cold-start settings. Supplementary Material