Interestingness First Classifiers

Ryoma Sato [email protected] · 2025

Read Interestingness First Classifiers by Ryoma Sato [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-19780.txt.

Introduction

Machine learning usually chases accuracy, but sometimes the more interesting model is the one that surprises. Ryoma Sato argued in 2025 for classifiers built to be interesting first, choosing features for novelty rather than raw predictive power.

His framework, EUREKA, asks a language model to rank features by how interesting they seem, then builds transparent classifiers from those picks alone. Predicting room occupancy from humidity rather than carbon dioxide loses some accuracy but tells a richer story. In a set of academic papers, the method found that titles containing a colon predict future citations. Sato presents such models as tools for discovery and communication.

Catalog record

CREATORRyoma Sato [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-19780

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Interestingness First Classifiers Ryoma Sato [email protected] Abstract Most machine learning models are designed to maximize predictive accuracy. In this work, we explore a different goal: building classifiers that are interesting. An “interesting classifier” is one that uses unusual or unexpected features, even if its accuracy is lower than the best possible model. For example, predicting room congestion from CO2 levels achieves near-perfect accuracy but is unsurprising. In contrast, predicting room congestion from humidity is less accurate yet more nuanced and intriguing. We introduce EUREKA, a simple framework that selects features according to their perceived interestingness. Our method leverages large language models to rank features by their interestingness and then builds interpretable classifiers using only the selected interesting features. Across several benchmark datasets, EUREKA consistently identifies features that are non-obvious yet still predictive. For example, in the Occupancy Detection dataset, our method favors humidity over CO2 levels and light intensity, producing classifiers that achieve meaningful accuracy while offering insights. In the Twin Papers dataset, our method discovers the rule that papers with a colon in the title are more likely to be cited in the future. We argue that such models can support new ways of knowledge discovery and communication, especially in settings where moderate accuracy is sufficient but novelty and interpretability are valued. 1 Introduction Machine learning research usually focuses on building models with the highest possible accuracy. This is natural, since accurate predictions are useful for applications such as medical diagnosis [ 13 , 10 , 14 ] , recommender systems [ 25 , 16 , 24 ] , or speech recognition [ 42 , 19 , 43 ] . However, accuracy is not the only possible goal. Since data involve aleatoric uncertainty [ 20 , 21 ] , perfect accuracy may be out of reach. There may not exist any “true” laws waiting to be discovered. This leads to our motivation: faithfulness should not be regarded as an absolute objective, and even rules that are not entirely faithful may still be worth pursuing if they offer interesting insights. For illustration, consider the problem of classifying whether a user is an adult based on their profile. If the profile contains an age feature, then the simple classification rule age \geq 18 achieves 100% accuracy. On social networking services, users may occasionally misreport their age, but in most cases they provide approximately correct information, so an accuracy of around 99% is still realistic. This rule is extremely accurate, but it is not interesting. Let us consider a more unusual rule. Suppose we define the classifier: “if the registered email address ends with @icloud.com then the user is a minor, whereas the user is an adult if it ends with @aol.com.” Such a rule might achieve some level of accuracy. Although its accuracy would not be particularly high, it is engaging precisely because it is unexpected. This is the kind of classifier we would like to obtain. A somewhat more academic example is the detection of depressive symptoms. Numerous tests have been proposed for this task. A well-known one is the Beck Depression Inventory, which asks participants to answer 21 questions on a four-point scale—such as whether they feel depressed, whether they are pessimistic about the future, or whether they feel guilty—and judges depression based on the total score. This test is widely used for diagnosing depression, but whether the decision rule itself is “interesting” is questionable. By contrast, a more intriguing rule is known [ 46 ] : “if a person uses first-person pronouns (I, my, me, myself) frequently in an essay, then they are likely to be depressed.” Those with depressive tendencies are believed to pay greater attention to information about themselves than to information about the outside world or people around them. Of course, this rule is far from perfect: it is heavily influenced by an individual’s writing style, and even cheerful people may frequently use first-person pronouns. In fact, while the rule is statistically significant, its predictive power is quite low (around r = 0.13) [ 9 ] . Nevertheless, it is considered interesting, and despite its limited accuracy, it has gained recognition in psychology, inspiring a range of related and follow-up studies [ 9 ] . As a more practical example used in this paper, consider the task of detecting whether an office is occupied (the Occupancy Detection dataset). In this dataset, a simple rule based on whether the lights are on achieves around 99% accuracy. Offices are typically lit during working hours, and in most companies people do not remain in an office with the lights off, so the relation “lights on = people present” holds almost universally. However, if the problem can be solved in this trivial way, it is not very interesting. As we will show later, the same problem can also be addressed with the rule “if humidity is high, then people are present,” which achieves around 85% accuracy. Of course, high humidity alone cannot distinguish between rainy weather and actual human presence, and even when people are present the humidity does not always rise significantly, so the accuracy cannot be perfect. Nevertheless, the idea that occupancy can be inferred from a humidity sensor sits at an intriguing middle ground—it feels like it could work, but also like it might not, making it nuanced and interesting. Moreover, beyond mere curiosity, the fact that non-obvious sensors can partially solve the task suggests practical potential: applications that do not require extremely high accuracy might be realized at lower cost than conventional approaches. Related topics include feature selection, interpretability, and explainability. However, these are generally aimed at maximizing accuracy, which distinguishes them from our focus on interestingness. For example, in the adulthood classification problem, feature selection would naturally identify age as the most important feature, leading to the interpretable rule age \geq 18. This rule is interpretable and highly accurate, but not interesting. In the office occupancy detection problem, the feature light would be selected, producing the rule light \geq 1. Again, this is interpretable and accurate, but not interesting. In this paper, we introduce EUREKA (Exploring Unexpected Rules for Expanding Knowledge boundAries), a simple framework for building interesting classifiers. The method uses large language models (LLMs) to compare pairs of features and judge which one would produce a more interesting rule. From these pairwise comparisons, we derive a global ranking of features and train interpretable classifiers using only the top-ranked ones. Our experiments across multiple benchmark datasets show that EUREKA consistently highlights features that are not the most predictive but still achieve non-trivial accuracy. For instance, instead of selecting light intensity to predict office occupancy, which is highly predictive but obvious, EUREKA automatically prefers humidity, which provides a weaker but more surprising signal. The main contributions of this work are: • We define the concept of interesting classifiers, which prioritize unexpectedness over accuracy. • We propose EUREKA, a simple framework that leverages LLM-based pairwise comparison to rank features by interestingness. • We demonstrate, through experiments on several datasets, that classifiers built from top-ranked interesting features can still reach meaningful accuracy while providing surprising and interpretable insights. 2 Proposed Method 2.1 Problem Setting We consider a standard classification problem on tabular data. The input is

Read the complete text