The FIL Hypothesis: Inductive Biases Help with Kernel Engineering
Nikolai Rozanov, Subhabrata Dutta, Preslav Nakov, Iryna Gurevych · 2026
Read The FIL Hypothesis: Inductive Biases Help with Kernel Engineering by Nikolai Rozanov, Subhabrata Dutta, Preslav Nakov, Iryna Gurevych (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30442.txt.
Introduction
Nikolai Rozanov, Subhabrata Dutta, Preslav Nakov, and Iryna Gurevych argue that AI's famous Bitter Lesson misses a hidden cost: slow feedback. The lesson says general methods scaling with data beat built-in knowledge. Historic AI wins enjoyed near-instant verification of predictions. Games and classification give feedback in seconds or less. Science and physical-world tasks may take hours to weeks. That delay is the Feedback Information Loop duration.
With long loops, data-driven methods cannot gather enough verification steps. The authors propose leaning on expert knowledge instead. Inductive biases constrain the solution space in their method. The approach is orthogonal to data-driven scaling. GPU kernel engineering supplies the first validation test. That domain has a nontrivial feedback loop. Adding inductive biases beat data-driven approaches there. The hypothesis reframes when hand-built knowledge should win. Long feedback loops, not task type, are the signal.
Catalog record
| CREATOR | Nikolai Rozanov, Subhabrata Dutta, Preslav Nakov, Iryna Gurevych |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74 KB |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30442
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The FIL Hypothesis: Inductive Biases Help with Kernel Engineering Nikolai Rozanov, Subhabrata Dutta, Preslav Nakov, Iryna Gurevych Abstract The Bitter Lesson , which posits that general-purpose methods that scale with computation and data ultimately outperform those with built-in human knowledge, has become a dominant paradigm in the era of Large Language Models. We revisit this principle by observing a new and critical scaling dimension: the duration of the Feedback Information Loop (FIL), the time required for a system to receive a verification signal after generating a prediction. Most historic successes in Artificial Intelligence (AI) have benefited from near instantaneous feedback (e.g., games or classification tasks), but we argue that future AI applications in science and the physical world will inherently involve FILs ranging from hours to weeks. This trend poses a fundamental scaling limit, as obtaining enough verification steps required by purely data-driven methods becomes practically impossible. Additionally, we propose a method that is orthogonal to purely data-driven approaches, based on human-inspired expert knowledge. The method relies on inductive biases and constraining the solution space. We provide an initial validation of the hypothesis and the method, by studying the real-world GPU programming task, a domain with non-trivial FIL, and demonstrate that incorporating inductive biases yields superior performance over data-driven approaches. The code is released under: https://github.com/ai-nikolai/robust_kernelbench . 1 NLP Department, MBZUAI, Abu Dhabi, UAE 2 Ubiquitous Knowledge Processing, TU Darmstadt, Darmstadt, Germany 3 Department of Computing, Imperial College London, London, UK 1 Introduction The Bitter Lesson , published nearly seven years ago, states that Artificial Intelligence (AI) systems that benefit from scaling with computation and data are those that outperform other approaches “by a big margin”, notably those that rely on human-inspired inductive biases [ 6 ] . Since the publication, this lesson has seen even more empirical evidence with Large Language Models (LLMs) not only when solving various classical Natural Language Processing (NLP) tasks [ 46 ; 45 ] that seemed extremely hard only a decade ago, but also when solving tasks such as problems from the Math Olympiad [ 25 ] , the International Collegiate Programming Competition (ICPC) [ 53 ] , as well as Agentic AI tasks such as Web Navigation, Robotics and many others [ 37 ; 50 ] . A key method behind these successes was the reinforcement learning-based scaling [ 19 ] of LLMs. In line with the Bitter Lesson these approaches benefit from scaling with large amounts of data and computational effort. As can be seen in Table 1 , previous state-of-the-art approaches scaled and exceeded human performance with data scales ranging from around O(10^{6}) simulations or O(10^{12}) tokens of training data. Contrary to existing views, we propose a new dimension to think about such systems: the Feedback Information Loop (FIL). We define the Feedback Information Loop as the duration for an AI system to receive ‘feedback’ after it has made a prediction. This can be the reward (+1, 0 or -1) when playing chess or Go, the accuracy in classification tasks, a numerical reward on LLM generated responses and so on. The crucial observation is that most, if not all, existing problems enjoyed a very short Feedback Information Loop; given an AI response and a desired gold answer it is very quick to compute whether the AI system is right. For example, in chess it is very quick to compute if the position is a draw, a win, or a loss; likewise, in natural language processing, one can quickly tell whether the answer is correct, etc. Looking into state-of-the-art and future AI application areas, it is evident that the Feedback Information Loop is only bound to increase. This can be easily seen in applications such as the AI Scientist [ 31 ] (e.g., designing automatic GPU code, or machine learning experiments) or AI for the physical world (e.g., designing a chemical synthesis, or a new aerodynamic design), which will take several hours to several weeks to verify, as shown in Table 3 . The scaling to the required 1 1 1 The number of required data points for these methods to scale is consistently very large. 10^{6} is taken as a lower estimate given the existing problems. O(10^{6}) data points in such cases would yield practically impossible timelines for purely data driven approaches. This can be clearly seen in Table 2 , e.g., if a single verification takes an hour to verify, then 10^{6} batches would take around 114 years of verification, if it takes a day that would be nearly 3,000 years of verification, while if the verification takes a week it would take almost 20,000 years to verify. Hence purely data-driven approaches are becoming prohibitively expensive in the extended FIL setting. Previous approaches did not openly identify this issue; however, they can be seen as implicit attempts to solve the long FIL problem. Specifically, some existing work has attempted to overcome the orthogonal but related problem of sparse feedback signal [ 15 ; 8 ] ; however, the underlying problems have short FIL and do not fundamentally overcome the problem. The most direct comparison is existing work that aims to simulate the underlying process and thereby to implicitly approximate the Feedback Information Loop, such as World Models, model-based Reinforcement Learning, reward modeling, and other forms of modeling the environment or the feedback [ 20 ; 23 ] . Such approaches are limited, however, as shown in the latest comprehensive work [ 16 ] , specifically due to (1) the need for training data, and (2) the so-called “simulation to real” gap, which represents the aleatoric and epistemic uncertainty of the models. When looking at long FIL, we see that these two issues with environment models would only amplify. Hence, data-driven approximation techniques also do not represent a fundamental solution. Our main hypothesis , the FIL hypothesis, is that “as the Feedback Information Loop increases in duration, it will pose a fundamental limit to purely data-driven methods”, thereby extending the Bitter Lesson . Inspired by this observation and hypothesis, we propose a method based on inductive biases. We define inductive biases as constraints on the solution search space. These are often inspired by a human expert , which is orthogonal to purely data-driven approaches. In order to show an initial validation of the FIL-Hypothesis and our proposal to use human expert-based inductive biases, we chose a real-world problem that exhibits the slow Feedback Information Loop trend and designed an algorithm with human-inspired solution constraints. Specifically, we chose a challenging GPU programming task [ 36 ] in line with the AI Scientist paradigm, where AI can help the human solve real and challenging scientific tasks. Our results demonstrate that even use cases with only several minutes in their Feedback Information Loop duration already show signs of the improvement via human-inspired algorithm design; we outperform the state-of-the-art baseline methods by introducing our inductive bias across different model sizes, settings and evaluation measures. Table 1: Training data scale of the selected machine learning systems. System Primary Domain Type of Training Data Quantity Feedback Information Loop (FIL) AlphaGo Zero [ 41 ] Specialised RL Algorithm GO Games \sim 4.9M O(10^{6}) <1 second DeepSeek-R1 [ 19 ] Large Language Reasoning Model Verifiable Problems \sim 5M O(10^{6})