Mockingbird: How does LLM perform in general machine learning tasks?

Haoyu Jia, Yoshiki Obinata, Kento Kawaharazuka, Kei Okada · 2025

Read Mockingbird: How does LLM perform in general machine learning tasks? by Haoyu Jia, Yoshiki Obinata, Kento Kawaharazuka, Kei Okada (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04279.txt.

Introduction

Haoyu Jia and co-authors ask in a 2025 study whether a chat model can do ordinary machine learning work on its own. Their framework, Mockingbird, tells the model to role-play the needed functions, run a task, then reflect on its mistakes and try again. The loop mimics how a junior analyst might iterate.

On several standard tasks the model reached acceptable scores without task-specific code. Yet self-reflection alone could not beat the gains from expert-written documents or human feedback. The study maps both the promise and the ceiling of handing routine data work to a talking model.

Catalog record

CREATORHaoyu Jia, Yoshiki Obinata, Kento Kawaharazuka, Kei Okada
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-04279

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mockingbird: How does LLM perform in general machine learning tasks? Haoyu Jia, Yoshiki Obinata, Kento Kawaharazuka, Kei Okada Abstract Large language models (LLMs) are now being used with increasing frequency as chat bots, tasked with the summarizing information or generating text and code in accordance with user instructions. The rapid increase in reasoning capabilities and inference speed of LLMs has revealed their remarkable potential for applications extending beyond the domain of chat bots to general machine learning tasks. This work is conducted out of the curiosity about such potential. In this work, we propose a framework Mockingbird to adapt LLMs to general machine learning tasks and evaluate its performance and scalability on several general machine learning tasks. The core concept of this framework is instructing LLMs to role-play functions and reflect on its mistakes to improve itself. Our evaluation and analysis result shows that LLM-driven machine learning methods, such as Mockingbird , can achieve acceptable results on common machine learning tasks; however, solely reflecting on its own currently cannot outperform the effect of domain-specific documents and feedback from human experts. se counter=codecounter]tcbcodebox[3][] title=Code \thetcbcounter : #2, halign title=flush center, colback=blue!5!white, colframe=blue!40!black, enhanced, listing only, listing options=style=csharp, #3, #1 I Introduction Currently, large language models (LLM) are widely used in intelligent systems as a chat bot to assist users in summarizing, processing, and generating text. However, this only exploits the natural language processing capabilities of LLMs; there are many other capabilities are not fully utilized, such as the significantly improved reasoning capabilities of recent LLMs [ 1 ] . A natural idea is to extend LLMs to non-linguistic tasks. Currently, a popular solution is to use LLMs as code generators to generate code for machine learning pipelines ( [ 2 ] , [ 3 ] ); this type of method can significantly enhance automated machine learning workflows, but the static nature of pipeline code limits the use of the dynamic capabilities of LLMs. In 2020, [ 4 ] has found that language models are able to learn from their context, proving the basic feasibility of this idea; since then, many researchers have studied the properties of this ability in specific domains: [ 5 ] ; [ 6 ] ; [ 7 ] ; [ 8 ] [ 9 ] ; these work have well explained the nature of in-context learning ability, but provide little overall insight into the integration of LLMs as run-time components into automatic intelligent systems with a wider range of tasks. In order to fill this paucity and to encourage more domains to benefit from the progress of LLM, we propose Mockingbird , a versatile and controllable framework for adapting LLMs to general machine tasks, and a corresponding open-source platform implementing this framework. Inspired by the finding that the intelligence of LLMs is merely role-playing [ 10 ] , we assume that LLMs can also acquire good performance in role-playing functions, and design our framework upon this assumption. The fundamental component of Mockingbird is mock function , a specific type of functions that do not have method bodies, but only have function declaration including documentation and method signatures (contracts on the function parameters and return values). In this framework, users are able to use mock functions as if they were ordinary functions with function bodies. In contrast to other LLM-drive code generators, Mockingbird does not implement these mock functions with code generated by LLMs at compile time; instead, it instructs LLMs to ’role-play’ them at runtime. To elaborate further, the platform furnishes LLMs with program metadata including the method signature and accompanying documentation, retrieved from the program; then, function calls to mock functions are redirected to LLMs; parameters are packed into user messages, and return values are unpacked from assistant messages. In this manner, LLMs are employed as general-purpose functions. By leveraging the in-context learning capability of LLMs, users can influence the behavior of these mock functions with minimal effort by modifying the input-output message pairs presented within the chat histories of LLMs. Furthermore, we present several optional techniques that enhance practicality of this framework. For instance, we introduce the substitution script acceleration, which replaces LLM-driven executors with substitution-scripts generated by LLMs, thereby reducing the time consumption significantly. Figure 1 shows a high-level overview of this framework. Fig. 1: High-level overview of Mockingbird . Mock functions redirect ordinary function calls to the LLM, instructing the LLM to initiate reasoning and then generate the return value. Mock trainers use feedback to instruct the LLM to conduct reflection process on its previous errors. Optional modules such as substitution script, memory compression and memory replacing are introduced to further improve the practicality of this framework. This framework offers an intuitive interface for systems that have difficulty adapting to chat-driven execution mode, thereby enabling them to leverage the capabilities of LLMs. This work investigates the potential of applying this framework to general machine learning tasks. Furthermore, we implement an extensible machine learning framework for mock functions , which is capable of automatically conducting reflection on the incorrect output during the training process. In comparison to conventional machine learning methods, such as those based on statistics or artificial neural networks, Mockingbird has the following distinctive advantages (a) the intrinsic knowledge of LLMs acquired from the pre-training data enables this framework to perform well on few-shot and zero-shot tasks; (b) this framework is not constrained by a strict input data schema, allowing it to process incomplete data entries with missing fields. (c) this framework is readily capable of utilizing tools and extracting information from non-structural sources, which are typically inaccessible to conventional machine learning techniques. The aforementioned advantages can serve to enhance the robustness and flexibility of automatic intelligent systems. We evaluate the general applicability of this framework across a range of machine learning tasks from Kaggle. Overall, it achieves acceptably competitive scores compared to conventional machine learning methods, even outperforming many human competitors on several datasets. II Mockingbird This framework is composed of following components: Mock Function A mock function is a function that is defined solely in terms of its method signature (types and other restrictions on parameters and return value), with optional documentation provided to describe its purpose and any remarks pertinent to its use, and it is role-played by LLMs. The fundamental responsibilities of mock functions can be summarized as follows: (a) providing LLMs with metadata to role-play the function; (b) handling conversions between program objects and chat messages in JSON format; (c) guaranteeing the formal correctness of responses generated by LLMs through the validation of JSON schema. Figure 2 illustrates the fundamental workflow of a mock function. Mock functions can be employed directly without training process; however, in such instances, it is advisable to provide comprehensive annotation regarding the anticipated behavior of these functions as documentation comments within the code itself. Fig. 2: Basic workflow of a mock function. Mock functions automatically handle the conversion between program objects and chat messages, thus communicating between the program and LLMs. Mock Invocation Mock invocations are designed to conveniently update history invocations in the context. They represent a mapping of invocations in the program space to request-response message pairs in the chat history. Mock invocations automatically update the contents of request and

Read the complete text