Kernel-FFI: Transparent Foreign Function Interfaces for Interactive Notebooks
Hebi Li, Forrest Sheng Bao, Qi Xiao, Jin Tian · 2025
Read Kernel-FFI: Transparent Foreign Function Interfaces for Interactive Notebooks by Hebi Li, Forrest Sheng Bao, Qi Xiao, Jin Tian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-23205.txt.
Introduction
Li, Bao, Xiao, and Tian make programming languages talk to each other inside interactive notebooks. Data scientists in Jupyter-style notebooks constantly need a library that exists only in another language, but today's foreign function interfaces demand hand-written bindings and break on recursive calls or objects. Kernel-FFI rewrites cross-language calls automatically at the source level, so no manual setup is needed.
Foreign objects can be referenced and are cleaned up automatically across the language boundary. A side channel solves the notebook kernel's blocking behavior, allowing recursive and asynchronous calls to proceed. The authors describe the framework as language-agnostic and plan to release it openly, aiming to make multi-language notebook work feel like single-language work.
Catalog record
| CREATOR | Hebi Li, Forrest Sheng Bao, Qi Xiao, Jin Tian |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-23205
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Kernel-FFI: Transparent Foreign Function Interfaces for Interactive Notebooks Hebi Li, Forrest Sheng Bao, Qi Xiao, Jin Tian Abstract . Foreign Function Interfaces (FFIs) are essential for enabling interoperability between programming languages, yet existing FFI solutions are ill-suited for the dynamic, interactive workflows prevalent in modern notebook environments such as Jupyter. Current approaches require extensive manual configuration, introduce significant boilerplate, and often lack support for recursive calls and object-oriented programming (OOP) constructs—features critical for productive, multi-language development. We present Kernel-FFI, a transparent, language-agnostic framework that enables seamless cross-language function calls and object manipulation within interactive notebooks. Kernel-FFI employs source-level transformation to automatically rewrite cross-language invocations, eliminating the need for manual bindings or boilerplate. Kernel-FFI provides robust support for OOP by enabling foreign object referencing and automatic resource management across language boundaries. Furthermore, to address the blocking nature of Jupyter kernels and support recursive and asynchronous foreign calls, we introduce a novel side-channel communication mechanism. Our tool will be open-sourced and available at https://codepod.io/docs/kernel-ffi . 1. Introduction Modern software development increasingly relies on multi-language architectures to leverage the unique strengths of different programming languages. Python excels in data science and machine learning through its rich ecosystem of libraries such as TensorFlow and PyTorch, while JavaScript dominates web development with frameworks like React and Node.js. Similarly, C++ provides low-level control for performance-critical applications, and Lisp offers unparalleled flexibility for building Domain Specific Languages (DSLs). However, enabling seamless interaction between these languages remains a significant challenge, particularly in interactive development environments. Foreign Function Interfaces (FFIs) are the traditional approach for cross-language communication, allowing functions written in one language to be invoked from another. Interactive development environments have become essential tools for modern software development, particularly in domains requiring rapid prototyping and exploratory programming. Jupyter Notebooks ( Kluyver et al. 2016 ) exemplify this paradigm, offering a browser-based environment that integrates code, text, and execution results in a literate programming model. By supporting incremental evaluation through a Read-Eval-Print-Loop (REPL) model ( McCarthy et al. 1965 ; van Binsbergen et al. 2020 ) , Jupyter has achieved widespread adoption in data science, machine learning, and education ( Wang et al. 2020 ; Randles et al. 2017 ; Perkel 2018 ; Pimentel et al. 2019 ) . The 2024 GitHub Octoverse survey reports over 1.5 million GitHub repositories with Jupyter Notebooks, marking a 92% year-over-year increase ( GitHub 2024 ) . Despite supporting multiple language kernels, Jupyter kernels operate in isolation, with no native support for cross-language communication. This limitation significantly constrains the potential for multi-language development within the Jupyter ecosystem. The research landscape for Jupyter-based FFIs remains largely unexplored, presenting a significant gap in the interactive programming literature. While numerous FFI frameworks exist, they primarily focus on external C library bindings or client-server communications, often requiring extensive manual configuration and boilerplate code. These approaches are ill-suited for the dynamic, exploratory nature of modern interactive programming environments. Developing an FFI framework for Jupyter’s interactive programming paradigm presents several fundamental challenges. First, existing FFIs require substantial manual setup, including boilerplate code generation, memory management, separate library compilation, and complex data serialization. In Jupyter’s interactive exploratory programming paradigm, where developers incrementally write and modify code—adding and removing functions, changing function signatures, and redefining classes—the overhead of traditional FFI setup becomes prohibitive. Second, the blocking nature of Jupyter kernels complicates recursive foreign calls. Naive approaches that send foreign code directly to other kernels introduce blocking behavior, preventing the seamless integration required for interactive development workflows. Third, Object-Oriented Programming (OOP) is ubiquitous in modern programming languages and widely used in Jupyter code cells, yet existing FFIs often lack support for these paradigms. The inability to instantiate foreign classes, call methods on foreign objects, or maintain object state across language boundaries severely limits the practical utility of current FFI solutions. In response to these challenges, we present Kernel-FFI , a novel framework designed to enable seamless cross-language interoperability within Jupyter Notebooks. Kernel-FFI addresses the limitations of existing FFIs through three key innovations: • Transparent Source-Level Transformation: Kernel-FFI automatically rewrites cross-language invocations at the source level, allowing developers to call functions in other languages as if they were native. This eliminates the need for manual bindings and boilerplate code, significantly reducing development effort and enabling truly interactive multi-language development. • Foreign Object Referencing: Kernel-FFI provides comprehensive support for OOP paradigms by enabling developers to create object instances in one language and reference them in another. This enables seamless interaction with object-oriented code across languages, simplifying the management of stateful libraries and complex data structures. • Non-blocking Recursive Calls: Kernel-FFI introduces side-channel communication to enable non-blocking recursive calls between kernels. This allows developers to write interactive code that can seamlessly switch between languages without introducing latency or blocking behavior, supporting complex multi-language workflows. Kernel-FFI is designed to be language-agnostic, supporting a wide range of programming languages commonly used in data science, web development, and system programming. The framework leverages Jupyter’s existing messaging protocol to facilitate communication between kernels, ensuring compatibility with the Jupyter ecosystem while extending its capabilities. By providing a transparent and efficient interface for cross-language function calls, Kernel-FFI enables developers to leverage the strengths of different languages without the burden of complex setup or manual bindings. Our tool is open-sourced and available at https://codepod.io/docs/kernel-ffi . 2. Approach In this section, we first formulate the problem of cross-language function calls in Jupyter Notebooks, and then describe the approach to solve the problem. 2.1. Problem Formulation The problem of cross-language interoperability in Jupyter Notebooks is to enable programs written in a source language L_{s} to seamlessly use functions, classes, and variables defined in a target language L_{t} , in a way that is transparent, interactive, and robust to the dynamic nature of notebook development. To achieve this, we require a system that: • Dynamically discovers and maintains a registry of programming constructs (functions, classes, variables) defined in each kernel, reflecting the current state of the notebook as code is added, modified, or removed. • Automatically transforms cross-language usages in the source program P_{s} into a sequence of intermediate representations (IRs) that encode the intent of the foreign call. These IRs must support function calls, variable references, method calls on foreign objects, class instantiation, and object deletion. • Serializes and deserializes arguments and return values across language boundaries, including the ability to reference and manage foreign objects via a global variable/object store, preserving object identity and state. • Ensures semantic preservation, type safety, and state consistency across kernels, so that the transformed program P_{s}^{\prime} behaves as if all constructs were native to L_{s} . Formally, for each cross-language usage in P_{s} , the system must: (1) Identify the construct in the registry and generate an appropriate IR (as defined in Section 2.5 and the formal theory). (2) Serialize arguments (including object references) into a language-agnostic format. (3) Transmit the IR to the target kernel, where it is decoded, executed, and the result is re-encoded. (4) Deserialize the result in the source kernel, reconstructing native values or proxy objects as needed. (5) Update the global object store to maintain consistent state and enable further cross-language interactions. The system must guarantee that, for any sequence of cross-language operations, the observable behavior of P_{s}^{\prime} is semantically equivalent to