A Generic and Efficient Python Runtime Verification System and its Large-scale Evaluation

Zhuohang Shen, Mohammed Yaseen, Denini Silva, Kevin Guan, Junho Lee, Marcelo d’Amorim, Owolabi Legunsen · 2025

Read A Generic and Efficient Python Runtime Verification System and its Large-scale Evaluation by Zhuohang Shen, Mohammed Yaseen, Denini Silva, Kevin Guan, Junho Lee, Marcelo d’Amorim, Owolabi Legunsen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-06324.txt.

Introduction

Checking a running Python program against formal rules gets a general platform in PyMOP, a 2025 system led by Zhuohang Shen with six co-authors. Runtime verification has found hundreds of bugs in Java projects, but Python tools were domain-bound or slow. PyMOP supports five specification logics, five monitoring algorithms, and dozens of ready specifications for popular libraries, with three instrumentation strategies.

The evaluation is large: hundreds of thousands of unit tests across nearly fifteen hundred GitHub projects. The fastest Java-default algorithm is often not fastest for Python, PyMOP outruns two recent dynamic analysis systems by orders of magnitude, and dozens of the bugs it helped find are already fixed. The authors position it as infrastructure for Python verification research.

Catalog record

CREATORZhuohang Shen, Mohammed Yaseen, Denini Silva, Kevin Guan, Junho Lee, Marcelo d’Amorim, Owolabi Legunsen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-06324

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Generic and Efficient Python Runtime Verification System and its Large-scale Evaluation Zhuohang Shen, Mohammed Yaseen, Denini Silva, Kevin Guan, Junho Lee, Marcelo d’Amorim, Owolabi Legunsen Abstract Runtime verification (RV) now scales for testing thousands of open-source Java projects, helping find hundreds of bugs. The popular Python ecosystem could use such benefits. But, today’s Python RV systems are limited to a domain or specification logic, or slow. We propose PyMOP, a generic, extensible, and efficient RV system for Python. PyMOP supports five logics, implements five existing monitoring algorithms, ships with 73 API specs of Python and widely-used libraries, supports three instrumentation strategies, and users can easily add more of these. On 290,133 unit tests in 1,463 GitHub projects, we find mainly that (i) the default monitoring algorithm for Java is often not the fastest for Python; (ii) PyMOP is up to 1 ​ t ​ r ​ u ​ e ​ 168.32 1true168.32 x faster than two recent dynamic analysis systems; and (iii) 44 of 121 bugs that PyMOP helped find so far were fixed by developers. PyMOP’s generality and efficiency position it well as an excellent platform for the next advances on RV for Python. 1 Introduction Runtime verification (RV) [ 145 , 99 , 81 ] monitors program runs against formal specifications (specs); it is now used in deployed systems [ 6 , 122 ] . RV also amplifies the bug-finding ability of tests. Monitoring passing tests in thousands of Java projects against specs of JDK APIs found hundreds of bugs [ 91 , 114 , 103 , 75 ] . Despite these advances, RV systems are limited for Python, an increasingly important software ecosystem [ 159 ] . For example, PythonRV [ 137 ] supports a fraction of Python syntax, and its overheads were not evaluated. LogScope [ 9 ] and PyContract [ 44 ] analyze logs offline; they do not work online, e.g. , in continuous integration (CI). Domain-specific Hat-RV [ 172 ] and VyPR2 [ 48 , 92 , 47 ] only target embedded systems and performance monitoring of web services, respectively. DynaPyt [ 59 ] and DyLin [ 58 ] are dynamic analysis systems; they perform RV if their manually written “checkers” are seen as monitors. In sum, today’s Python RV systems have one or more of these limitations: (i) They are hard-wired to a spec logic, but no logic can express all specs [ 140 ] and users may want other logics [ 156 ] . (ii) Those that use formal logic, e.g. , linear temporal logic (LTL) can only monitor regular patterns; they cannot monitor, e.g. , context-free [ 113 ] or Turing-complete [ 112 ] patterns. (iii) Those that require manually writing monitors cannot use algorithms for synthesizing efficient monitors from specs [ 82 , 148 , 141 , 113 , 112 , 158 , 146 , 85 ] . (iv) None uses parametric monitoring algorithms [ 29 , 10 , 109 ] that help RV scale for testing in Java. (v) None was evaluated at scale with dozens of specs and thousands of projects like for Java [ 91 , 75 ] . (vi) None supports online and offline monitoring. We propose PyMOP to address these limitations; it is a generic and efficient RV system for Python and the first Python instance of Monitoring-Oriented Programming (MOP) [ 31 ] . MOP scaled well for simultaneous monitoring of multiple specs [ 33 , 29 , 109 , 49 ] . Also, JavaMOP [ 94 , 109 ] (the MOP instance for Java) spurred a lot of RV research on RV for Java [ 30 , 27 , 28 , 21 , 94 , 93 , 90 , 143 , 116 ] . PyMOP is generic in four ways. First, it is not hard-wired to a logic, so users can add plugins for more logics. PyMOP ships with five logic plugins (more planned in the future): past- and future-time LTL, extended regular expressions (ERE), FSM, and context-free grammars (CFG). Second, PyMOP does not require a DSL; users can write specs directly in Python or in a JSON-like frontend. Third, PyMOP has an API to support different instrumentation strategies; it currently supports three. Lastly, PyMOP is not restricted to a domain—the API specs that we check concern Python and widely used libraries, e.g. , Tensorflow (machine learning), SciPy (scientific computing), and Pandas (data analysis). PyMOP’s efficiency comes from two sources: it (i) automatically synthesizes efficient monitors from specs; and (ii) uses parametric trace-slicing based monitoring algorithms [ 33 , 29 ] (§ 3.1 has a trace slicing primer). We implement PyMOP as a plugin for \mathtt{pytest} [ 125 ] , the most popular Python testing framework. We also conduct a large-scale evaluation, using (i) 290,133 unit tests in 1,463 projects; (ii) three tools: PyMOP, DyLin, and DynaPyt; (iii) 73 specs of correct API usage in Python and widely-used libraries, 12 of which relate multiple object types; (iv) five monitoring algorithms; and (v) three instrumentation strategies. The projects come from many domains: machine learning, data processing, infrastructure-as-code, blockchains, etc. We find that the most complex online monitoring algorithm (and the only one in JavaMOP) is the fastest in 69.3% projects by 58.50 seconds on average (median: 0.98 seconds). But, times for all four online monitoring algorithms are within 2 seconds in 849 projects, and within 5 seconds in 979 projects. That is, if users do not mind 2s or 5s slowdowns, the choice of algorithm does not matter for 59.1% and 68.1% of these projects, respectively, using these specs. We also compare PyMOP with DynaPyt [ 59 ] and DyLin [ 58 ] —two recent Python dynamic analyzers—using 22 specs that DynaPyt and DyLin support. All three tools monitor the code under test (CUT). PyMOP always monitors 3rd-party libraries (necessary for RV during testing [ 75 ] ) and the Python runtime (to check conformance with its own APIs). But, DyLin and DynaPyt cannot monitor the Python runtime, so we evaluate them without libraries (as their authors did [ 59 , 58 ] ), and with libraries. PyMOP with libraries and Python-runtime monitoring is up to 419.97x (avg: 6.33x) faster than running DyLin and DynaPyt without monitoring libraries. Those speedups translate to up to 1true851.32 seconds (avg: 34.96 seconds). DynaPyt with libraries succeeds only in 935 projects. In this mode, PyMOP is up to 1true168.32 x (avg: 456.86 x) faster. Those speedups translate to up to 3true483.90 seconds (avg: 2true016.24 seconds). Despite being generic and monitoring more code, PyMOP scales better than DyLin and DynaPyt . We so far reported 93 of 121 bugs that PyMOP helped us find to developers; 44 were confirmed or fixed; 12 were rejected. We also discovered four bugs in the Python interpreter itself [ 38 ] . One has been fixed in a newer version [ 39 ] , one is open [ 42 ] , and the other two were confirmed [ 41 , 40 ] . These results are promising: PyMOP’s generality, and the degree to which it outperforms specialized techniques, suggest that it is better than current alternatives as a platform for future research and development on RV for Python. This paper makes the following

Read the complete text