Curiosity-driven RL for symbolic equation solving
Kevin P. O’Keeffe · 2025
Read Curiosity-driven RL for symbolic equation solving by Kevin P. O’Keeffe (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17022.txt.
Introduction
Kevin P. O'Keeffe tests whether curiosity helps machines solve equations in a 2025 paper. Symbolic solving means rearranging an equation step by step. Earlier learning methods handled only simple one-variable linear equations. Harder equations involve roots, exponentials, and trigonometric functions.
O'Keeffe combines a standard reinforcement learning method with curiosity rewards. Curiosity rewards the agent for visiting unfamiliar problem states. Actions are chosen on a graph of possible equation forms. With that setup, the agent learned to solve nonlinear equations. The result is an early proof of concept. It suggests curiosity-driven exploration may help broader symbolic reasoning. Future systems might use the same drive for theorem proving or algebra tutoring.
Catalog record
| CREATOR | Kevin P. O’Keeffe |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-17022
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Curiosity-driven RL for symbolic equation solving Kevin P. O’Keeffe Abstract We explore if RL can be useful for symbolic mathematics. Previous work showed contrastive learning can solve linear equations in one variable. We show model-free PPO Schulman et al. 2017 augmented with curiosity-based exploration and graph-based actions can solve nonlinear equations such as those involving radicals, exponentials, and trig functions. Our work suggests curiosity-based exploration may be useful for general symbolic reasoning tasks. \workshoptitle MATH-AI 1 Introduction Reinforcement learning (RL) has been applied to diverse fields ( Ng et al. 2006 ; Degrave et al. 2022 ; Vinyals et al. 2019 ) but has not yet been widely adopted in symbolic mathematics, where agents perform tasks like solving algebraic equations or evaluating integrals analytically. Traditional approaches to symbolic mathematics rely on hand-engineered systems like Mathematica or Maple. An RL-based approach, where agents learn transformations autonomously, could reduce manual curation and find novel solution techniques. Applying RL to symbolic math is difficult because the state space is combinatorially large, as each equation can branch into numerous sub-expressions. The action space is also large and dynamic: at every step, the agent can apply various algebraic manipulations to different sub-expressions, so the number and type of actions change with the equation’s form. We here show RL agents combined with curiosity-based exploration and a graph-based action space can overcome these challenges. The agents learn to solve a wide range of algebraic equations such as those involving radicals and trig functions. Though perhaps simple from a human perspective, solving such equations using RL is non-trivial and to our knowledge has never before been done. Moreover, they are a key building block for more advanced tasks like solving differential equations. 2 Related Work Linear Equations with Primitive Action Spaces . Poesia et al. ( Poesia et al. 2021 ) pioneered the use of RL on symbolic equations. They showed agents trained with a contrastive loss can solve linear equations with integer coefficients. Their formulation differs from ours in that their action space is at a primitive granularity: states and actions are represented as raw text (strings of symbols), and each algebraic operation (associativity, commutativity, adding a constant, etc.) corresponds to a single low-level action. We represent equations as expression trees and define higher-level actions. They also use sparse rewards (we use dense rewards imbued with curiosity) and do not study nonlinear equations such as rational equations with complex functions (exp, sin, etc). See Appendix. Q-Learning for Symbolic Equations . Dabelow and Ueda ( Dabelow & Ueda 2024 ) used Q-learning in a SymPy-based framework, restricting the agent to linear equation of form a_{0}+a_{1}x=a_{2}+a_{3}x and a_{0}+b_{0}\,c\;+\;(a_{1}+b_{1}\,c)\,x\;=\;a_{2}+b_{2}\,c\;+\;(a_{3}+b_{3}\,c)\ with coefficients in \mathbb{Z},\mathbb{Q},\mathbb{R},\mathbb{C} . They do not study nonlinear equations. 3 Problem Formulation Our Markov Decision Process (MDP) formulation is Figure 1: Expression trees for ax+b and ax^{2}+bx+c . States : equations like ax+b=0 or cx+d=-x/b . We vectorize these using preorder traversal over the equations’ expression tree (Figure 1 ) and pad to a maximum length L=50 (suitable length for the equation sizes we here consider). Then we map operations and symbols to integers \{\text{add}:1,\text{sub}:2,\text{mul}:3,\ldots,x:5,a:6,\ldots\} . We encode both the lhs and rhs of the equation in this manner and then concatenate. An example embedding is x+a\rightarrow[add,x,a]\rightarrow[1,5,8,PAD,\dots] Actions : represented as (operation,term) pairs, such as (sub,b) or (div,a) . Formally, \displaystyle O \displaystyle=\{\,\mathrm{add},\ \mathrm{subtract},\ \mathrm{mult},\ \mathrm{divide}\,\}\cup\{\mathrm{square},\ \mathrm{sqrt},\ \mathrm{exp},\ \mathrm{log},\mathrm{sin},\mathrm{cos}\,\mathrm{asin},\ \mathrm{acos}\,\} (1) \displaystyle T \displaystyle=\operatorname{SubExpr}(\text{lhs})\cup\operatorname{SubExpr}(\text{rhs}), (2) \displaystyle A \displaystyle=(O\times T)\cup\{(Expand,None),(\mathrm{collect},x),(\mathrm{multiply},-1)\}. (3) Notice the first set of operations take two arguments, the second one argument. For the terms, we choose the list of sub-expressions in the lhs and rhs expression trees. Example of sub-expressions are \displaystyle ax+b=0\Rightarrow\{a,x,ax,b\} (4) \displaystyle(ax+b)/(cx+d)+e=0\Rightarrow\{a,x,ax,b,c,d,cx,cx+d,ax+b\} (5) This term set is expressive enough to solve rational equations and all other equation types we consider in this paper. It is also dynamic: the list of sub-expressions is derived from the equation/state and thus has variable length. Looping back to the operations, we also include (mul,-1) and \displaystyle\text{expand:}\quad cx+d+e(ax+b)\rightarrow aex+be+cx+d=0 \displaystyle\text{collect $x$:}\quad ax+bx\rightarrow(a+b)x\ These allow the agent to perform key algebraic steps (Appendix). Finally, we index the action set A serially, cap it at size |A|=50 1 1 1 We explored other values like |A|=40,70,100 and found no changes in performance , and mask illegal actions (e.g., division by zero, see Appendix). Rewards . Define the complexity C of an equation as the total number of nodes and edges in the expression tree 2 2 2 We judged number of nodes+edges correlates better with algebraic complexity than number of nodes alone. . The complexities of the equations in Figure 1 are C(ax+b)=5+4=9 and C(ax^{2}+bx+c)=10+9=19 . The reward is then \displaystyle R(\text{action})=C(\text{equation})-C(\text{equation after action}) (6) The intuition here is to encourage the agent to take actions which simplify the equation. State Transition Function . We wrote our code in Python and used SymPy to apply operations to the equations. At each step, we keep track of a lhs and rhs of an equation e.g. lhs=(ax+b) and rhs=0 . We apply actions to both the lhs and rhs. For example, (sub,b) results in (lhs,rhs)=(ax,-b) , and then (div,a) results in (lhs,rhs)=(x,-b/a) . The terminal condition for the environment is when lhs=x and the substitution of the rhs into the original equation simplifies to 0 . Limitations . Importantly, this MDP formulation only works on equations which are ‘closed,’ in the sense that solving them requires manipulating the terms already present in the equation/sub-expression list. By contrast, solving ’open’ equations requires adding new, out-of-equation terms or clever substitutions. A classic example is the quadratic equation ax^{2}+bx+c which is solved by completing the square – adding (b/2a)^{2} to each side. This is ’generative’ reasoning, since the term (b/2a)^{2} is not in the term set we have defined. Equations that require these more exotic actions are beyond the scope of the current work (and were also not studied in all previous works Poesia et al. 2021 ; Dabelow & Ueda 2024 ). 4 Results Fixed equation environments . We begin with six simple algebraic equations displayed in the first column of Table 1 . The first two, ax+b,a/x+b , are easily solved, and are included as baselines. The remaining four are more challenging, having a nested structure and more unknowns. Equations of this form are found in elementary textbooks on algebra, making them natural first test cases. Table 1: Mean success rates ± s.e. over N_{\text{trial}}=10 , N_{\text{train}}=3\times 10^{6} . Equation A* A2C PPO PPO-ICM PPO-RIDE PPO-NGU PPO-RND ax+b 1.00\pm 0.00 1.00\pm 0.00 1.00\pm 0.00 1.00\pm 0.00 1.00\pm 0.00 1.00\pm 0.00 1.00\pm 0.00 a/x+b 1.00\pm 0.00 1.00\pm 0.00 1.00\pm 0.00 1.00\pm 0.00 1.00\pm 0.00 1.00\pm 0.00 1.00\pm 0.00 c(ax+b)+d 1.00\pm 0.00 0.20\pm 0.13 0.10\pm 0.09 0.70\pm 0.14 0.60\pm 0.15 0.40\pm 0.15 1.00\pm 0.00 c+d/(ax+b) 0.00\pm 0.00 0.00\pm 0.00 0.20\pm 0.13 0.50\pm 0.16 0.40\pm 0.15 0.50\pm 0.16 0.80\pm 0.13 (ax+b)+e(cx+d) 0.00\pm 0.00 0.00\pm 0.00 0.00\pm 0.00 0.20\pm 0.13 0.00\pm 0.00 0.20\pm 0.13 0.40\pm 0.15 e+(ax+b)/(cx+d) 0.00\pm 0.00 0.00\pm 0.00 0.00\pm 0.00 0.10\pm 0.09 0.00\pm 0.00 0.30\pm 0.14 0.50\pm 0.16 We consider a fixed equation environment, where the agent tried to solve a single equation during every episode. We used the A2C and PPO algorithms from Stable Baselines 3 Raffin et al. 2021 and a homemade A^{*}