Solving the 2 2 -Level Atom Non-LTE Problem using Soft Actor-Critic Reinforcement Learning

Brandon Panos 1, Ivan Milić 2 1 University of Applied Sciences and Arts Northwestern Switzerland (FHNW), Bahnhofstrasse 6, 5210 Windisch, Switzerland 2 Institute for Solar Physics (KIS), Georges-Köhler-Allee 401a, 79110 Freiburg, Germany · 2025

Read Solving the 2 2 -Level Atom Non-LTE Problem using Soft Actor-Critic Reinforcement Learning by Brandon Panos 1, Ivan Milić 2 1 University of Applied Sciences and Arts Northwestern Switzerland (FHNW), Bahnhofstrasse 6, 5210 Windisch, Switzerland 2 Institute for Solar Physics (KIS), Georges-Köhler-Allee 401a, 79110 Freiburg, Germany (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15679.txt.

Introduction

Brandon Panos and Ivan Milić solve the two-level atom non-LTE radiative transfer problem with reinforcement learning in a 2025 paper. To read a star's air, physicists must work out how light and atoms settle into balance away from local equilibrium. Classic solvers lean on approximate operators and many rounds of iteration.

Their agent learns the depth-by-depth source function as if it were a control task. A radiative transfer engine scores each try, and a soft actor-critic method steers the policy by reward alone. No answer key and no gradient through the solver are needed. A simple setup shows the agent can find the self-consistent state.

Catalog record

CREATORBrandon Panos 1, Ivan Milić 2 1 University of Applied Sciences and Arts Northwestern Switzerland (FHNW), Bahnhofstrasse 6, 5210 Windisch, Switzerland 2 Institute for Solar Physics (KIS), Georges-Köhler-Allee 401a, 79110 Freiburg, Germany
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-15679

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Solving the 2 2 -Level Atom Non-LTE Problem using Soft Actor-Critic Reinforcement Learning Brandon Panos 1, Ivan Milić 2 1 University of Applied Sciences and Arts Northwestern Switzerland (FHNW), Bahnhofstrasse 6, 5210 Windisch, Switzerland 2 Institute for Solar Physics (KIS), Georges-Köhler-Allee 401a, 79110 Freiburg, Germany Abstract We present a novel reinforcement learning (RL) approach for solving the classical 2-level atom non-LTE radiative transfer problem by framing it as a control task in which an RL agent learns a depth-dependent source function S ⁡ ( τ ) S(\tau) that self-consistently satisfies the equation of statistical equilibrium (SE). The agent’s policy is optimized entirely via reward-based interactions with a radiative transfer engine, without explicit knowledge of the ground truth. This method bypasses the need for constructing approximate lambda operators ( Λ ∗ \Lambda^{*} ) common in accelerated iterative schemes. Additionally, it requires no extensive precomputed labeled datasets to extract a supervisory signal, and avoids backpropagating gradients through the complex RT solver itself. Finally, we show through experiment that a simple feedforward neural network trained greedily cannot solve for SE, possibly due to the moving target nature of the problem. Our Λ ∗ − Free \Lambda^{*}-\text{Free} method offers potential advantages for complex scenarios (e.g., atmospheres with enhanced velocity fields, multi-dimensional geometries, or complex microphysics) where Λ ∗ \Lambda^{*} construction or solver differentiability is challenging. Additionally, the agent can be incentivized to find more efficient policies by manipulating the discount factor, leading to a reprioritization of immediate rewards. If demonstrated to generalize past its training data, this RL framework could serve as an alternative or accelerated formalism to achieve SE. To the best of our knowledge, this study represents the first application of reinforcement learning in solar physics that directly solves for a fundamental physical constraint. \pagerange Solving the 2 -Level Atom Non-LTE Problem using Soft Actor-Critic Reinforcement Learning – References keywords algorithms – numerical methods – simulations – radiative transfer 1 Introduction Radiative transfer (RT) calculations are critical for interpreting light from diverse astronomical sources, including the early universe ( Korber et al. 2023 ) , supernovae ( Chen et al. 2022 ) , galaxy formations ( Sethuram et al. 2023 ) , stars, and our Sun ( Uitenbroek 2001 ) . Researchers infer physical variables (e.g., magnetic field strength, temperature, density, velocity) by analyzing spectra from specific atomic transitions sensitive to these local conditions. The inference process typically involves iteratively running forward RT simulations and adjusting parameters until the calculated spectrum matches the observation. The resulting parameters are then hypothesized to represent the object’s physical state. This method crucially assumes that the simulation accurately captures all necessary physics. However, rigorously modelling the radiation-matter interaction, especially under non-Local Thermodynamic Equilibrium (non-LTE) conditions relevant for many spectral lines, is often computationally intractable. The non-LTE conditions denote any departure of the system from the state of LTE and directly imply a strong, nonlinear coupling between the state of gas and radiation. Understanding this coupling is also extremely important in the context of radiative (magneto) hydrodynamical simulations, where the radiation also serves as a crucial means of energy transport ( Przybylski et al. 2022 ; Gudiksen et al. 2011 , e.g.) . As a result, researchers frequently adopt unwarranted simplifying assumptions such as plane-parallel geometry (effective 1 D atmospheres), complete frequency redistribution (CRD), stationary backgrounds, low-order numerical methods, and coarse spatial grids, which all call into question the reliability of the inferred physical parameters. The problem of RT involves solving a massive coupled set of linear equations, which in the non-LTE regime is done iteratively using techniques such as the Lambda Iteration (LI) or Accelerated Lambda Iteration (ALI) ( Rybicki & Hummer 1991 ; Rybicki & Hummer 1992 ; Rybicki & Hummer 1994 ) , Gauss-Seidel ( Trujillo Bueno & Fabiani Bendicho 1995 ) , implicit lambda ( Milić & Atanacković 2014 ) , multi-grid ( Steiner 1991 ; Štěpán & Trujillo Bueno 2013 ) , bi-conjugate gradient methods ( Anusha et al. 2009 ) , and matrix-free approaches ( Benedusi et al. 2023 ) . Despite algorithmic progress, accurate and complete simulations still remain computationally prohibitive. For this reason, researchers have started using machine learning-based methods to accelerate RT simulations, including learning a mapping between LTE and non-LTE level populations in 3 D ( Chappell & Pereira 2022 ) , predicting the departure coefficients of atomic level populations ( Vicente Arévalo et al. 2022 ) , and replacing the computational core of simulations with physics-informed neural networks (PINNs). Direct emulation of the simulation engine has resulted in impressive offline performance gains for numerical weather predictions ( Lagerquist et al. 2021 ) , the forward modeling of galaxy spectral energy distributions ( Sethuram et al. 2023 ) , and accurate predictions of the complete 4 D hydrogen fraction evolution of the Epoch of Reionization ( Korber et al. 2023 ) . Additionally, neural fields and PINNs have been applied to the inverse problem in solar physics, specifically for spectropolarimetric inversions to infer atmospheric parameters like the magnetic field ( Díaz Baso et al. 2025 ; Jarolim et al. 2025 ) . Despite these advancements, PINNs are still brittle under generalization, do not scale well, and struggle to function as RT emulators in real-time online tasks ( Mu et al. 2023 ) . Reinforcement learning (RL) represents a powerful branch of machine learning that remains underutilized within the sciences despite possessing qualities seemingly highly advantageous for simulation tasks. Many computational physics problems involve iterative schemes to reach a solution, which, when employing supervised function approximation or emulation, can result in a fast accumulation of errors. Unlike most approximation paradigms, RL optimizes policies for sequential decision-making by maximizing a cumulative long-term reward signal. This framework inherently balances exploration (sampling novel strategies) and exploitation (leveraging effective known strategies), enabling the potential discovery of non-intuitive or more efficient solutions. Additionally, the design of the reward function provides flexibility that can promote the exploration of fundamentally new solutions in the case of a reward sparse framework (usually requiring more episodes and therefore compute), while dense, structured rewards can incorporate domain knowledge which forces the policy into the realm of imitation or expert learning ( Zare et al. 2024 ) . RL has achieved unprecedented success within the gaming industry thanks to its ability to automate the algorithmic discovery process, resulting in superhuman performance in complex strategic games such as Chess ( Silver et al. 2017 ) , Go ( Silver et al. 2016 ) , and Dota 2 ( Berner et al. 2019 ) . By formulating scientific problems within an RL framework ("gamification"), researchers have achieved substantial breakthroughs, including predicting protein structures with atomic-level accuracy ( Jumper et al. 2021 ) , developing novel turbulence modeling strategies via multi-agent RL ( Novati et al. 2021 ) , and discovering faster algorithms for fundamental basic computations such as matrix multiplication ( Fawzi et al. 2022 ) . These examples illustrate the potential of RL in the sciences, beyond traditional game environments. In this work, we propose a novel solution to the RT problem that dispenses with manual heuristics and avoids constructing labeled datasets for supervision. Instead, our approach relies exclusively on a carefully designed reward function that allows an RL agent

Read the complete text