The pros and cons of using deep reinforcement learning or genetic algorithms to design control schemes for quantum state transfer on qubit chains Sofía Perón Santana (1) (2), Ariel Fiuri (1), Martín Domínguez (1), Omar Osenda (1) (2) Abstract In recent years, control methods based on different optimization techniques have shed light on the possibilities of processing information in many quantum systems. When exploring the transmission of quantum states, faster transmission times are mandatory to avoid the deleterious effects of multiple sources of decoherence that spoil the transmission process. In particular, using Reinforcement Learning to devise sequences of step-wise external controls provides good transfer policies at short transmission times. We present two approaches to control the transmission of quantum states in qubit chains using external controls to force the dynamical evolution of the chain state. The first approach relies on the well-known Genetic Algorithm to generate a sequence of external controls, while the second approach uses a variant of Reinforcement Learning. The Genetic algorithm achieves excellent transmission fidelity at as short transmission times as Reinforcement Learning, surpassing the fidelities achieved by the latter method. Nevertheless, the Reinforcement Learning method offers robust control policies when the control pulses are noisy enough, owing to an imperfect timing of the pulses, deficient control devices, or other sources of phase decoherence. We present the regime where each method is best suited to control the transmission of arbitrary qubit states. I Introduction Controlling the time evolution of the quantum state 6 ; 50 ; 37 is the fulcrum on which quantum technologies, such as computing 7 , quantum state transfer 20 ; 1 ; 68 ; 47 ; 9 , state preparation 51 , and cryptography, operate. The exquisite level of control needed to perform quantum algorithms that provide sensible answers is, at present, beyond the accuracy levels required to perform large-scale calculations, although the frontier between what is doable and not is moving by the day. Optimal Control Theory is a method commonly employed to construct external control pulses for quantum 59 ; 40 and classical systems, owing to its high accuracy and clear physical interpretation, which is grounded in the variational principle. Nevertheless, its use is far from direct, and when the number of actuators increases, the numerical cost can become prohibitive. There is a growing demand for methods that provide control policies for the increasing number of platforms where quantum information is processed 6 ; 50 ; 37 . From the design of control pulses for single-quantum gates to the control of shallow quantum circuits 38 , other machine learning methods are being developed or adapted for quantum systems 67 ; 45 ; 33 ; 28 . In the long run, controlling open quantum demands methods resilient against decoherence and, ideally, faulty implementations of the control pulses 57 ; 24 ; 44 . This goal suggests that those control methods that rely on variable environments to train an agent should be up to the task. Reinforcement learning methods naturally incorporate the described ingredients. The numerous implementations available for studying a given problem should allow choosing the most appropriate Reinforcement Learning version for the investigation at hand 26 ; 53 ; 34 . Regrettably, there is a lot of trial and error to overcome before gaining insight into which traits of a given version of the Reinforcement Learning method are indispensable and which are superfluous 6 ; 50 ; 39 ; 37 ; 41 . Among many problems, reinforcement learning methods allow the preparation of random pure states in systems of up to seven qubits using different control sets 41 , enhance the transmission of quantum states in qubit chains, and improve the performance of quantum gates. Although the transmission of quantum states achieved using Reinforcement Learning with Deep Q-Networks with an \epsilon greedy policy for short to moderate qubit chain lengths is not of a remarkable quality 67 . Nevertheless, it shows that Reinforcement Learning could provide control sequences compatible with fast transmission. Training reinforcement Learning models without noise or some disturbances does not result in a well-trained model 54 ; 58 . Indeed, for an open quantum system or one with faulty controls, training with noise is a matter of using it as an advantage, a problem, and transforming it into an opportunity. A trained model provides robust control sequences for a broad spectrum of situations 19 ; 55 , assuming the training included a large set of possibilities. Otherwise, faced with a slightly different problem, the model will not provide a sensible answer. On the other hand, in general, Genetic Algorithms 31 work best for systems without fluctuations and provide very good to excellent solutions to problems when a given quantity needs to be optimised. In most situations, the resulting controls fail when used on systems with fluctuations or faulty implementations. Qubit chains with site-dependent interactions, designed using the Genetic Algorithm (GA), have demonstrated excellent transmission performance 45 ; 33 ; 5 ; 36 ; 13 . The preparation of particular sets of quantum states using GA is possible. All in all, there is an increasing field of applications. In the most common cases analysed using the Genetic Algorithm, the setting results in tuning the parameters of a time-independent Hamiltonian. Nevertheless, more recently, the design of control pulses has been considered. In this work, we aim to compare the performance of both the Genetic Algorithm and the Reinforcement Algorithm when applied to the problem of transmitting quantum states over homogeneous qubit chains. The study of quantum state transfer with 61 ; 29 ; 1 ; 68 ; 42 ; 60 and without external controls 48 ; 32 ; 56 ; 49 ; 11 ; 15 ; 43 ; 52 , using qubit chains with site-dependent interactions 25 ; 45 ; 16 ; 69 or homogeneous ones 21 , has led to numerous protocols verified in different experimental setups, such as optical systems 10 ; 4 , superconducting qubit processors 27 , magnetic systems 66 ; 8 ; 65 , cold atoms 28 ; 3 , quantum dot-based qubit chains 62 ; 64 ; 14 ; 22 ; 30 and others 63 . As such, quantum state transfer is one of the tasks instrumental for assessing the utility and reach of every control technique 23 . GA is best suited for problems without fluctuations, whereas DRL algorithms find their best applications in problems with fluctuating environments. In situations with fluctuating environments, it is possible to validate the trained model 19 ; 55 . Even starting the physical system under study in the same initial state results in different answers, which, for a properly trained model, would produce high-quality quantum state transfer. So, using an appropriate noise model, we can compare the performance of control sequences designed using both types of algorithms to drive quantum state transfer in qubit chains. It makes sense to study the generation of control sequences and their properties first without fluctuations, for both types of algorithms, and then with fluctuations, paying attention to the appearance of new phenomena. We organised the paper as follows. In Section II , we present the qubit chain model Hamiltonian and details about the algorithms employed in the paper. Section III contains the results for the transmission probability obtained using the Genetic Algorithm. Section IV presents results about control strategies found using Reinforcement Learning and a comparison with the results obtained using the Genetic Algorithm. Section V is devoted to the effects of fluctuations when they are present on the time evolution of the quantum state. Here, we train and validate DQN models that generate control sequences to drive quantum state transfer. At the same time, by using control sequences generated by the GA without fluctuations, we test their robustness by adding fluctuations to those sequences. The fluctuations model employed in both the GA and DQN algorithms is the same. Finally, in Section VI , we summarise our results and present our conclusions. To make the presentation lighter and enhance the readability of the manuscript, we defer most technical details, tables with parameters, and precise definitions of some quantities to the Appendices. Appendix A contains what is understood by mutation type, crossover type, number of generations, saturation and other quantities that modify the behaviour of the GA. Most of the Figures which include data obtained using control sequences generated through the GA use the parameters listed in the Table included in this Appendix. Appendix B contains a pseudo-code detailing the main ingredients of the Deep Q-network algorithm employed in this work. DQN is a variant of Deep Reinforcement Learning methods, which depends on artificial neural networks to approximate the transition probabilities needed for the underlying Markov Decision Process. The architecture of the networks, learning rates, reward and other necessary quantities included in the pseudo-code are the subject of this Appendix. Appendices C and D are devoted to characterising the control sequences obtained using the GA, in particular, what control actions are involved in constructing successful control sequences and the quality of the quantum state transfer obtained using a fixed set of parameters for larger and larger systems. II Model and methods We will study the transmission of quantum states over qubit chains equipped with the XX Hamiltonian, H=-\sum_{i=1}^{N-1}J\left(\sigma_{i}^{x}\sigma_{i+1}^{x}+\sigma_{i}^{y}\sigma_{i+1}^{y}\right)+\sum_{i=1}^{N}h_{i}\sigma_{i}^{z}. (1) The ”external fields” \{h_{i}\} provide the control needed to force the time evolution from the initial state to the desired target. Different combinations of fields could be applied and analysed. Nevertheless, simple combinations are preferable. On modern quantum processors, applying a control pulse on the z direction to a given qubit is equivalent to using a phase gate on it. If the qubits have magnetic dipolar momentum, the control pulses are, effectively, external magnetic fields, but this is not the most general case. For instance, in superconducting qubits, the phase gate is implemented via a radio frequency pulse. The Hamiltonian in Eq. ( 1 ) commutes with the total magnetization, allowing the study of the transmission of quantum states in subspaces with a fixed number of excitations. To control the time evolution of the quantum state, we use control functions \{h_{i}\} that are piecewise constant functions of time. Over a given time interval, only a particular set of control functions is non-zero. So, for the time interval \tau_{j} , the control term of the Hamiltonian is given by \sum_{h_{i}(\tau_{j})\neq 0}h_{i}(\tau_{j})\sigma_{i}^{z}. (2) Calling H_{j} to the sum of the XX Hamiltonian and the control term valid for the \tau_{j} interval, we get that H_{j}=-\sum_{i=1}^{N-1}J\left(\sigma_{i}\sigma_{i+1}+\sigma_{i}\sigma_{i+1}\right)+\sum_{h_{i}(\tau_{j})\neq 0}h_{i}(\tau_{j})\sigma_{i}^{z}. (3) So, we assume that the quantum state transfer takes place at a transmission time T\in\left[0,t\right] , and t=\tau_{1}+\tau_{2}+\ldots+\tau_{n-1}+\tau_{n}. (4) Then, the vector state of the qubit chain at time t is given by \psi(t)=\exp{(i H_n \tau_n)}\exp{(i H_{n-1 }\tau_{n-1})}\ldots\exp{(i H_2 \tau_2)}\exp{(i H_1 \tau_1)}\psi(0)=U\psi(0). (5) So, controlling the quantum state transfer involves finding a sequence of controls \{h_{i}(\tau_{j})\},\quad\forall j=1,2,\ldots,n (6) such that the initial state, dwelling at one extreme of the qubit chain at t=0 , is transferred with as much fidelity as possible at the other extreme of the chain. At this stage, the transmission protocol changes depending on the number of qubits used to encode the initial state and other details. Nevertheless, we will focus on the simplest one, the transmission of an arbitrary one-qubit state. The probability of transmission of a one-qubit arbitrary quantum state, at time t , on the computational basis can be written as P(t)=|\langle\mathbf{N}|U(t)|\mathbf{1}\rangle|^{2}, (7) where U(t) is the evolution operator in Eq. ( 5 ), |\mathbf{1}\rangle=|1\otimes o\otimes 0\otimes\ldots\otimes 0\rangle and |\mathbf{N}\rangle=|0\otimes 0\otimes 0\otimes\ldots\otimes 1\rangle are the one-excitation states with the excitation located at the first and last sites of the chain, respectively. The averaged fidelity of transmission is a simple function of P(t) , f(t)=\frac{P(t)}{6}+\frac{\sqrt{P(t)}\cos{\gamma}}{3}+\frac{1}{2}, (8) where the average considers all the possible initial states, and \gamma is a phase difference, which satisfies that \cos{\gamma}=1 by applying an appropriate uniform and time-independent magnetic field to the chain. While in modern quantum superconducting processors (see Reference 12 and references terein), it is possible to control the time evolution of the quantum state by applying multiple one- and two-qubit gates simultaneously, we will use only one-qubit controls acting on a single qubit or a few of them. Mainly, we will consider that only a single h_{i}(\tau_{j}) is non-zero for a given time interval \tau_{j} . Figure 1: The cartoon in the figure depicts a system of N qubits and its time evolution. The initial state, shown at the leftmost extreme of the cartoon, corresponds to a one-excitation quantum state. The step-wise evolution operator for a given interval, U_{k}=U(\tau_{k}) , acts over all the qubits and is shown as a gate that applies over the whole system. The specific form of each evolution operator depends on the controls operating on the corresponding time interval. We call an action the set of controls h_{i}(\tau_{j}) that are different from zero for a given time interval \tau_{j} . The objective of the Genetic Algorithm and the Reinforcement Learning algorithm is to determine sequences of actions resulting in high-quality quantum state transfer. The cartoon in Figure 1 depicts the time evolution of the quantum state of a system of N qubits for a given sequence of actions. Without going into too much detail, it is worth mentioning the traits that enable formulating the quantum state transfer problem in a form amenable to study using GA and RL techniques. We define the temporal step \Delta t so, for all k , \tau_{k}=\Delta t . The actions belong to an action space , A , and there is an state space , S . The state of the system is a vector with 2N real coefficients derived from the N complex quantum vector state. All the possible states form the state space . Besides, it is useful to introduce the transition probability, which is the probability that a given action a drives the system from the state s(t) to the state s^{\prime}(t+\Delta t) . There are numerous ways to denote the transition probability, but we choose the notation P_{a}(s,s^{\prime}) . Applying an action that drives the state from s to s^{\prime} also produces an immediate reward, R_{a}(s,s^{\prime}) , which quantifies how promising the action is in terms of the task assigned to the agent. The tuple (S,A,P_{a},R_{a}) determines a Markov Decision Process (MDP) 53 ; 26 ; 35 . An objective of the MDP is the optimisation of the cumulative reward function through a policy, usually denoted by \pi . The policy that maximises the cumulative reward is called an optimal policy 53 ; 26 ; 35 . Deep Reinforcement Learning methods aim to learn the optimal policy by training an agent and are preferred when the dimension of the state space and the action space becomes too large. As we will see in more detail later in this work, the GA determines a sequence of actions which is the chromosome of the fittest individual in a population of control sequences. On the other hand, the DRL chosen by us to study quantum state transfer, the Deep Q-network, is an instance of an MDP. The agent of the DQN learns to recommend sequences. Finally, when comparing the results obtained using the GA and DQN methods, we enforce some fairness criteria, the same action sets, halting criteria, the computational cost evaluated in a reference computational system, and the best solution corresponds to the maximum value of the transmission probability found in a given time interval \left[0,t_{max}\right] . In this case, the best solution offered by the DQN method appears in any training episode, each one lasting from t=0 to t=t_{max} . These points are the subjects of Sections III and IV. We devote Section V to obtaining a well-trained agent that recommends control sequences that endure fluctuating dynamics. III Control policies designed using Genetic Algorithm The Genetic algorithm is biologically motivated, and its rules mimic rules found in the formation of new populations with chromosomes that come from the genetic pool of previous populations, with the addition of random mutations 31 . The purpose of the algorithm is to increase the fitness of the population, just as is the case in biology. The fitness quantifies how much a given chromosome is adapted and evaluated for all the individuals that form a population. As we consider a discrete algorithm, the offspring of a given population, together with a number of the parents, form the new generation of the population. Central to the successful execution of the algorithm are the number of parents allowed to mate, typically those with the highest fitness, the genes each parent contributes to the chromosome of a given offspring, and the number of parents, among other factors. All the quantities described above form the set of hyper-parameters of the algorithm 31 ; 18 , and their values affect the size and characteristics of the space where it searches for the optimal solution. In Appendix A , we present further details about the Genetic Algorithm and carefully describe the rules that produce the genetic pool of the successive generations. The fitness function requires further analysis since different choices could lead to quantitative and qualitatively distinct solutions. We will return to this point once we formulate the search for the optimal quantum state transfer problem in a problem adapted for the Genetic Algorithm. As stated in the previous Section, we seek a control sequence which results in high values of the probability of transmission. So, it is reasonable to consider the chromosome of an individual to be the set of actions that drives the dynamical evolution from the initial state up to the final state. Each gene on a chromosome corresponds to an action. The initial population required by the algorithm comprises the chromosomes of all individuals in the population, and each gene corresponds to a randomly selected action from the set of possible actions. The cartoon in Figure 2 depicts the main ingredients of our implementation of the genetic algorithm. Figure 2: The cartoon in the Figure presents the main ingredients of the Genetic Algorithm. a) The sixteen possible actions, each of which can appear on a control sequence at any position in it. b) An initial population of four individuals, each one endowed with its own chromosome. The chromosome contains the genes, each one an action. c) For each sequence c_{i} (or chromosome), the algorithm calculates its fitness f_{i} , which corresponds to the maximum value attained by the transmission probability along the time interval \sum_{k}\tau_{k} . d) The algorithm selects a set of fittest individuals (in this case, two parents). e) and f) the chromosomes of the new members of the population result from the application of two rules, uniform crossover and swap mutation. g) The new population contains the set of parents and their offspring. The cartoon depicts a case where a) there are sixteen possible actions and b) an initial population with four individuals, each one with its corresponding chromosome. The length of the chromosome is not specified, but it is the same for all the individuals. The fitness evaluation takes place in c). d) After the fitness evaluation, only the parents with the highest fitness remain in the population. The next generation individuals comprise the two parents and two offspring, whose chromosomes result from e) the crossover of the parents’ genes. Sometimes, a mutation occurs that swaps the position of some genes in the chromosome of an individual. For the fitness function, we choose the maximum value attained by the transmission probability over a time interval \left[0,t\right] , where t\lesssim N , since we seek fast transmission times. In other optimisation methods, such as Optimal Control Theory, the time at which transmission probability attains its maximum value is chosen a priori . Our results show that both the Genetic and Reinforcement Algorithms find control policies compatible with transmission times near the shortest times allowed by the Quantum Speed Limit 1 ; 67 ; 61 . However, the quality of the transmission differs. In this Section, we consider two types of actions. Following the idea that, with current technology, it is possible to apply one-qubit gates to all the qubits in a chain. The action-by-site approach considers that only one field h_{i}=h is non-zero in any particular time-bin \tau_{j} , where h is the strength of the control. As we also consider that \tau_{j}=\Delta t=t/n , where n is the number of controls applied (or the number of phase gates), the phase gate applied to, say, the k -th qubit is given by U_{phase}=e^{ih\Delta t\sigma_{k}^{z}}. (9) In actual implementations, the values of h and \Delta t are bound by what is doable with the control means at disposal. Nevertheless, a practical bound follows from the fact that a phase gate is periodic. So, in an actual case, the time interval \Delta t should be large enough to allow the control system to apply any phase gate such that h\Delta t\leq 2\pi . As a consequence, the number of one-qubit gates needed to explore the time interval \left[0,t\right] is given by N_{q}=\frac{t}{\Delta t}=\frac{3}{4}\frac{N}{\Delta t}, (10) where we assume that the transmission time t\lesssim 3N/4 . Having the previous analysis in mind, we tested the performance of the GA using different combinations of the parameters h and \Delta t and summarised our findings in Figure 3 . As the data points indicate, the GA identifies action sequences that result in high-quality quantum state transfer for many different values of both parameters. Figure 3: The figure shows the maximum value of transmission probability achieved by the genetic algorithm for a chain of N=64 spins for different values of h and \Delta t . For comparison purposes, we also implemented the control actions proposed by Zhang and collaborators in Reference 67 , where an action acts upon up to three qubits in one or the other extremes of the chain. In this case, the number of actions is the same irrespective of the chain length. We include this set of actions in Appendix A . Figure 4 shows the performance of the GA using both sets of actions. The results obtained using the action-by-site scheme outperform those obtained using the set of actions by Zhang and collaborators, even when the number of actions is similar. In what follows, we use \Delta t=0.15 , J=1 and h=100 , to meet our comparison criteria. Figure 4: The figure summarises the effect of the initial conditions on the values obtained for the transmission probability. The GA is a biased random search algorithm; consequently, it is always advisable to study the variability of the results obtained. Panel a) shows the maximum value obtained for the transmission probability considering 30 different initial populations. The solid dots correspond to the values of the transmission probability, the circular dots correspond to the maximum over all the realisations of the initial populations, and the square dots to the value obtained by averaging the maxima of the realisations. The site-by-site action set results are much stable than the action set with a fixed number of elements. Note the different sizes of the bar errors, which correspond to one standard deviation. We obtained the data points by imposing convergence criteria, by asking each algorithm to halt after the transmission probability reaches a target figure of 0.99 , after a total of 1000 generations or after 30 generations without an increment in fitness. Panel b) shows the time evolution obtained using the control sequences whose transmission probabilities appear in panel a). Panel a) of Figure 4 shows the best values found for the transmission probability using the two sets of actions described so far. The data points obtained using actions-by-site show that the quality of transmission remains remarkably high for all the chain lengths included in the figure; instead, the quality of the transmission strongly decays when the algorithm employs the action set proposed by Zhang and coworkers. In both cases, the solid square dots correspond to the best values found running the algorithm for M=10 different initial populations. The solid circular dots correspond to the averaged values of the maximum transmission probabilities found for each realisation of the initial populations. Not only are the transmission probabilities larger for the action-by-site actions set, but their standard deviations are much smaller. Panel b) shows the time evolution of the transmission probabilities calculated using the control sequences designed for the GA, corresponding to the sequences that result in the probabilities shown in panel a). The many red and blue curves correspond to the sequences of both action sets. The green curve shows the unforced autonomous evolution of the transmission probability. There are some remarkable facts appearing after a close examination of panel b). First, the action set proposed by Zhang generates action control sequences for which the transmission probability follows the natural evolution quite closely, up to the transmission time where the transmission probability produced by the natural evolution attains its maximum value. After this time, the control sequence prevents P(t) from decaying, driving the time evolution up to a new maximum value, at a transmission time that is slightly larger than the time where the transmission probability corresponding to the natural evolution attains its maximum, which is compatible with the Quantum Speed Limit. The action-by-site action set strategy generates control sequences such that the transmission probabilities do not follow the natural time evolution so closely, starting their growth a bit later than the growth shown by the natural time evolution. Nevertheless, the growth of the transmission probability forced by these control sequences outgrows the other control strategy and reaches a very high maximum at practically the quantum Speed Limit time. Finally, the two types of control sequences almost freeze the time evolution of the state, forcing the excitation to remain in the last site of the chain. IV Control policies designed using Reinforcement Learning Reinforcement learning is another machine learning method which proceeds by using a trial-and-error strategy. There are numerous variants, depending on the Markov decision process (MDP) formulation (state, action, and reward), as well as on-policy vs. off-policy, value-based vs. policy-based, among other possibilities 53 ; 26 . Also crucial in determining which variant of RL is appropriate for a given problem are the characteristics of the data used to learn. The book by Shengbo Eben Li 26 lists several Reinforcement Learning algorithms and Deep Reinforcement Learning (DRL) algorithms, describing their advantages and drawbacks. For instance, the Proximal Policy Optimisation (PPO) algorithm is used on Reference 41 to find control sequences for the state preparation problem, and Zhang and collaborators 67 used a Deep Q-Network algorithm (DQN) to improve the transmission of quantum states and reveal the constraints imposed by the Quantum Speed Limit to the efficiency of the transmission. DRL algorithms are beneficial, in addition to being a natural choice, when the product of the state space and the action space is huge, or the state space is continuous. In this case, the network’s role is to approximate the probability density of transitions between states given the set of actions that force temporal evolution. In this Section, we also use DQN to design control sequences using the two sets of actions described so far. We will keep the description of the DQN algorithm to a minimum, since both References 53 ; 26 describe it extensively. Nevertheless, we include definitions of the hyperparameters used by the algorithm and other quantities in Appendix B , and use the main text to present the relationship between the quantum state transfer problem and the characteristic functions that enter into the DQN algorithm. For our purposes, it is sufficient to say that the DQN algorithm relies on the Q-network, Q(s,a;\theta) , the target network Q(s,a;\theta^{-}) , and the replay memory needed to sample random transitions, where s is the state of the system, and a are the actions already introduced. \theta and \theta^{-} are the weights of each network, which change accordingly to different updating rules. In some contexts, \theta and \theta^{-} receive the names of the current and old parameters, respectively. The replay memory stores tuples e_{t}=(s_{t},a_{t},r_{t},s_{t+1}) , where s_{t} is the state of the system at time t , a_{t} is the action applied at this time to drive the system, r_{t} is the reward obtained, and s_{t+1} is the state after aplying the action a_{t} . The replay memory at time t comprises the previous experience tuples, D_{t}=(e_{0},e_{1},\ldots,e_{t}) . In the case of QST, the state at time t is given by s_{t}=(\Re{\psi_1 (t)},\Re{\psi_2 (t)},\ldots,\Re{\psi_N (t)},\Im{\psi_1 (t)},\Im{\psi_2 (t)},\ldots,\Im{\psi_N (t)}), (11) where \psi_{k}(t) is the k component of the chain vector state at time t . The updating of the current parameters minimises the loss function given by L(\theta_{i})=\mathbb{E}\left[\left(r+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime};\theta_{i}^{-})-Q(s,a_{i},\theta_{i})\right)^{2}\right], (12) where the expectation value runs over the actions a , the states s and s^{\prime} and the replay memory. We follow the notation used in Reference 35 for the weight parameters \theta and \theta^{-} . In the jargon used in RL, an agent drives the time evolution of the state (the quantum state of the chain) by applying one of the possible actions. At time t , the agent selects the action accordingly to the state and the transition policy, which is a list of probabilities that the agent takes action a when it is in the state s , and in doing so tries to maximise the discounted reward, defined as R_{t}=\sum_{i=1}\gamma^{i-1}\,r_{i}, (13) where r_{i} is the reward at time t_{i} , and \gamma is known as the discount factor. During training, the agent follows an \epsilon -greedy policy: with probability \epsilon , it selects an action uniformly at random from the action set (exploration), and with probability 1-\epsilon , it selects the greedy action a_{t}=\arg\max_{a}Q_{\theta}(s_{t},a) (exploitation). The value of \epsilon should be explicitly specified, including whether it is held constant or decayed over training episodes or time steps. During inference or deployment, performance is evaluated under a purely greedy policy (i.e., \epsilon=0 ), unless otherwise stated, so that reported results reflect the learned policy rather than exploratory behaviour. Appendix B contains the details about the training procedure, hyperparameters, and other definitions that shape the working of the DRL algorithm. We do not include these details since numerous works deal with this issue; see, for instance, References 53 ; 26 ; 67 ; 50 ; 37 . Figure 5 shows the results for the transmission probability using the DRL algorithm described above. The training procedure used to obtain the control sequences that produce a good quality quantum state transfer has been described extensively in References 67 and 17 . The results displayed in the Figure show that the DRL algorithm is unable to find control sequences compatible with high-quality quantum state transfer, except for short to very short chains. Compare panels a) of Figures 5 and 4 . On the other hand, panel b) of Figure 5 shows that for several realisations of the initial setting of the DRL algorithm, the time evolution of the transmission probability reaches a maximum for a time near the QSL time. Reference 67 pointed out this phenomenon, and it is the main advantage of this method. Nevertheless, several drawbacks hinder the utility of the algorithm. First of all, the control sequence that produces the maximum value of the transmission probability may appear at any of the numerous training episodes required to reach relatively stable values for the transmission probability along the time window where the DRL algorithm trains. Figure 5: The figure shows (a) the maximum transmission probabilities obtained using the DQN algorithm for spin chains of different lengths, and (b) the transmission probability as a function of time for a chain of 32 qubits. The action set used to obtain the data is the one proposed by Zhang and coworkers. For the values of the hyperparameters, see the Table 3 . While the initial state of the transmission problem is the same in all cases, that is not the case with the initial weights of the Q and target networks, even when the algorithm’s hyperparameters are kept unchanged. To test the robustness of the method, the algorithm randomly sorts the initial weights. Panel (a) presents results from 30 independent network realisations, reporting both the highest transmission probability achieved across all realisations and the average of the maximum transmission probabilities obtained in each realisation. The circular dots correspond to the averaged maximum probabilities, and the circular dots to the maximum obtained over all the realisations. The error bars correspond to the standard deviation of the data. Panel b) shows the transmission probability vs time. The black lines correspond to the forced evolution driven by the actions that achieve the maxima for the different network initial conditions, whose average and best value we plot in panel a). The green line corresponds to the autonomous, unforced time-evolution. The second drawback that hinders the utility of the DRL algorithm arises from the fact that, often, after all the training episodes, the neural network is unable to provide a control sequence that effectively leads to high-quality quantum state transmission. In other words, the method finds a good control sequence at some training episode. Nevertheless, later training episodes modify the neural network parameters, effectively impairing the neural network’s ability to provide good control sequences. A defective sampling of the transition probability by the neural network is usually responsible for this well-known phenomenon. Let us remember that Deep Q-networks replaced tabular methods in problems where the state and action spaces are too big, and this phenomenon appeared. Both fine-tuning the hyperparameters of the underlying network and adapting to fluctuating environments collaborate to alleviate the problem. In this case, the neural network obtained after the complete training process is robust, providing good control sequences which show resilient transmission even when the control sequence applied to the qubit chain differs from the one suggested by the trained model. Training under a fluctuating environment and the effect of poor implementation of the control sequences are the subjects of the study presented in the next Section. Figure 6: The figure shows the running time required by the genetic and the DQN algorithms to reach their respective convergence criteria for qubit chains of different lengths. The black and red dots correspond to the time employed by the GA and DQN algorithms, respectively. For the GA algorithm, the criterion that halts the run is achieving the target probability of 0.99 . The criterion that halts the algorithm run in the DQN method depends on the chain length. For larger chains, the DQN algorithm halts because no improvement in the maximum value of the transmission probability appears despite further training episodes. The hyperparameters used to obtain the data in the Figure are tabulated in 3 The third main drawback is the numerical cost of the Deep Q-network algorithm, at least when compared with the efficiency attainable by parallelising the GA. Figure 6 shows the time required by the two algorithms to reach their respective convergence criteria for chains of different lengths. For the chain lengths shown (from short to moderate length), the DQN algorithm employs running times between 2 and 3 orders of magnitude larger than those required by the GA. This difference explains why, for the GA, it is possible to automate the search for suitable parameters, test different cost functions and action sets. Instead, for the DQN algorithm, we must choose a reduced set of options to reach the best possible physical parameters and reward functions. V Robustness against noisy control pulses The inclusion of dynamic noise when simulating the time evolution of a forced qubit chain obeys several reasons. For instance, the dynamic noise can arise due to an imperfect implementation of the phase gates, noisy electronics, or because quantum systems are actually open. Having this in mind, and because we want to test the idea of training the DQN algorithm in a fluctuating environment, we proceed to introduce imperfect control sequences by using a random term. From the point of view of Reinforcement Learning, including a fluctuating environment is known as domain randomisation , a technique that forces the agent to learn robust policies. The application of this technique ensures that a single instance falls within the training distribution, making the model (in our case, the DQN model) more adaptable. See, for example, References 54 ; 58 . Closely related to the concept of robust models generated through domain randomisation is that of validation, a process that effectively checks the robustness of the model by testing the agent through several instances. See References 19 ; 55 . To add a fluctuating environment, we proceed as follows. At each temporal step and with a certain probability p , each component of the vector state changes as \psi_{k}(\tau_{j})\longrightarrow\psi_{i}(\tau_{j})\exp{(i \delta\xi_k)}, (14) where \delta is the noise amplitude and \{\xi_{k}\} are random independent variables uniformly distributed, such that \xi_{k}\in\left[-1,1\right] . The cartoon in Figure 7 depicts the time evolution of the quantum state of the chain under this updating rule. We model noise by using a noise gate N^{\zeta}_{\tau_{j}} which is defined as: N_{\tau_{j}}^{\zeta}\ =\left\{\begin{matrix}\begin{pmatrix}e^{i\delta\,\xi_{1}^{j}}&\dotsc&0\\ \vdots&e^{i\delta\,\xi_{k}^{j}}&\vdots\\ 0&\dotsc&e^{i\delta\,\xi_{N}^{j}}\end{pmatrix}\ &\text{if }\zeta\
60 , the best values do not reach this target. The averaged maximum transmission probability shows a more pronounced decay (circular dots). The increasing difficulty faced by the algorithm to attain the desired target is also signalled by the growing size of the error bar (shown using vertical segments).