Using economic value signals from primate prefrontal cortex in neuro-engineering applications Tevin Rouse, Shira Lupkin, Vincent McGinty Abstract 1. Center for Molecular and Behavioral Neuroscience, Rutgers University–Newark, Newark, NJ, USA. 2. Department of Neurobiology, University of Chicago, Chicago, IL, USA. Abstract Neural signals related to movement can be measured from intracranial recordings and used in brain-machine interface devices (BMI) to restore physical function in impaired patients. In this study, we explore the use of more abstract neural signals related to economic value in a BMI context. Using data collected from the orbitofrontal cortex in non-human primates, we develop deep learning-based neural decoders that can predict the monkey’s choice in a value-based decision-making task. Out-of-sample performance was improved by augmenting the training set with synthesized data, showing the feasibility of using limited training data. We further demonstrate that we can predict the monkey’s choice sooner using a neural forecasting module that is equipped with task-related information. These findings support the feasibility of user preference-informed neuroengineering devices that leverage abstract cognitive signals. Introduction Decoding neural signals is a primary focus of neuroengineering applications. By extracting the neural correlates of a particular cognitive process, neuroengineering systems like brain machine interfaces (BMIs) can help restore or augment that process. For example, BMI systems can partially restore motor functions by decoding movement-related signals from cortical motor areas, allowing users to control robotic limbs ( Velliste et al. (2008) , Oby et al. (2020) , Shenoy et al. (2003) ) or on-screen cursors ( Irwin et al. (2017) , Kim et al. (2008) , Gilja et al. (2012) , Jarosiewicz et al. (2015) ), Sussillo et al. (2012) , Fan et al. (2014) ). Recent work has expanded on these ideas to decode neural correlates of sensory signals ( Gao et al. (2014) , Guenther et al. (2009) , Hill et al. (2004) , Jia et al. (2017) , O’Doherty et al. (2011) , Heelan et al. (2019) ). Other work in non-human primates has focused on invasive BMI applications for the prefrontal cortex to decode signals related to eye movements and spatial locations in a virtual maze ( Johnston et al. (2023) ). Despite these advances, very few studies have explored the use of abstract cognitive signals related to subjective mental contents ( Card et al. (2024) , Wairagkar et al. (2023) , Chinchani et al. (2022) , Even-Chen et al. (2017) ) . By detecting signals that are related to a user’s goals or beliefs, systems can be developed to aid higher cognitive functions. In this work, we seek to explore the development of BMI systems that can extract abstract neural signals encoding economic values. Economic value (also called utility) is the subjective perception of the benefit or harm associated with a given stimulus, outcome, or course of action. Values are typically learned through experience and depend on an individual’s behavioral context, goals, and internal state. Neural representations of value form the foundation for goal-directed decision-making ( Castegnetti et al. (2021) , O’Doherty (2011) ). Thus, the ability to extract subjective value information from neural activity is an important step in decoding abstract mental contents such as goals and desires. The neural correlates of value have been extensively characterized in animal models including nonhuman primates ( McGinty and Lupkin (2023) , Sharma et al. (2024) , Padoa-Schioppa and Assad (2006) , Padoa-Schioppa and Conen (2017) , Thorpe et al. (1983) , Conen and Padoa-Schioppa (2015) , Kimmel et al. (2020) , Padoa-Schioppa (2013) , McGinty et al. (2016) , Kennerley et al. (2009) , Yamada et al. (2018) , Rich and Wallis (2016) , Padoa-Schioppa and Assad (2006) , Tremblay and Schultz (1999) ). Thus, this work seeks to operationalize neural representations of subjective value to explore the development of value-based BMI systems. Much of the work on the neural representation of subjective value has focused on the orbitofrontal cortex (OFC), a prefrontal cortical region that has been shown to be critical for value-driven behaviors. OFC neurons not only encode economic value, but the disruption of this encoding disrupts many kinds of decision-making behavior, in both humans and animal models ( Rudebeck and Murray (2008) , Rudebeck and Murray (2011) , Torregrossa et al. (2008) ). Given the role of OFC in the decision-making process, our objective is to assess whether OFC-derived signals can be used in a BMI context to help the user achieve their goal. In this study we use offline computational approaches to decode subjective values and intended choices from the activity of OFC neural ensembles, using an existing data set from monkeys performing a simple economic decision task. We use the performance of our neural choice decoder to gauge the potential of a BMI that relies on OFC-derived value signals. Because value signals are context-dependent and dynamic, we use a reinforcement learning approach in developing a value-based neural decoder. This framework permits updating the decoder with changes in a participant’s subjective evaluations. We will explore two challenges that need to be addressed to realize a value-focused BMI system using a deep learning framework: identifying neural network architectures that can potentially anticipate a user’s goals, and limited trial counts available for training the system. We will begin our assessment of BMI feasibility from the OFC by outlining a value-based two-alternative forced choice scenario. We will then demonstrate, using a transformer encoder-based agent (i.e. a neural decoder), the ability to extract value-related information from neural data recorded in nonhuman primates, and to use this information to predict choice behavior. Using this framework, we also explore the capabilities of a system that can make faster predictions by forecasting neural activity using multi-modal information within a decision-making context. We observed that we could obtain above-chance predictions on which item the monkey desired, including in cases where neither item was objectively better. We also observed that using forecasted neural activity allowed the decoder to make choice predictions sooner, but only when other information about the task conditions (i.e., the stimuli present in each trial) was incorporated into the forecasting process. Methods Problem Description Figure 1: Economic value-based BMI (brain machine interface) framework. (A) Shown is a two-alternative forced-choice economic decision task that the monkeys performed while OFC neural activity data was collected. For task and behavior details see Lupkin and McGinty (2023) . The monkey initiates a trial by fixating on a dot in the center of the screen, after which two covered stimuli appear. When the monkey saccades towards one of the stimuli, the stimulus in the direction of the saccade is revealed. Crowders (i.e., the visual elements surrounding each stimulus) force the monkey to look at the stimulus to gain information from it; they cannot identify the stimuli using peripheral vision. Each stimulus is associated with a reward ranging from 1-5 drops of juice; physically distinct stimuli can have the same reward association, meaning that in some trials there is no objectively better (largest reward) choice. Monkeys press a lever to indicate their choice, after which they receive the associated juice reward. (B) We consider a hypothetical scenario where a neural decoder extracts information from the neural activity obtained from the monkey while it is examining the stimuli. The decoder is tasked with extracting value information related to the stimuli and predicting the monkey’s choice. See Fig. 2A and Fig. 4A for model architectures tested in this study. The objective of this study is to design a value-based BMI system that predicts economic decisions. We tested this design by building a neural decoder that uses the neural activity collected from monkeys performing a two-alternative economic decision task (a subset of the data used in McGinty and Lupkin (2023) ). On each trial, the decoder takes a multivariate time series (i.e., spike count data from a group of cells over time) and extracts the relevant value signals needed to predict the monkey’s choice. To explore the effect of the decoder design on prediction performance, we consider two different decoder architectures, detailed below. All modeling was done using Pytorch 2.1.1 and scikit-learn 1.2.0. Behavioral Task We use data from McGinty and Lupkin (2023) , which used the following two-alternative forced-choice task to understand the neural mechanisms of value-based decision making. In this task, monkeys were tasked with examining two items on a display and selecting one (Fig. 1A). Each item, upon it being chosen, yields a fixed amount of juice reward of 1, 2, 3, 4, or 5 drops. On each trial, the monkeys were allowed to examine each option as long as they wanted, and to choose when ready. When the reward associations of the two items differed by 2 drops or more, the monkeys almost always chose the larger item; but when they were equal or near-equal, the monkeys’ decisions were variable from trial-to trial (see Lupkin and McGinty (2023) , their Fig. 1D). Data and Preprocessing We use 23 sessions of neural data collected from two rhesus macaques performing the value-based decision-making task in Fig. 1A (Monkey C: 10 sessions, Monkey K: 13 sessions); this is a subset of the data reported in McGinty and Lupkin (2023) .For this work we only used cells that were identified to be in the posterior portion of the orbitofrontal cortex. We excluded sessions with less than 10 cells and fewer than 400 trials as well as sessions that contained Na values. All analyses were performed within each session’s data. For a given session, the neural data consists of spike count data collected from single neurons in the orbitofrontal cortex (OFC, mean of 27.6 SEM 2.7 cells per session). Spikes from single cells were counted in 50 milliseconds (ms), non-overlapping time bins. Given the structure of the task, we examined data time locked to when the monkey viewed the second target in each trial, which is the time point at which the monkey has acquired all the information necessary to make a decision (i.e. the identities of both targets). We used spike count data from 200 ms before viewing the second target to 400 ms after viewing the second target. The neural data in each session had the following structure: Trials x (Number of Cells) x (12 Time Bins of 50ms each). The behavioral data includes the choice that the monkey made for each trial along with the identity of both stimuli. Generalizability for the trained agent was assessed by a series of random train-test splits, in which the fraction of training trials was varied between 20 \% to 80 \% in increments of 10 \% . For each session we perform only one train-test split at each of the training fractions tested. We performed this data split randomly since we had no reason to maintain the original trial order of the session. Since data was collected after the monkey was trained on a stimulus set (see Lupkin and McGinty (2023) , for details), we would not expect within-session learning effects if we had maintained the original trial order. Before model fitting and testing, we preprocessed the neural data as follows: for each cell and time bin in the series, we mean centered and scaled the data to unit variance, and then scaled the data again to the interval [-1,1] . The statistics used in the scaling process were derived for each cell using the portion of the data used in training. Reinforcement Learning Approach An agent (i.e. the neural decoder) was trained with an environment constructed from collected neural data. The agent was tasked with taking neural data as input and predicting which of the two stimuli the monkey chose on a given trial. (Note that choices were variable when the two targets indicated equal or near-equal rewards, see Lupkin and McGinty (2023) for details.) While there are several supervised approaches that could be used to train the agent, in this study we use a reinforcement learning (RL) approach because an RL-based agent could more easily be extended to accommodate the dynamic, context-dependent nature of many value-based choices in natural settings. For further details, see Discussion. In the RL setting used here, an environment consists of a series of states. Because our environment is built from the trials performed by the monkey, each state corresponds to a trial. At each state the agent would be provided spike count data from a single trial. The agent must then use the neural activity from the trial to predict whether the monkey chose the first item viewed or the second item viewed. When the agent’s choice matched the monkey’s choice (i.e. a correct prediction), it received a feedback signal from the environment of 1 , and received 0 otherwise. The agent was not provided with information about which item was viewed first, or which part of the neural activity corresponded to the first- or second-viewed item; thus, the agent must learn to extract the relevant information from the time course of each trial. Given that each session’s data was limited (mean of 631.4 trials), we added uniform noise (bound of 0.1 around 0 ) on each trial to create synthetic trial data to increase the size of the training set ( Sáiz-Abajo et al. (2005) , Holmstrom et al. (1992) ). This noise was added to the scaled data (see Data Preprocessing section) and clipped to maintain the [-1,1] value range. We explored values for the noise’s bounds between 0.1 and 0.2 but these variations were quantitatively similar. Training was organized into episodes. Each episode consisted of a set of training trials in which the agent uses the neural signal to render a prediction about the monkeys’ choice and receives feedback about whether the prediction was correct. Thus, each training trial consists of a time series of activity from the recorded cells, a predicted choice, and a feedback signal. The neural time series used for training were synthesized from the original data as described above, with the number of unique synthesized training trials in each episode matching the fraction of trials used for training, which varied between 20\% and 80\% . Each subsequent episode used a different set of synthetic training trials. The agent completed 250 of these episodes, at which point we sampled a subset of 10,000 trials across all episodes and used these 10,000 trials to update the agent’s parameters. The process above was repeated for 300 iterations. Once the agent’s performance converged, we examined the agent’s choice prediction capabilities on held-out data absent added noise. All analyses related to the agent’s performance are evaluated in the noiseless case. All agent models were trained using a Q learning approach which is estimated by the equation Q(s,a)=r+g*max_{a^{{}^{\prime}}}(s^{{}^{\prime}},a^{{}^{\prime}}) . In this equation g corresponds to the discount factor. Because in this task there are no inter-trial dependencies (the decision on trial t does not bear on trial t+1 ), we only considered a discount factor value of g=0 . We used the mean squared error (MSE) as the loss function and Adam optimizer with an initial learning rate of 1e-4 for parameter updating. For agents that use an Internal Model to forecast neural activity (see Model Architecture below, results in Figs. 4 and 5), we used a two-step training process. The Internal Model is a type of variational autoencoder and so can be trained to reconstruct the input data. In the first step, the Internal Model (a Stochastic Recurrent Neural Network, SRNN) was trained to reconstruct neural activity using only the training data (i.e. not the held-out data). At each time step t , the ELBO (evidence lower bound) loss was computed. We trained this autoencoder minimizing the negative sum of the ELBO loss terms computed at each time t. After we trained the Internal Model, the decision-making agent was trained as described above with the internal SRNN model weights held fixed (i.e. frozen). With the Internal Model held fixed, the agent uses this model to reconstruct the provided neural data and rely on permitted portions of the reconstructions to make choice predictions (see Model Architecture section for details on data masking). The training process was performed until the agent’s performance converged. After training the internal-model-equipped agents, we consider two ways to examine its performance. The first approach (Fig. 4) requires the agent to make choice predictions using the reconstructions of the provided neural data produced by the Internal Model. This performance evaluation is equivalent to the conditions used in the training process. Performance values derived from this analysis will be considered as a performance upper bound for the agent. The second approach (Fig. 5) requires the agent to first forecast neural activity and then use this forecasted data to make choice predictions. By forecasting we mean that neural data predictions made by the Internal Model on an upcoming part of the trial are influenced by its neural data predictions on previous parts of the trial. Performance under this approach is closest to that of a hypothetical device providing assistive signals to a user in real time. To assess how much the Internal Model contributes to the agents performance, we trained a version of the Core model (see Model Architecture section for details) in the same manner as the internal-model-equipped agent. We will refer to the Core model trained in this manner as the BASE agent/model (Fig. 5). Model Architecture We considered two model variants. The objective of both models was to process multivariate time series data and extract the relevant neural signals needed to predict the monkey’s choice. Core model variant The first model variant used a single transformer encoder module to interpret the neural data. We refer to this model as the “Core model/agent”. The architecture for the “Core model” (Fig. 2) consisted of first embedding the data using a linear layer and then applying positional encoding (using a sine function) to maintain the temporal ordering. Using a Bert style approach ( Devlin (2018) ), the output along with a class token is then fed into a 2-layer transformer encoder. The first dimension (i.e., the dimension corresponding to the class token position) is then extracted and used for computing the Q values corresponding to the possible actions that the agent could take. The layer that outputs the predicted Q values contains two linear layers with an instance normalization layer between them. The action with the largest Q value is the one taken by the agent. Internal Model variant The second variant is like the Core model but also contains an environment model (referred to as “Internal Model” in Fig. 4) whose output is interpreted by a second transformer encoder. The Internal Model provides the agent with a forecast of future neural activity based on the neural activity observed up to a time point t