Mesocorticostriatal Reinforcement Learning of State Representation and Value with Implications for the Mechanisms of Schizophrenia

Abstract

Mesocorticostriatal dopamine projections are crucial for value learning, motivational control, and cognitive functions. However, while dopamine's role in value learning as reward-prediction-error (RPE) has been much understood, precise roles in motivational control and cognitive functions remain more elusive. Computationally, this corresponds to that while the operation of mesostriatal dopamine could be minimally described by simple reinforcement learning (RL) models with one-dimensional reward/RPE and fixed state representation, (1) how reward-specific motivational control can be achieved through heterogeneous dopamine responses, and (2) how sophisticated cortical state representation can be formed through mesocortical dopamine, cannot be captured by such simple models. To address both of these at once, we combined recent models for each of them: the “Reward Bases (RB),” which achieved reward-specific motivational control through multidimensional RPE (but with fixed cortical representation), and the “online value-recurrent-neutral-network (OVRNN),” which achieved state representation learning through training of RNN by RPE (but of one-dimensional). We show the combined model can achieve both functions simultaneously via double “feedback alignments” of the cortical and striatal downstream connections to the mesocorticostriatal dopamine projections. Crucially, cortical inhibition-dominance is a key for successful learning. Excessive excitation leads to aberrant persistent activity, which disrupts the alignments and impairs reward-specific motivational control and credit assignment. This implies how negative and positive symptoms of schizophrenia could emerge from excitation/inhibition imbalance, and we show how our model could explain altered brain activations in patients. Our model thus provides an integrated computational account for dopamine's functions, with implications on how its dysfunctions link to schizophrenia.