Dorsal-Ventral Reinforcement Learning Network Connectivity and Incentive-Driven Changes in Exploration

Abstract

Probabilistic reinforcement learning (RL) tasks assay how individuals make decisions under uncertainty. The use of internal models (model-based) or direct learning from experiences (model-free), and the degree of choice stochasticity across alternatives (i.e., exploration), can all be influenced by the state space of the decision-making task. There is considerable individual variation in the balance between model-based and model-free control during decision-making, and this balance is affected by incentive motivation. The effect of variable reward incentives on the arbitration between model-based and model-free learning remains understudied, and individual differences in neural signatures and cognitive traits that moderate the effect of reward on model-free/model-based control are unknown. Here we combined a two-stage decision-making task utilizing differing reward incentives with computational modeling, neuropsychological tests, and neuroimaging to address these questions. Results showed the prospect of greater reward decreased exploration of alternative options and increased the balance toward model-based learning. These behavioral effects were replicated across two independent datasets including both sexes. Individual differences in processing speed and analytical thinking style affected how reward altered the dependence on both systems. Using a systems neuroscience-inspired approach to resting-state functional connectivity, we found reduced exploration of the options during the first stage of our task under high relative to low incentives was predicted by increased cross-network coupling between ventral and dorsal RL circuitry. These findings suggest that integrity of functional connections between stimulus valuation (ventral) and action valuation (dorsal) RL networks is associated with changes in the balance between explore–exploit decisions under changing reward incentives.