Understanding human metacontrol and its pathologies using deep neural networks

Abstract

Error monitoring is crucial for inferring how controllable an environment is, and thus for estimating the value of control processes (metacontrol). In this study, we use computational simulations with deep neural networks to investigate its behavioral and neural correlates. We trained both humans and deep reinforcement learning (RL) agents to perform a reward-guided learning task that required adaptation to changes in action controllability. Deep RL agents could only solve the task when designed to explicitly predict action prediction errors that fire in the medial prefrontal cortex. When trained this way, they displayed signatures of metacontrol that closely resembled those observed in humans. Moreover, when deep RL agents were trained to over- or underestimate controllability, they developed behavioral pathologies partially matching those of humans who reported depressive, anxious, or compulsive traits on transdiagnostic questionnaires. These findings open up avenues for studying metacontrol using deep neural networks.