In fact, we found that the meta-RL agent could learn to quickly adapt in a wide domain of tasks with different rules and structures. And because the network learned how to adapt to a variety of tasks, it also learned general principles about how to learn efficiently.
Importantly, we saw that the majority of learning took place in the recurrent network, which supports our proposal that dopamine plays a more integral role in the meta-learning process than previously thought. Dopamine is traditionally understood to strengthen synaptic links in the prefrontal system, reinforcing particular behaviours. In AI, this means the dopamine-like reward signal adjusts the artificial synaptic weights in a neural network as it learns the right way to solve a task. However, in our experiments the weights of the neural network were frozen, meaning they couldn’t be adjusted during the learning process, yet, the meta-RL agent was still able to solve and adapt to new tasks. This shows us that dopamine-like reward isn’t only used to adjust weights, but it also conveys and encodes important information about abstract task and rule structure, allowing faster adaptation to new tasks.
Neuroscientists have long observed similar patterns of neural activations in the prefrontal cortex, which is quick to adapt and flexible, but have struggled to find an adequate explanation for why that’s the case. The idea that the prefrontal cortex isn’t relying on slow synaptic weight changes to learn rule structures, but is using abstract model-based information directly encoded in dopamine, offers a more satisfactory reason for its versatility.
In demonstrating that the key ingredients thought to give rise to meta-reinforcement learning in AI also exist in the brain, we’ve posed a theory that not only fits with what is known about both dopamine and prefrontal cortex but that also explains a range of mysterious findings from neuroscience and psychology. In particular, the theory sheds new light on how structured, model-based learning emerges in the brain, why dopamine itself contains model-based information, and how neurons in the prefrontal cortex become tuned to learning-related signals. Leveraging insights from AI which can be applied to explain findings in neuroscience and psychology highlights the value each field can offer the other. Going forward, we anticipate that much benefit can be gained in the reverse direction, by taking guidance from specific organisation of brain circuits in designing new models for learning in reinforcement learning agents.
This work was completed by Jane X. Wang, Zeb Kurth-Nelson, Dharshan Kumaran, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Demis Hassabis and Matthew Botvinick.
Source: https://deepmind.com/blog/article/prefrontal-cortex-meta-reinforcement-learning-system
