Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes

Authors: J Rae, JJ Hunt, T Harley, I Danihelka, A Senior, G Wayne, A Graves, T Lillicrap

We can recall vast numbers of memories, making connections between superficially unrelated events. As you read a novel, you’ll likely remember quite precisely the last few things you’ve read, but also plot summaries, connections and character traits from far back in the novel.

Many machine learning models of memory, such as Long Short Term Memory, struggle at these sort of tasks. The computational cost of these models scales quadratically with the number of memories they can store so they are quite limited in how many memories they can have. More recently, memory augmented neural networks such as the Differentiable Neural Computer or Memory Networks, have shown promising results by adding memory separate from the computation and solving tasks such as reading short stories and answering questions [e.g. Babi].

However, while these new architectures show promising results on small tasks, they use “soft-attention’’ for accessing their memories, meaning that at every timestep they touch every word in memory. So while they can scale to short stories, they’re a long way from reading novels.

In this work, we develop a set of techniques to use sparse approximations of such models to dramatically improve their scalability. In these sparse models only a tiny subset of the memory is touched at each timestep. Importantly, we show we can do this without harming the ability of the models to learn. This means that the sparse memory augmented neural networks are able to solve the same kind of tasks but require 1000s of times less resources, and look like a promising technique, with further refinement, for reading novels.

For further details and related work, please see the paper: https://arxiv.org/abs/1610.09027

Check it out at NIPS:

Wed Dec 7th 06:00 – 09:30 PM @ Area 5+6+7+8 #17

Source: https://deepmind.com/blog/article/deepmind-papers-nips-part-3