(Submitted on 29 Mar 2020)

Abstract: We present Catalyst.RL, an open-source PyTorch framework for reproducible and
sample efficient reinforcement learning (RL) research. Main features of
Catalyst.RL include large-scale asynchronous distributed training, efficient
implementations of various RL algorithms and auxiliary tricks, such as n-step
returns, value distributions, hyperbolic reinforcement learning, etc. To
demonstrate the effectiveness of Catalyst.RL, we applied it to a physics-based
reinforcement learning challenge “NeurIPS 2019: Learn to Move – Walk Around”
with the objective to build a locomotion controller for a human musculoskeletal
model. The environment is computationally expensive, has a high-dimensional
continuous action space and is stochastic. Our team took the 2nd place,
capitalizing on the ability of Catalyst.RL to train high-quality and
sample-efficient RL agents in only a few hours of training time. The
implementation along with experiments is open-sourced so results can be
reproduced and novel ideas tried out.