Large-Scale Study of
Curiosity-Driven Learning

Yuri Burda *
Harri Edwards *
Deepak Pathak *
OpenAI
OpenAI
UC Berkeley

Amos Storkey
Trevor Darrell
Alexei A. Efros
Univ. of Edinburgh
UC Berkeley
UC Berkeley
* alphabetical ordering, equal contribution
[Download Paper]
[GitHub Code]


Reinforcement learning algorithms rely on carefully engineering environment rewards that are extrinsic to the agent. However, annotating each environment with hand-designed, dense rewards is not scalable, motivating the need for developing reward functions that are intrinsic to the agent. Curiosity is a type of intrinsic reward function which uses prediction error as reward signal.

In this paper:
(a) We perform the first large-scale study of purely curiosity-driven learning, i.e. without any extrinsic rewards, across 54 standard benchmark environments, including the Atari game suite. Our results show surprisingly good performance, and a high degree of alignment between the intrinsic curiosity objective and the hand-designed extrinsic rewards of many game environments.
(b) We investigate the effect of using different feature spaces for computing prediction error and show that random features are sufficient for many popular RL game benchmarks, but learned features appear to generalize better (e.g. to novel game levels in Super Mario Bros.).
(c) We demonstrate limitations of the prediction-based rewards in stochastic setups.


Curiosity-Driven Learning Without Extrinsic Rewards

A snapshot of the 54 environments investigated in the paper. We show that agents are able to makeprogress using no extrinsic reward, or end-of-episode signal, and only using curiosity.

[Click here to download game-play videos of all 54 environments]
[no reward, only curiosity] [32MB]




Source Code and Environment

We have released the TensorFlow based implementation on the github page. Try our code!
[GitHub]


Paper and Bibtex

[Paper]  [ArXiv]

Citation
 
Yuri Burda, Harri Edwards, Deepak Pathak,
Amos Storkey, Trevor Darrell and Alexei A. Efros. Large-Scale Study of Curiosity-Driven Learning
In arXiv:1808.04355 2018.

[Bibtex]
@inproceedings{pathak18largescale,
  Author = {Burda, Yuri and
  Edwards, Harri and Pathak, Deepak and
  Storkey, Amos and Darrell, Trevor and
  Efros, Alexei A.},
  Title = {Large-Scale Study of
  Curiosity-Driven Learning},
  Booktitle = {arXiv:1808.04355},
  Year = {2018}
}


Related Work

Pathak, Agrawal, Efros, Darrell. Curiosity-driven Exploration by Self-supervised Prediction.
In ICML 2017.[website]


Acknowledgements

We would like to thank Chris Lu for help in designing the Unity environments, Phillip Isola and Alex Nichols for feedback on an early draft of the paper. We are grateful to the members of BAIR and OpenAI for fruitful discussions and comments. DP is supported by the Facebook graduate fellowship.