2020-10-05 · ICLR · 1,321 citations · club pick

Mastering Atari with Discrete World Models

Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, Jimmy Ba

Published at ICLR (the arXiv record still lists it as a preprint). 1,321 citations, 173 of them influential, as of the last refresh.

Abstract

Intelligent agents need to generalize from past experience to achieve goals in complex environments. World models facilitate such generalization and allow learning behaviors from imagined outcomes to increase sample-efficiency. While learning world models from image inputs has recently become feasible for some tasks, modeling Atari games accurately enough to derive successful behaviors has remained an open challenge for many years. We introduce DreamerV2, a reinforcement learning agent that learns behaviors purely from predictions in the compact latent space of a powerful world model. The world model uses discrete representations and is trained separately from the policy. DreamerV2 constitutes the first agent that achieves human-level performance on the Atari benchmark of 55 tasks by learning behaviors inside a separately trained world model. With the same computational budget and wall-clock time, Dreamer V2 reaches 200M frames and surpasses the final performance of the top single-GPU agents IQN and Rainbow. DreamerV2 is also applicable to tasks with continuous actions, where it learns an accurate world model of a complex humanoid robot and solves stand-up and walking from only pixel inputs.

arXiv comment: Published at ICLR 2021. Website: https://danijar.com/dreamerv2

Ten-minute slide kit

Six slides is the whole talk: what was broken, what people tried, what these authors did, what the numbers say, where it falls over, and the sentence people should remember.

SLIDE 1

The problem

To successfully operate in unknown environments, reinforcement learning agents need to learn about their environments over time. World models are an explicit way to represent an agent's knowledge about its environment. Compared to model-free reinforcement learning that learns through trial and error, world models facilitate generalization and can predict the outcomes of potential actions to enable planning \(Sutton,

SLIDE 2

What came before

Model-free Atari The majority of agents applied to the Atari benchmark have been trained using model-free algorithms. DQN \(Mnih et al., 2015\) showed that deep neural network policies can be trained using Q-learning by incorporating experience replay and target networks. Several works have extended DQN to incorporate bias correction as in DDQN \(Van Hasselt et al., 2015\), prioritized experience replay \(Schaul et al., 2015\), architectural improvements \(Wang et al., 2016\), and distributional value learning…

SLIDE 3

The method

Danijar Hafner ∗ Google Research Timothy Lillicrap DeepMind Mohammad Norouzi Google Research Jimmy Ba University of Toronto World models summarize an agent's experience into a predictive model that can be used in place of the environment to learn behaviors. When inputs are high-dimensional images, it is beneficial to learn compact state representations of the inputs to predict ahead in this learned latent space \(Watter et al., 2015; Karl et al., 2016; Ha and Schmidhuber, 2018\). These models are called…

SLIDE 4

What they measured

We evaluate DreamerV2 on the well-established Atari benchmark with sticky actions, comparing to four strong model-free algorithms. DreamerV2 outperforms the four model-free algorithms in all scenarios. For an extensive comparison, we report four scores according to four aggregation protocols and give a recommendation for meaningfully aggregating scores across games going forward. We also ablate the importance of discrete representations in the world

SLIDE 5

Where it breaks

We present DreamerV2, a model-based agent that achieves human-level performance on the Atari 200M benchmark by learning behaviors purely from the latent-space predictions of a separately trained world model. Using a single GPU and a single environment instance, DreamerV2 outperforms top model-free single-GPU agents Rainbow and IQN using the same computational budget and training time. To develop DreamerV2, we apply several small modifications to the Dreamer agent \(Hafner et al.,

SLIDE 6

One-line takeaway

Introduces discrete latent states (categorical RSSM)

Assembled from the paper's own PDF, parsed with its layout intact, 83,107 characters of it, then split on the paper's own section headings. Extractive, not generated: every sentence here is lifted from the paper. The takeaway line is the club's own one-liner from its reading list. Check it before you present it.

Figures worth putting on a slide

Figure 1: Gamer normalized median score on the Atari benchmark of 55 games with sticky actions at 200M steps. DreamerV2 is the first agent that learns purely within a world model t
Figure 1.](#page-0-0) Learning successful behaviors purely within the world model demonstrates that the world model learns to accurately represent the environment. To achieve this,
Figure 2.](#page-2-0) The RSSM uses a sequence of deterministic recurrent states ht, from which it computes two distributions over stochastic states at each step. The posterior sta
Figure 2: World Model Learning. The training sequence of images xt is encoded using the CNN. The RSSM uses a sequence of deterministic recurrent states ht. At each step,
Figure 3.](#page-4-0) The KL loss both trains the prior and regularizes how much information the posterior incorporates from the image. The regularization increases robustness to n
Figure 3: Actor Critic Learning. The world model learned in [Figure 2](#page-2-0) is used for learning a policy from trajectories imagined in the compact latent space. The trajecto

Presented at

Saturday, May 9, 2026

Saturday Robotics, session 7

Robotics & World Models Reading Club 07: Learning to Dream: World Models, Imagination, Path to Foundation Models for Control — Los Altos

Listed on the event page as “Mastering Atari with Discrete World Models (2020)”. The arXiv title above is the record.

Why the club picked it. Introduces discrete latent states (categorical RSSM)