2018-11-12 · ICML · 2,029 citations · club pick

Learning Latent Dynamics for Planning from Pixels

Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, James Davidson

Published at ICML (the arXiv record still lists it as a preprint). 2,029 citations, 253 of them influential, as of the last refresh.

Abstract

Planning has been very successful for control tasks with known environment dynamics. To leverage planning in unknown environments, the agent needs to learn the dynamics from interactions with the world. However, learning dynamics models that are accurate enough for planning has been a long-standing challenge, especially in image-based domains. We propose the Deep Planning Network (PlaNet), a purely model-based agent that learns the environment dynamics from images and chooses actions through fast online planning in latent space. To achieve high performance, the dynamics model must accurately predict the rewards ahead for multiple time steps. We approach this using a latent dynamics model with both deterministic and stochastic transition components. Moreover, we propose a multi-step variational inference objective that we name latent overshooting. Using only pixel observations, our agent solves continuous control tasks with contact dynamics, partial observability, and sparse rewards, which exceed the difficulty of tasks that were previously solved by planning with learned models. PlaNet uses substantially fewer episodes and reaches final performance close to and sometimes higher than strong model-free algorithms.

arXiv comment: 20 pages, 12 figures, 1 table

Ten-minute slide kit

Six slides is the whole talk: what was broken, what people tried, what these authors did, what the numbers say, where it falls over, and the sentence people should remember.

SLIDE 1

The problem

Planning is a natural and powerful approach to decision making problems with known dynamics, such as game playing and simulated robot control \(Tassa et al., 2012; Silver et al., 2017; Moravcík et al. ˇ , 2017\). To plan in unknown environments, the agent needs to learn the dynamics from experience. Learning dynamics models that are accurate enough for planning has been a long-standing

SLIDE 2

What came before

Previous work in model-based reinforcement learning has focused on planning in low-dimensional state spaces \(Gal et al., 2016; Higuera et al., 2018; Henaff et al., 2018; Comparison of PlaNet to the model-free algorithms A3C and D4PG reported by Tassa et al. \(2018\). The training curves for these are shown as orange lines in Figure 4 and as solid green lines in Figure 6 in their paper. From these, we estimate the number of episodes that D4PG takes to achieve…

SLIDE 3

The method

For planning, we need to evaluate thousands of action sequences at every time step of the agent. Therefore, we use a recurrent state-space model (RSSM) that can predict forward purely in latent space, similar to recently proposed models \(Karl et al., 2016; Buesing et al., 2018; Doerr et al., 2018\). This model can be thought of as a non-linear Kalman filter or sequential VAE. Instead of an extensive comparison to prior architectures, we highlight two findings that can guide future designs of dynamics models: our…

SLIDE 4

What they measured

We evaluate PlaNet on six continuous control tasks from pixels. We explore multiple design axes of the agent: the stochastic and deterministic paths in the dynamics model, iterative planning, and online experience collection. We refer to the appendix for hyper parameters \(Appendix A\) and additional experiments \(Appendices C to E\). Besides the action repeat, we use the same hyper parameters for all

SLIDE 5

Where it breaks

We present PlaNet, a model-based agent that learns a latent dynamics model from image observations and chooses actions by fast planning in latent space. To enable accurate long-term predictions, we design a model with both stochastic and deterministic paths. We show that our agent succeeds at several continuous control tasks from image observations, reaching performance that is comparable to the best model-free algorithms while using 200× fewer episodes and similar or less computation

SLIDE 6

One-line takeaway

First strong demonstration of planning directly in latent space (RSSM)

Assembled from the paper's own PDF, parsed with its layout intact, 68,526 characters of it, then split on the paper's own section headings. Extractive, not generated: every sentence here is lifted from the paper. The takeaway line is the club's own one-liner from its reading list. Check it before you present it.

Figures worth putting on a slide

Figure 1: Image-based control domains used in our experiments. The images show agent observations before downscaling to 64 × 64 × 3 pixels. (a) The cartpole swingup task has a fixe
Figure 2: Latent dynamics model designs. In this example, the model observes the first two time steps and predicts the third. Circles represent stochastic variables and squares det
Figure 3: Unrolling schemes. The labels si|j are short for the state at time i conditioned on observations up to time j. Arrows pointing at shaded circles indicate log-l
Figure 1.](#page-1-0) These environments provide qualitatively different challenges. The cartpole swingup task requires a long planning horizon and to memorize the cart when it is
Figure 4: Comparison of PlaNet to model-free algorithms and other model designs. Plots show test performance over the number of collected episodes. We compare PlaNet using our RSSM
Figure 5: Comparison of agent designs. Plots show test performance over the number of collected episodes. We compare PlaNet, a version that collects data under random actions (rand

Presented at

Saturday, May 9, 2026

Saturday Robotics, session 7

Robotics & World Models Reading Club 07: Learning to Dream: World Models, Imagination, Path to Foundation Models for Control — Los Altos

Listed on the event page as “PlaNet (2018)”. The arXiv title above is the record.

Why the club picked it. First strong demonstration of planning directly in latent space (RSSM)