2018-03-27 · 1,987 citations · club pick

World Models

David Ha, Jürgen Schmidhuber

World models Reinforcement learning & control

No peer-reviewed venue on record yet. 1,987 citations, 99 of them influential, as of the last refresh.

Abstract

We explore building generative neural network models of popular reinforcement learning environments. Our world model can be trained quickly in an unsupervised manner to learn a compressed spatial and temporal representation of the environment. By using features extracted from the world model as inputs to an agent, we can train a very compact and simple policy that can solve the required task. We can even train our agent entirely inside of its own hallucinated dream generated by its world model, and transfer this policy back into the actual environment. An interactive version of this paper is available at https://worldmodels.github.io/

Ten-minute slide kit

Six slides is the whole talk: what was broken, what people tried, what these authors did, what the numbers say, where it falls over, and the sentence people should remember.

SLIDE 1

The problem

Humans develop a mental model of the world based on what they are able to perceive with their limited senses. The decisions and actions we make are based on this internal model. Jay Wright Forrester, the father of system dynamics, described a mental model as: *The image of the world around us, which we carry in our head, is just a

SLIDE 2

What came before

There is extensive literature on learning a dynamics model, and using this model to train a policy. Many concepts first explored in the 1980s for feed-forward neural networks (FNNs) \(Werbos, 1987; Munro, 1987; Robinson & Fallside, 1989; Werbos, 1989; Nguyen & Widrow, 1989\) and in the 1990s for RNNs \(Schmidhuber, 1990a;b; 1991a; 1990c\) laid some of the groundwork for *Learning to Think* \(Schmidhuber, 2015a\). The more recent PILCO \(Deisenroth & Rasmussen, 2011; Duvenaud, 2016; McAllister & Rasmussen, 2016\)…

SLIDE 3

The method

We present a simple model inspired by our own cognitive system. In this model, our agent has a visual sensory component that compresses what it sees into a small representative code. It also has a memory component that makes predictions about future codes based on historical information. Finally, our agent has a decision-making component that decides what actions to take based only on the representations created by its vision and memory

SLIDE 4

What they measured

In this section, we describe how we can train the Agent model described earlier to solve a car racing task. To our knowledge, our agent is the first known solution to achieve the score required to solve this

SLIDE 5

Where it breaks

We have demonstrated the possibility of training an agent to perform tasks entirely inside of its simulated latent space dream world. This approach offers many practical benefits. For instance, running computationally intensive game engines require using heavy compute resources for rendering the game states into image frames, or calculating physics not immediately relevant to the

SLIDE 6

One-line takeaway

Introduced the core idea of learning a compressed latent simulator for control

Assembled from the paper's own PDF, parsed with its layout intact, 102,050 characters of it, then split on the paper's own section headings. Extractive, not generated: every sentence here is lifted from the paper. The takeaway line is the club's own one-liner from its reading list. Check it before you present it.

Figures worth putting on a slide

Figure 1. A World Model, from Scott McCloud's *Understanding Comics.* [\(McCloud,](#page-17-0) [1993;](#page-17-0) [E,](#page-15-0) [2012\)](#page-15-0)
Figure 2. What we see is based on our brain's prediction of the future [\(Kitaoka,](#page-17-0) [2002;](#page-17-0) [Watanabe et al.,](#page-20-0) [2018\)](#page-20-0).
Figure 3. In this work, we build probabilistic generative models of OpenAI Gym environments. The RNN-based world models are trained using collected observations recorded from the a
Figure 4. Our agent consists of three components that work closely together: Vision (V), Memory (M), and Controller (C)
Figure 5. Flow diagram of a Variational Autoencoder (VAE).
Figure 6. RNN with a Mixture Density Network output layer. The MDN outputs the parameters of a mixture of Gaussian distribution used to sample a prediction of the next latent vecto

Presented at

Saturday, May 9, 2026

Saturday Robotics, session 7

Robotics & World Models Reading Club 07: Learning to Dream: World Models, Imagination, Path to Foundation Models for Control — Los Altos

Listed on the event page as “Ha & Schmidhuber — World Models (2018)”. The arXiv title above is the record.

Why the club picked it. Introduced the core idea of learning a compressed latent simulator for control