2025-09-29 · 117 citations · club pick

Training Agents Inside of Scalable World Models

Danijar Hafner, Wilson Yan, Timothy Lillicrap

World models Reinforcement learning & control

No peer-reviewed venue on record yet. 117 citations, 6 of them influential, as of the last refresh.

Abstract

World models learn general knowledge from videos and simulate experience for training behaviors in imagination, offering a path towards intelligent agents. However, previous world models have been unable to accurately predict object interactions in complex environments. We introduce Dreamer 4, a scalable agent that learns to solve control tasks by reinforcement learning inside of a fast and accurate world model. In the complex video game Minecraft, the world model accurately predicts object interactions and game mechanics, outperforming previous world models by a large margin. The world model achieves real-time interactive inference on a single GPU through a shortcut forcing objective and an efficient transformer architecture. Moreover, the world model learns general action conditioning from only a small amount of data, allowing it to extract the majority of its knowledge from diverse unlabeled videos. We propose the challenge of obtaining diamonds in Minecraft from only offline data, aligning with practical applications such as robotics where learning from environment interaction can be unsafe and slow. This task requires choosing sequences of over 20,000 mouse and keyboard actions from raw pixels. By learning behaviors in imagination, Dreamer 4 is the first agent to obtain diamonds in Minecraft purely from offline data, without environment interaction. Our work provides a scalable recipe for imagination training, marking a step towards intelligent agents.

arXiv comment: Website: https://danijar.com/dreamer4/

Ten-minute slide kit

Six slides is the whole talk: what was broken, what people tried, what these authors did, what the numbers say, where it falls over, and the sentence people should remember.

SLIDE 1

The problem

To solve complex tasks in embodied environments, intelligent agents need to deeply understand the world and choose successful actions. World models offer a promising approach towards this goal by learning to predict the future outcomes of potential actions from the perspective of an agent, such as a robot or a video game player. This way, world models equip agents with a deep understanding of the world and the ability to choose actions by planning or reinforcement learning in

SLIDE 2

What came before

Our world model is based on the paradigm of diffusion models16,17, where the network is trained to restore the a data point given a corrupted version . The signal level determines the mixture of noise and data and is randomized during training, where corresponds to pure noise and means clean data. We build on the flow matching formulation18,19 because of its simplicity, where the network predicts the velocity vector that points towards the clean data: The signal level is typically sampled…

SLIDE 3

The method

Danijar Hafner\* Wilson Yan\* Timothy Lillicrap **World models learn general knowledge from videos and simulate experience for training behaviors in imagination, offering a path towards intelligent agents. However, previous world models have been unable to accurately predict object interactions in complex environments. We introduce Dreamer 4, a scalable agent that learns to solve control tasks by reinforcement learning inside of a fast and accurate world model. In the complex video game Minecraft, the world model…

SLIDE 4

What they measured

We perform a wide range of experiments to evaluate and explore the capabilities of Dreamer 4. The majority of our experiments focus on Minecraft, a complex video game that features infinite open worlds including monsters and hundreds of items that can be mined or crafted, with raw pixel observations and low-level mouse and keyboard actions. We primarily use the VPT dataset15 that contains 2541 hours of contractor gameplay with 360p video and mouse and keyboard actions at 20 FPS. To improve generations…

SLIDE 5

Where it breaks

We present Dreamer 4, a scalable agent that learns to solve challenging control tasks by imagination training inside of a fast and accurate world model. Dreamer 4 is the first agent to obtain diamonds in Minecraft purely from offline data, without online interaction. This achievement demonstrates its learning successful long-horizon strategies in complex

SLIDE 6

One-line takeaway

Replaces RSSM with diffusion/transformer-based world models

Assembled from the paper's own PDF, parsed with its layout intact, 94,070 characters of it, then split on the paper's own section headings. Extractive, not generated: every sentence here is lifted from the paper. The takeaway line is the club's own one-liner from its reading list. Check it before you present it.

Figures worth putting on a slide

Figure 1:** Dreamer 4 learns to solve complex control tasks by reinforcement learning inside of its world model. We decode the imagined training sequences for visualization, showin
Figure 2:** World model design. Dreamer 4 consists of a causal tokenizer and an interactive dynamics model, which both use the same block-causal transformer architecture. The token
Figure 2.](#page-3-0) The tokenizer compresses video frames into continuous representations and the dynamics model predicts the representations given interleaved actions, both usin
Figure 3:** Agent performance in Minecraft without environment interaction. All methods have access to the same contractor dataset[15](#page-19-3) with image inputs and
Figure 4:** Agent ablations on the offline diamond challenge. We report success rates and time needed to reach an item for four milestone items. Dreamer 4 outperforms methods based
Figure 5:** Human interaction. A human player counterfactually interacts with the world model in real time via mouse and keyboard to perform the same task from the same initial ima

Presented at

Saturday, May 9, 2026

Saturday Robotics, session 7

Robotics & World Models Reading Club 07: Learning to Dream: World Models, Imagination, Path to Foundation Models for Control — Los Altos

Listed on the event page as “Training Agents Inside of Scalable World Models (2025)”. The arXiv title above is the record.

Why the club picked it. Replaces RSSM with diffusion/transformer-based world models