Mastering Diverse Domains through World Models

Danijar Hafner,¹² Jurgis Pasukonis,¹ Jimmy Ba,² Timothy Lillicrap¹

Abstract

Developing a general algorithm that learns to solve tasks across a wide range of applications has been a fundamental challenge in artificial intelligence. Although current reinforcement learning algorithms can be readily applied to tasks similar to what they have been developed for, configuring them for new application domains requires significant human expertise and experimentation. We present DreamerV3, a general algorithm that outperforms specialized methods across over 150 diverse tasks, with a single configuration. Dreamer learns a model of the environment and improves its behavior by imagining future scenarios. Robustness techniques based on normalization, balancing, and transformations enable stable learning across domains. Applied out of the box, Dreamer is the first algorithm to collect diamonds in Minecraft from scratch without human data or curricula. This achievement has been posed as a significant challenge in artificial intelligence that requires exploring farsighted strategies from pixels and sparse rewards in an open world. Our work allows solving challenging control problems without extensive experimentation, making reinforcement learning broadly applicable.

<span id="page-0-0"></span>

Figure 1: Benchmark summary. a, Using fixed hyperparameters across all domains, Dreamer outperforms tuned expert algorithms across a wide range of benchmarks and data budgets. Dreamer also substantially outperforms a high-quality implementation of the widely applicable PPO algorithm. b, Applied out of the box, Dreamer learns to obtain diamonds in the popular video game Minecraft from scratch given sparse rewards, a long-standing challenge in artificial intelligence for which previous approaches required human data or domain-specific heuristics.

¹Google DeepMind. ²University of Toronto. Correspondence: mail@danijar.com ¹

<span id="page-1-0"></span>

Figure 2: Diverse visual domains used in the experiments. Dreamer succeeds across these domains, ranging from robot locomotion and manipulation tasks over Atari games, procedurally generated ProcGen levels, and DMLab tasks, that require spatial and temporal reasoning, to the complex and infinite world of Minecraft. We also evaluate Dreamer on non-visual domains.

Introduction

Reinforcement learning has enabled computers to solve tasks through interaction, such as surpassing humans in the games of Go and Dota[1](#page-11-0)[,2](#page-11-1) . It is also a key component for improving large language models beyond what is demonstrated in their pretraining data [3,](#page-11-2)[4](#page-11-3) . While PPO[⁵](#page-11-4) has become a standard algorithm in the field of reinforcement learning, more specialized algorithms are often employed to achieve higher performance. These specialized algorithms target the unique challenges posed by different application domains, such as continuous control[⁶](#page-11-5) , discrete actions[7](#page-11-6)[,8](#page-11-7) , sparse rewards[⁹](#page-11-8) , image inputs[¹⁰](#page-11-9), spatial environments[¹¹](#page-11-10), and board games[¹²](#page-11-11). However, applying reinforcement learning algorithms to sufficiently new tasks—such as moving from video games to robotics tasks requires substantial effort, expertise, and computational resources for tweaking the hyperparameters of the algorithm[¹³](#page-11-12). This brittleness poses a bottleneck in applying reinforcement learning to new problems and also limits the applicability of reinforcement learning to computationally expensive models or tasks where tuning is prohibitive. Creating a general algorithm that learns to master new domains without having to be reconfigured has been a central challenge in artificial intelligence and would open up reinforcement learning to a wide range of practical applications.

We present Dreamer, a general algorithm that outperforms specialized expert algorithms across a wide range of domains while using fixed hyperparameters, making reinforcement learning readily applicable to new problems. The algorithm is based on the idea of learning a world model that equips the agent with rich perception and the ability to imagine the future [14,](#page-11-13)[15,](#page-12-0)[16](#page-12-1). The world model predicts the outcomes of potential actions, a critic neural network judges the value of each outcome, and an actor neural network chooses actions to reach the best outcomes. Although intuitively appealing, robustly learning and leveraging world models to achieve strong task performance has been an open problem[¹⁷](#page-12-2). Dreamer overcomes this challenge through a range of robustness techniques based on normalization, balancing, and transformations. We observe robust learning not only across over 150 tasks from the domains summarized in [Figure 2,](#page-1-0) but also across model sizes and training budgets, offering a predictable way to increase performance. Notably, larger model sizes not only achieve higher scores but also require less interaction to solve a task.

To push the boundaries of reinforcement learning, we consider the popular video game Minecraft that has become a focal point of research in recent years [18](#page-12-3)[,19](#page-12-4)[,20](#page-12-5), with international competitions held for developing algorithms that autonomously learn to collect diamonds in Minecraft[\\*](#page-1-1). Solving this

<span id="page-1-1"></span>^\*The MineRL Diamond Competitions were held in 2019, 2020, and 2021 and provided a dataset of human expert trajectories: https://minerl.io/diamond. Competitions in the following years focused on a wide range of tasks.

<span id="page-2-0"></span>

Figure 3: Training process of Dreamer. The world model encodes sensory inputs into discrete representations z^t that are predicted by a sequence model with recurrent state h^t given actions a^t . The inputs are reconstructed to shape the representations. The actor and critic predict actions a^t and values v^t and learn from trajectories of abstract representations predicted by the world model.

problem without human data has been widely recognized as a substantial challenge for artificial intelligence because of the sparse rewards, exploration difficulty, long time horizons, and the procedural diversity of this open world game [¹⁸](#page-12-3). Due to these obstacles, previous approaches resorted to using human expert data and domain-specific curricula[19,](#page-12-4)[20](#page-12-5). Applied out of the box, Dreamer is the first algorithm to collect diamonds in Minecraft from scratch.

Learning algorithm

We present the third generation of the Dreamer algorithm[21,](#page-12-6)[22](#page-12-7). The algorithm consists of three neural networks: the world model predicts the outcomes of potential actions, the critic judges the value of each outcome, and the actor chooses actions to reach the most valuable outcomes. The components are trained concurrently from replayed experience while the agent interacts with the environment. To succeed across domains, all three components need to accommodate different signal magnitudes and robustly balance terms in their objectives. This is challenging as we are not only targeting similar tasks within the same domain but aim to learn across diverse domains with fixed hyperparameters. This section introduces the world model, critic, and actor along with their robust loss functions, as well as tools for robustly predicting quantities of unknown orders of magnitude.

World model learning

The world model learns compact representations of sensory inputs through autoencoding[²³](#page-12-8) and enables planning by predicting future representations and rewards for potential actions. We implement the world model as a Recurrent State-Space Model (RSSM) [²⁴](#page-12-9), shown in [Figure 3.](#page-2-0) First, an encoder maps sensory inputs x^t to stochastic representations z^t . Then, a sequence model with recurrent state h^t predicts the sequence of these representations given past actions at−1. The concatenation of

<span id="page-3-0"></span>

Figure 4: Multi-step video predictions of a DMLab maze (top) and a quadrupedal robot (bottom). Given 5 context images and the full action sequence, the model predicts 45 frames into the future without access to intermediate images. The world model learns an understanding of the underlying structure of each environment.

h^t and z^t forms the model state from which we predict rewards r^t and episode continuation flags c^t ∈ {0, 1} and reconstruct the inputs to ensure informative representations:

[Figure 4](#page-3-0) visualizes long-term video predictions of the world world. The encoder and decoder use convolutional neural networks (CNN) for image inputs and multi-layer perceptrons (MLPs) for vector inputs. The dynamics, reward, and continue predictors are also MLPs. The representations are sampled from a vector of softmax distributions and we take straight-through gradients through the sampling step[25,](#page-12-10)[22](#page-12-7). Given a sequence batch of inputs x1:^T , actions a1:^T , rewards r1:^T , and continuation flags c1:^T , the world model parameters ϕ are optimized end-to-end to minimize the prediction loss Lpred, the dynamics loss Ldyn, and the representation loss Lrep with corresponding loss weights βpred = 1, βdyn = 1, and βrep = 0.1:

$$\mathcal{L}(\phi) \doteq E_{q_\phi} \left[ \sum_{t=1}^T (\beta_{\text{pred}} \mathcal{L}_{\text{pred}}(\phi) + \beta_{\text{dyn}} \mathcal{L}_{\text{dyn}}(\phi) + \beta_{\text{rep}} \mathcal{L}_{\text{rep}}(\phi)) \right]. \quad (2)$$

The prediction loss trains the decoder and reward predictor via the symlog squared loss described later, and the continue predictor via logistic regression. The dynamics loss trains the sequence model to predict the next representation by minimizing the KL divergence between the predictor pϕ(z^t | ht) and the next stochastic representation qϕ(z^t | h^t , xt). The representation loss, in turn, trains the representations to become more predictable allowing us to use a factorized dynamics predictor for fast sampling during imagination training. The two losses differ in the stop-gradient operator sg(·) and their loss scale. To avoid a degenerate solution where the dynamics are trivial to predict but fail

to contain enough information about the inputs, we employ free bits [²⁶](#page-12-11) by clipping the dynamics and representation losses below the value of 1 nat ≈ 1.44 bits. This disables them while they are already minimized well to focus learning on the prediction loss:

$$\begin{aligned}\mathcal{L}_{\text{pred}}(\phi) &= -\ln p_\phi(z_t \mid z_t, h_t) - \ln p_\phi(r_t \mid z_t, h_t) - \ln p_\phi(c_t \mid z_t, h_t) \\ \mathcal{L}_{\text{dyn}}(\phi) &= \max(1, \text{KL}[\text{sg}(q_\phi(z_t \mid h_t, x_t)) \parallel p_\phi(z_t \mid h_t)]) \\ \mathcal{L}_{\text{rep}}(\phi) &= \max(1, \text{KL}[q_\phi(z_t \mid h_t, x_t) \parallel \text{sg}(p_\phi(z_t \mid h_t))])\end{aligned}\tag{3}$$

Previous world models require scaling the representation loss differently based on the visual complexity of the environment[²¹](#page-12-6). Complex 3D environments contain details unnecessary for control and thus prompt a stronger regularizer to simplify the representations and make them more predictable. In games with static backgrounds and where individual pixels may matter for the task, a weak regularizer is required to extract fine details. We find that combining free bits with a small representation loss resolves this dilemma, allowing for fixed hyperparameters across domains. Moreover, we transform vector observations using the symlog function described later, to prevent large inputs and large reconstruction gradients, further stabilizing the trade-off with the representation loss.

We occasionally observed spikes the in KL losses in earlier experiments, consistent with reports for deep variational autoencoders [²⁷](#page-12-12). To prevent this, we parameterize the categorical distributions of the encoder and dynamics predictor as mixtures of 1% uniform and 99% neural network output, making it impossible for them to become deterministic and thus ensuring well-behaved KL losses. Further model details and hyperparameters are included in the supplementary material.

Critic learning

The actor and critic neural networks learn behaviors purely from abstract trajectories of representations predicted by the world model[¹⁴](#page-11-13). For environment interaction, we select actions by sampling from the actor network without lookahead planning. The actor and critic operate on model states st .⁼ {h^t , zt} and thus benefit from the Markovian representations learned by the recurrent world model. The actor aims to maximize the return R^t .= P^∞ ^τ=0 γ τ rt+^τ with a discount factor γ = 0.997 for each model state. To consider rewards beyond the prediction horizon T = 16, the critic learns to approximate the distribution of returns[²⁸](#page-12-13) for each state under the current actor behavior:

Starting from representations of replayed inputs, the world model and actor generate a trajectory of imagined model states s1:^T , actions a1:^T , rewards r1:^T , and continuation flags c1:^T . Because the critic predicts a distribution, we read out its predicted values v^t .= E[vψ(· | ^st)] as the expectation of the distribution. To estimate returns that consider rewards beyond the prediction horizon, we compute bootstrapped λ-returns[²⁹](#page-12-14) that integrate the predicted rewards and the values. The critic learns to predict the distribution of the return estimates R^λ ^t using the maximum likelihood loss:

$$\mathcal{L}(\psi) \doteq - \sum_{t=1}^T \ln p_\psi(R_t^\lambda \mid s_t) \quad R_t^\lambda \doteq r_t + \gamma c_t \left( (1-\lambda)v_t + \lambda R_{t+1}^\lambda \right) \quad R_T^\lambda \doteq v_T \quad (5)$$

While a simple choice would be to parameterize the critic as a Normal distribution, the return distribution can have multiple modes and vary by orders of magnitude across environments. To stabilize and accelerate learning under these conditions, we parameterize the critic as categorical distribution with exponentially spaced bins, decoupling the scale of gradients from the prediction targets as described later. To improve value prediction in environments where rewards are challenging to predict, we apply the critic loss both to imagined trajectories with loss scale βval = 1 and to trajectories sampled from the replay buffer with loss scale βrepval = 0.3. The critic replay loss uses the imagination returns R^λ t at the start states of the imagination rollouts as on-policy value annotations for the replay trajectory to then compute λ-returns over the replay rewards.

Because the critic regresses targets that depend on its own predictions, we stabilize learning by regularizing the critic towards predicting the outputs of an exponentially moving average of its own parameters. This is similar to target networks used previously in reinforcement learning[⁷](#page-11-6) but allows us to compute returns using the current critic network. We further noticed that the randomly initialized reward predictor and critic networks at the start of training can result in large predicted rewards that can delay the onset of learning. We thus initialize the output weight matrix of the reward predictor and critic to zeros, which alleviates the problem and accelerates early learning.

Actor learning

The actor learns to choose actions that maximize return while exploring through an entropy regularizer[³⁰](#page-12-15). However, the correct scale for this regularizer depends both on the scale and frequency of rewards in the environment. Ideally, we would like the agent to explore more if rewards are sparse and exploit more if rewards are dense or nearby. At the same time, the exploration amount should not be influenced by arbitrary scaling of rewards in the environment. This requires normalizing the return scale while preserving information about reward frequency.

To use a fixed entropy scale of η = 3 × 10−⁴ across domains, we normalize returns to be approximately contained in the interval [0, 1]. In practice, substracting an offset from the returns does not change the actor gradient and thus dividing by the range S is sufficient. Moreover, to avoid amplifying noise from function approximation under sparse rewards, we only scale down large return magnitudes but leave small returns below the threshold of L = 1 untouched. We use the Reinforce estimator[³¹](#page-13-0) for both discrete and continuous actions, resulting in the surrogate loss function:

$$\mathcal{L}(\theta) \doteq - \sum_{t=1}^T \text{sg}\left(\left(R_t^\lambda - v_\psi(s_t)\right)/\max(1, S)\right) \log \pi_\theta(a_t | s_t) + \eta \text{H}\left[\pi_\theta(a_t | s_t)\right] \quad (6)$$

The return distribution can be multi-modal and include outliers, especially for randomized environments where some episodes have higher achievable returns than others. Normalizing by the smallest and largest observed returns would then scale returns down too much and may cause suboptimal convergence. To be robust to these outliers, we compute the range from the 5th to the 95th return percentile over the return batch and smooth out the estimate using an exponential moving average:

$$S \doteq \text{EMA}(\text{Per}(R_t^\lambda, 95) - \text{Per}(R_t^\lambda, 5), 0.99) \quad (7)$$

Previous work typically normalizes advantages[⁵](#page-11-4) rather than returns, which puts a fixed amount of emphasis on maximizing returns over entropy regardless of whether rewards are within reach. Scaling up advantages when rewards are sparse can amplify noise that outweighs the entropy regularizer and stagnates exploration. Normalizing rewards or returns by standard deviation can fail under sparse rewards where their standard deviation is near zero, drastically amplifying rewards regardless of their size. Constrained optimization targets a fixed entropy on average across states [32,](#page-13-1)[33](#page-13-2) regardless of achievable returns, which is robust but explores slowly under sparse rewards and converges lower under dense rewards. We did not find stable hyperparameters across domains for these approaches. Return normalization with a denominator limit overcomes these challenges, exploring rapidly under sparse rewards and converging to high performance across diverse domains.

Robust predictions

Reconstructing inputs and predicting rewards and returns can be challenging because the scale of these quantities can vary across domains. Predicting large targets using a squared loss can lead to divergence whereas absolute and Huber losses[⁷](#page-11-6) stagnate learning. On the other hand, normalizing targets based on running statistics[⁵](#page-11-4) introduces non-stationarity into the optimization. We suggest the symlog squared error as a simple solution to this dilemma. For this, a neural network f(x, θ) with inputs x and parameters θ learns to predict a transformed version of its targets y. To read out predictions yˆ of the network, we apply the inverse transformation:

$$\mathcal{L}(\theta) \doteq \frac{1}{2}(f(x, \theta) - \text{symlog}(y))^2 \quad \hat{y} \doteq \text{symexp}(f(x, \theta)) \quad (8)$$

Using the logarithm as transformation would not allow us to predict targets that take on negative values. Therefore, we choose a function from the bi-symmetric logarithmic family[³⁴](#page-13-3) that we name symlog as the transformation with the symexp function as its inverse:

The symlog function compresses the magnitudes of both large positive and negative values. Unlike the logarithm, it is symmetric around the origin while preserving the input sign. This allows the optimization process to quickly move the network predictions to large values when needed. The symlog function approximates the identity around the origin so that it does not affect learning of targets that are already small enough.

For potentially stochastic targets, such as rewards or returns, we introduce the symexp twohot loss. Here, the network outputs the logits for a softmax distribution over exponentially spaced bins bⁱ ∈ B. Predictions are read out as the weighted average of the bin positions weighted by their predicted probabilities. Importantly, the network can output any continuous value in the interval because the weighted average can fall between the buckets:

The network is trained on twohot encoded targets[8](#page-11-7)[,28](#page-12-13), a generalization of onehot encoding to continuous values. The twohot encoding of a scalar is a vector with |B| entries that are all 0 except at the indices k and k + 1 of the two bins closest to the encoded scalar. The two entries sum up to 1, with linearly higher weight given to the bin that is closer to the encoded continuous number. The network is then trained to minimize the categorical cross entropy loss for classification with soft targets. Note that the loss only depends on the probabilities assigned to the bins but not on the continuous values associated with the bin locations, decoupling the size of the gradients from the size of the targets:

$$\mathcal{L}(\theta) \doteq - \text{twohot}(y)^T \log \text{softmax}(f(x, \theta)) \quad (11)$$

Applying these principles, Dreamer transforms vector observations using the symlog functions, both for the encoder inputs and the decoder targets and employs the synexp twohot loss for the reward predictor and critic. We find that these techniques enable robust and fast learning across many diverse domains. For critic learning, an alternative asymmetric transformation has previously been proposed[³⁵](#page-13-4), which we found less effective on average across domains. Unlike alternatives, symlog transformations avoid truncating large targets [⁷](#page-11-6) , introducing non-stationary from normalization[⁵](#page-11-4) , or adjusting network weights when new extreme values are detected[³⁶](#page-13-5) .

<span id="page-7-0"></span>

Figure 5: Fraction of trained agents that discover each of the three latest items in the Minecraft Diamond task. Although previous algorithms progress up to the iron pickaxe, Dreamer is the only compared algorithm that manages to discover a diamond, and does so reliably.

Results

We evaluate the generality of Dreamer across 8 domains—with over 150 tasks—under fixed hyperparameters. We designed the experiments to compare Dreamer to the best methods in the literature, which are often specifically designed and tuned for the benchmark at hand. We further compare to a high-quality implementation of PPO[⁵](#page-11-4) , a standard reinforcement learning algorithm that is known for its robustness. We run PPO with fixed hyperparameters chosen to maximize performance across domains and that reproduce strong published results of PPO on ProcGen[³⁷](#page-13-6). To push the boundaries of reinforcement learning, we apply Dreamer to the challenging video game Minecraft, comparing it to strong previous algorithms. Finally, we analyze the importance of individual components of Dreamer and its robustness to different model sizes and computational budgets. All Dreamer agents are trained on a single Nvidia A100 GPU each, making it reproducible for many research labs. A public implementation of Dreamer that reproduces all results is available on the project website.

Benchmarks We perform an extensive empirical study across 8 domains that include continuous and discrete actions, visual and low-dimensional inputs, dense and sparse rewards, different reward scales, 2D and 3D worlds, and procedural generation. [Figure 1](#page-0-0) summarizes the benchmark results, showing that Dreamer outperforms a wide range of previous expert algorithms across diverse domains. Crucially, Dreamer substantially outperforms PPO across all domains.

• Atari This established benchmark contains 57 Atari 2600 games with a budget of 200M frames, posing a diverse range of challenges[³⁸](#page-13-7). We use the sticky action simulator setting[³⁹](#page-13-8). Dreamer outperforms the powerful MuZero algorithm[⁸](#page-11-7) while using only a fraction of the computational resources. Dreamer also outperforms the widely-used expert algorithms Rainbow[⁴⁰](#page-13-9) and IQN[⁴¹](#page-13-10) .
• ProcGen This benchmark of 16 games features randomized levels and visual distractions to test the robustness and generalization of agents [⁴²](#page-13-11). Within the budget of 50M frames, Dreamer matches the tuned expert algorithm PPG[³⁷](#page-13-6) and outperforms Rainbow[42,](#page-13-11)[40](#page-13-9). Our PPO agent with fixed hyperparameters matches the published score of the highly tuned official PPO implementation[³⁷](#page-13-6) .
• DMLab This suite of 30 tasks features 3D environments that test spatial and temporal reasoning[⁴³](#page-13-12). In 100M frames, Dreamer exceeds the performance of the scalable IMPALA and R2D2+ agents[³⁵](#page-13-4) at 1B environment steps, amounting to a data-efficiency gain of over 1000%. We note that these baselines were not designed for data-efficiency but serve as a valuable comparison point for the performance previously achievable at scale.

<span id="page-8-0"></span>

Figure 6: Ablations and robust scaling of Dreamer. a, All individual robustness techniques contribute to the performance of Dreamer on average, although each individual technique may only affect some tasks. Training curves of individual tasks are included in the supplementary material. b, The performance of Dreamer predominantly rests on the unsupervised reconstruction loss of its world model, unlike most prior algorithms that rely predominantly on reward and value prediction gradients[7,](#page-11-6)[5,](#page-11-4)[8](#page-11-7) . c, The performance of Dreamer increases monotonically with larger model sizes, ranging from 12M to 400M parameters. Notably, larger models not only increase task performance but also require less environment interaction. d, Higher replay ratios predictably increase the performance of Dreamer. Together with model size, this allows practitioners to improve task performance and data-efficiency by employing more computational resources.

• Atari100k This data-efficiency benchmark comntains 26 Atari games and a budget of only 400K frames, amounting to 2 hours of game time[¹⁷](#page-12-2). EfficientZero[⁴⁴](#page-13-13) holds the state-of-the-art by combining online tree search, prioritized replay, and hyperparameter scheduling, but also resets levels early to increase data diversity, making a comparison difficult. Without this complexity, Dreamer outperforms the best remaining methods, including the transformer-based IRIS and TWM agents, the model-free SPR, and SimPLe[⁴⁵](#page-13-14) .
• Proprio Control This benchmark contains 18 control tasks with continuous actions, proprioceptive vector inputs, and a budget of 500K environment steps[⁴⁶](#page-14-0). The tasks range from classical control over locomotion to robot manipulation tasks, featuring dense and sparse rewards. Dreamer sets a new state-of-the-art on this benchmark, outperforming D4PG, DMPO, and MPO[³³](#page-13-2) .
• Visual Control This benchmark consists of 20 continuous control tasks where the agent receives only high-dimensional images as input and has a budget of 1M environment steps[⁴⁶](#page-14-0). Dreamer establishes a new state-of-the-art on this benchmark, outperforming DrQ-v2 and CURL[⁴⁷](#page-14-1), which are specialized to visual environments and leverage data augmentation.

• BSuite This benchmark includes 23 environments with a total of 468 configurations that are specifically designed to test credit assignment, robustness to reward scale and stochasticity, memory, generalization, and exploration[⁴⁸](#page-14-2). Dreamer establishes a new state-of-the-art on this benchmark, outperforming Boot DQN and other methods[⁴⁹](#page-14-3). Dreamer improves over previous algorithms especially in the scale robustness category.

Minecraft Collecting diamonds in the popular game Minecraft has been a long-standing challenge in artificial intelligence[18](#page-12-3)[,19](#page-12-4)[,20](#page-12-5). Every episode in this game is set in a unique randomly generated and infinite 3D world. Episodes last until the player dies or up to 36000 steps equaling 30 minutes, during which the player needs to discover a sequence of 12 items from sparse rewards by foraging for resources and crafting tools. It takes about 20 minutes for experienced human players to obtain diamonds [²⁰](#page-12-5). We follow the block breaking setting of prior work[¹⁹](#page-12-4) because the provided action space would make it challenging for stochastic policies to keep a key pressed for a prolonged time.

Because of the training time in this complex domain, extensive tuning would be difficult for Minecraft. Instead, we apply Dreamer out of the box with its default hyperparameters. As shown in [Figures 1](#page-0-0) and [5,](#page-7-0) Dreamer is the first algorithm to collect diamonds in Minecraft from scratch without using human data as was required by VPT[²⁰](#page-12-5) or adaptive curricula[¹⁹](#page-12-4). All the Dreamer agents we trained on Minecraft discover diamonds in 100M environment steps. While several strong baselines progress to advanced items such as the iron pickaxe, none of them discovers a diamond.

Ablations In [Figure 6,](#page-8-0) we ablate the robustness techniques and learning signals on a diverse set of 14 tasks to understand their importance. The training curves of individual tasks are included in the supplementary material. We observe that all robustness techniques contribute to performance, most notably the KL objective of the world model, followed by return normalization and symexp twohot regression for reward and value prediction. In general, we find that each individual technique is critical on a subset of tasks but may not affect performance on other tasks.

To investigate the effect of the world model, we ablate the learning signals of Dreamer by stopping either the task-specific reward and value prediction gradients or the task-agnostic reconstruction gradients from shaping its representations. Unlike previous reinforcement learning algorithms that often rely only on task-specific learning signals [7,](#page-11-6)[8](#page-11-7) , Dreamer rests predominantly on the unsupervised objective of its world model. This finding could allow for future algorithm variants that leverage pretraining on unsupervised data.

Scaling properties To investigate whether Dreamer can scale robustly, we train 6 model sizes ranging from 12M to 400M parameters, as well as different replay ratios on Crafter [⁵⁰](#page-14-4) and a DMLab task[⁴³](#page-13-12). The replay ratio affects the number of gradient updates performed by the agent. [Figure 6](#page-8-0) shows robust learning with fixed hyperparameters across the compared model sizes and replay ratios. Moreover, increasing the model size directly translates to both higher task performance and a lower data requirement. Increasing the number of gradient steps further reduces the interactions needed to learn successful behaviors. The results show that Dreamer learns robustly across model sizes and replay ratios and that its performance and provides a predictable way for increasing performance given computational resources.

Developing general-purpose algorithms has long been a goal of reinforcement learning research. PPO[⁵](#page-11-4) is one of the most widely used algorithms and is relatively robust but requires large amounts of experience and often yields lower performance than specialized alternatives. SAC[³²](#page-13-1) is a popular choice for continuous control and leverages experience replay for data-efficiency, but in practice requires tuning, especially for its entropy scale, and struggles under high-dimensional inputs[⁵¹](#page-14-5) . MuZero[⁸](#page-11-7) plans using a value prediction model and has been applied to board games and Atari, but the authors did not release an implementation and the algorithm contains several complex components, making it challenging to reproduce. Gato[⁵²](#page-14-6) fits one large model to expert demonstrations of multiple tasks, but is only applicable when expert data is available. In comparison, Dreamer masters a diverse range of environments with fixed hyperparameters, does not require expert data, and its implementation is open source.

Minecraft has been a focus of recent research. With MALMO[⁵³](#page-14-7), Microsoft released a free version of the successful game for research purposes. MineRL[¹⁸](#page-12-3) offers several competition environments, which we rely on as the basis for our experiments. The MineRL competition supports agents in exploring and learning meaningful skills through a diverse human dataset [¹⁸](#page-12-3). Voyager obtains items at a similar depth in the technology tree as Dreamer using API calls to a language model but operates on top of the MineFlayer bot scripting layer that was specifically engineered to the game and exposes high-level actions[⁵⁴](#page-14-8). VPT[²⁰](#page-12-5) trained an agent to play Minecraft through behavioral cloning based on expert data of keyboard and mouse actions collected by contractors and finetuning using reinforcement learning to obtain diamonds using 720 GPUs for 9 days. In comparison, Dreamer uses the MineRL competition action space to autonomously learn to collect diamonds from sparse rewards using 1 GPU for 9 days, without human data.

We present the third generation of the Dreamer algorithm, a general reinforcement learning algorithm that masters a wide range of domains with fixed hyperparameters. Dreamer excels not only across over 150 tasks but also learns robustly across varying data and compute budgets, moving reinforcement learning toward a wide range of practical applications. Applied out of the box, Dreamer is the first algorithm to collect diamonds in Minecraft from scratch, achieving a significant milestone in the field of artificial intelligence. As a high-performing algorithm that is based on a learned world model, Dreamer paves the way for future research directions, including teaching agents world knowledge from internet videos and learning a single world model across domains to allow artificial agents to build up increasingly general knowledge and competency.

Acknowledgements We thank Mohammad Norouzi, Jessy Lin, Abbas Abdolmaleki, John Schulman, Adam Kosiorek, and Oleh Rybkin for insightful discussions. We thank Bobak Shahriari, Denis Yarats, Karl Cobbe, and Hubert Soyer for sharing training curves of baseline algorithms. We thank Daniel Furrer, Andrew Chen, and Dakshesh Garambha for help with Google Cloud infrastructure.

References

• <span id="page-11-0"></span>1. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587): 484, 2016.
• <span id="page-11-1"></span>2. OpenAI. OpenAI Five. https://blog.openai.com/openai-five/, 2018.
• <span id="page-11-2"></span>3. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022.
• <span id="page-11-3"></span>4. Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314–21328, 2022.
• <span id="page-11-4"></span>5. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
• <span id="page-11-5"></span>6. Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.
• <span id="page-11-6"></span>7. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
• <span id="page-11-7"></span>8. Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. arXiv preprint arXiv:1911.08265, 2019.
• <span id="page-11-8"></span>9. Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397, 2016.
• <span id="page-11-9"></span>10. Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Côté, and R Devon Hjelm. Unsupervised state representation learning in atari. Advances in neural information processing systems, 32, 2019.
• <span id="page-11-10"></span>11. Danny Driess, Ingmar Schubert, Pete Florence, Yunzhu Li, and Marc Toussaint. Reinforcement learning with neural radiance fields. arXiv preprint arXiv:2206.01634, 2022.
• <span id="page-11-11"></span>12. David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017.
• <span id="page-11-12"></span>13. Marcin Andrychowicz, Anton Raichuk, Piotr Stanczyk, Manu Orsini, Sertan Girgin, Raphael ´ Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et al. What matters in on-policy reinforcement learning? a large-scale empirical study. arXiv preprint arXiv:2006.05990, 2020.
• <span id="page-11-13"></span>14. Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM SIGART Bulletin, 2(4):160–163, 1991.

• <span id="page-12-0"></span>15. Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 2786–2793. IEEE, 2017.
• <span id="page-12-1"></span>16. David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018.
• <span id="page-12-2"></span>17. Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Modelbased reinforcement learning for atari. arXiv preprint arXiv:1903.00374, 2019.
• <span id="page-12-3"></span>18. William H Guss, Cayden Codel, Katja Hofmann, Brandon Houghton, Noboru Kuno, Stephanie Milani, Sharada Mohanty, Diego Perez Liebana, Ruslan Salakhutdinov, Nicholay Topin, et al. The minerl competition on sample efficient reinforcement learning using human priors. arXiv e-prints, pages arXiv–1904, 2019.
• <span id="page-12-4"></span>19. Ingmar Kanitscheider, Joost Huizinga, David Farhi, William Hebgen Guss, Brandon Houghton, Raul Sampedro, Peter Zhokhov, Bowen Baker, Adrien Ecoffet, Jie Tang, et al. Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft. arXiv preprint arXiv:2106.14876, 2021.
• <span id="page-12-5"></span>20. Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. arXiv preprint arXiv:2206.11795, 2022.
• <span id="page-12-6"></span>21. Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603, 2019.
• <span id="page-12-7"></span>22. Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020.
• <span id="page-12-8"></span>23. Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
• <span id="page-12-9"></span>24. Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.04551, 2018.
• <span id="page-12-10"></span>25. Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.
• <span id="page-12-11"></span>26. Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29, 2016.
• <span id="page-12-12"></span>27. Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. arXiv preprint arXiv:2011.10650, 2020.
• <span id="page-12-13"></span>28. Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning, pages 449–458. PMLR, 2017.
• <span id="page-12-14"></span>29. Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
• <span id="page-12-15"></span>30. Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241–268, 1991.

• <span id="page-13-0"></span>31. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256, 1992.
• <span id="page-13-1"></span>32. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290, 2018.
• <span id="page-13-2"></span>33. Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint arXiv:1806.06920, 2018.
• <span id="page-13-3"></span>34. J Beau W Webber. A bi-symmetric log transformation for wide-range data. Measurement Science and Technology, 24(2):027001, 2012.
• <span id="page-13-4"></span>35. Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018.
• <span id="page-13-5"></span>36. Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojciech Czarnecki, Simon Schmitt, and Hado van Hasselt. Multi-task deep reinforcement learning with popart. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3796–3803, 2019.
• <span id="page-13-6"></span>37. Karl W Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In International Conference on Machine Learning, pages 2020–2027. PMLR, 2021.
• <span id="page-13-7"></span>38. Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, 2013.
• <span id="page-13-8"></span>39. Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523–562, 2018.
• <span id="page-13-9"></span>40. Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
• <span id="page-13-10"></span>41. Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning, pages 1096–1105. PMLR, 2018.
• <span id="page-13-11"></span>42. Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pages 2048–2056. PMLR, 2020.
• <span id="page-13-12"></span>43. Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint arXiv:1612.03801, 2016.
• <span id="page-13-13"></span>44. Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances in Neural Information Processing Systems, 34:25476–25488, 2021.
• <span id="page-13-14"></span>45. Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample efficient world models. arXiv preprint arXiv:2209.00588, 2022.

• <span id="page-14-0"></span>46. Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018.
• <span id="page-14-1"></span>47. Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021.
• <span id="page-14-2"></span>48. Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari, Satinder Singh, et al. Behaviour suite for reinforcement learning. arXiv preprint arXiv:1908.03568, 2019.
• <span id="page-14-3"></span>49. Olivia Dizon-Paradis, Stephen Wormald, Daniel Capecci, Avanti Bhandarkar, and Damon Woodard. Investigating the practicality of existing reinforcement learning algorithms: A performance comparison. Authorea Preprints, 2023.
• <span id="page-14-4"></span>50. Danijar Hafner. Benchmarking the spectrum of agent capabilities. arXiv preprint arXiv:2109.06780, 2021.
• <span id="page-14-5"></span>51. Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. arXiv preprint arXiv:1910.01741, 2019.
• <span id="page-14-6"></span>52. Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175, 2022.
• <span id="page-14-7"></span>53. Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artificial intelligence experimentation. In IJCAI, pages 4246–4247. Citeseer, 2016.
• <span id="page-14-8"></span>54. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023.
• <span id="page-14-9"></span>55. Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 implementation details of proximal policy optimization. The ICLR Blog Track 2023, 2022.
• <span id="page-14-10"></span>56. Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, et al. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.00979, 2020.
• <span id="page-14-11"></span>57. Simon Schmitt, Matteo Hessel, and Karen Simonyan. Off-policy actor-critic with shared experience replay. In International Conference on Machine Learning, pages 8545–8554. PMLR, 2020.
• <span id="page-14-12"></span>58. Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015.
• <span id="page-14-13"></span>59. Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. In International Conference on Machine Learning, pages 1059–1071. PMLR, 2021.
• <span id="page-14-14"></span>60. Liu Ziyin, Zhikang T Wang, and Masahito Ueda. Laprop: Separating momentum and adaptivity in adam. arXiv preprint arXiv:2002.04839, 2020.

• <span id="page-15-0"></span>61. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
• <span id="page-15-1"></span>62. Audrunas Gruslys, Will Dabney, Mohammad Gheshlaghi Azar, Bilal Piot, Marc Bellemare, and Remi Munos. The reactor: A fast and sample-efficient actor-critic agent for reinforcement learning. arXiv preprint arXiv:1704.04651, 2017.
• <span id="page-15-2"></span>63. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoderdecoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.
• <span id="page-15-3"></span>64. Matthijs Van Keirsbilck, Alexander Keller, and Xiaodong Yang. Rethinking full connectivity in recurrent neural networks. arXiv preprint arXiv:1905.12340, 2019.
• <span id="page-15-4"></span>65. Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523–562, 2018.
• <span id="page-15-5"></span>66. Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561, 2018.

Baselines

We employ the Proximal Policy Optimization (PPO) algorithm[⁵](#page-11-4) , which has become a standard choice in the field, to compare Dreamer under fixed hyperparameters across all benchmarks. There are a large number of PPO implementations available publicly and they are known to substantially vary in task performance [⁵⁵](#page-14-9). To ensure a comparison that is representative of the highest performance PPO can achieve under fixed hyperparameters across domains, we choose the high-quality PPO implementation available in the Acme framework[⁵⁶](#page-14-10) and select its hyperparameters in [Table 1](#page-16-0) following recommendations[55,](#page-14-9)[13](#page-11-12) and additionally tune its epoch batch size to be large enough for complex environments[⁴²](#page-13-11), its learning rate, and its entropy scale. We match the discount factor to Dreamer because it works well across domains and is a common choice in the literature[35,](#page-13-4)[8](#page-11-7) . We choose the IMPALA network architecture that we have found performed better than alternatives[⁴²](#page-13-11) and set the minibatch size to the largest possible for one A100 GPU. We verify the performance of our PPO implementation and hyperparameters on the ProcGen benchmark, where a highly tuned PPO implementation has been reported by the PPO authors[³⁷](#page-13-6). We find that our implementation matches or slightly outperforms this performance reference.

<span id="page-16-0"></span>

Table 1: PPO hyperparameters used across all benchmarks.

For Minecraft, we additionally tune and run the IMPALA and Rainbow algorithms because not successful end-to-end learning from scratch has been reported in the literature [¹⁸](#page-12-3). We use the Acme implementations [⁵⁶](#page-14-10) of these algorithms, use the same IMPALA network we used for PPO, and tuned the learning rate and entropy regularizers. For all other benchmarks, we compare to tuned expert algorithms reported in the literature as referenced in the results section.

Implementation

Experience replay We implement Dreamer using a uniform replay buffer with an online queue[⁵⁷](#page-14-11) . Specifically, each minibatch is formed first from non-overlapping online trajectories and then filled up with uniformly sampled trajectories from the replay buffer. We store latent states into the replay buffer during data collection to initialize the world model on replayed trajectories, and write the fresh latent states of the training rollout back into the buffer. While prioritized replay[⁵⁸](#page-14-12) is used by some of the expert algorithms we compare to and we found it to also improve the performance of Dreamer, we opt for uniform replay in our experiments for ease of implementation.

We parameterize the amount of training via the replay ratio. This is the fraction of time steps trained on per time step collected from the environment, without action repeat. Dividing the replay ratio by the time steps in a minibatch and by action repeat yields the ratio of gradient steps to env steps. For example, a replay ratio of 32 on Atari with action repeat of 4 and batch shape 16 × 64 corresponds to 1 gradient step every 128 env steps, or 1.5M gradient steps over 200M env steps.

Optimizer We employ Adaptive Gradient Clipping (AGC)[⁵⁹](#page-14-13), which clips per-tensor gradients if they exceed 30% of the L2 norm of the weight matrix they correspond to, with its default ϵ = 10−³ . AGC decouples the clipping threshold from the loss scales, allowing to change loss functions or loss scales without adjusting the clipping threshold. We apply the clipped gradients using the LaProp optimizer[⁶⁰](#page-14-14) with ϵ = 10−²⁰ and its default parameters β¹ = 0.9 and β² = 0.99. LaProp normalizes gradients by RMSProp and then smoothes them by momentum, instead of computing both momentum and normalizer on raw gradients as Adam does [⁶¹](#page-15-0). This simple change allows for a smaller epsilon and avoids occasional instabilities that we observed under Adam.

Distributions The encoder, dynamics predictor, and actor distributions are mixtures of 99% the predicted softmax output and 1% of a uniform distribution[⁶²](#page-15-1) to prevent zero probabilities and infinite log probabilities. The rewards and critic neural networks output a softmax distribution over exponentially spaced bins b ∈ B and are trained towards twohot encoded targets:

$$\text{twohot}(x)_i \doteq \begin{cases} |b_{k+1} - x| / |b_{k+1} - b_k| & \text{if } i = k \\ |b_k - x| / |b_{k+1} - b_k| & \text{if } i = k + 1 \\ 0 & \text{else} \end{cases} \quad k \doteq \sum_{j=1}^{|B|} \delta(b_j < x) \quad (12)$$

The output weights of twohot distributions are initialized to zero to ensure that the agent does not hallucinate rewards and values at initialization. For computing the expected prediction of the softmax distribution under bins that span many orders of magnitude, the summation order matters and positive and negative bins should be summed up separately, from small to large bins, and then added. Refer to the source code for an implementation.

Networks Images are encoded using stride 2 convolutions to resolution 6 × 6 or 4 × 4 and then flattened and are decoded using transposed stride 2 convolutions, with sigmoid activation on the output. Vector inputs are symlog transformed and then encoded and decoded using 3-layer MLPs. The actor and critic neural networks are also 3-layer MLPs and the reward and continue predictors are 1-layer MLPs. The sequence model is a GRU[⁶³](#page-15-2) with block-diagonal recurrent weights[⁶⁴](#page-15-3) of 8 blocks to allow for a large number of memory units without quadratic increase in parameters and FLOPs. The input to the GRU at each time step is a linear embedding of the sampled latent z^t , of the action a^t , and of the recurrent state to allow mixing between blocks.

Benchmarks

Protocols Summarized in [Table 2,](#page-18-0) we follow the standard evaluation protocols for the benchmarks where established. Atari[³⁸](#page-13-7) uses 57 tasks with sticky actions[⁶⁵](#page-15-4). The random and human reference scores used to normalize scores vary across the literature and we chose the most common reference values, replicated in [Table 6.](#page-25-0) DMLab[⁴³](#page-13-12) uses 30 tasks [⁶⁶](#page-15-5) and we use the fixed action space[36](#page-13-5)[,35](#page-13-4). We evaluate at 100M steps because running for 10B as in some prior work was infeasible. Because existing published baselines perform poorly at 100M steps, we compare to their performance at 1B steps instead, giving them a 10× data advantage. ProcGen uses the hard difficulty setting and the unlimited level set[⁴²](#page-13-11). Prior work compares at different step budgets[42](#page-13-11)[,37](#page-13-6) and we compare at 50M steps due to computational cost, as there is no action repeat. For Minecraft Diamond purely from sparse rewards, we establish the evaluation protocol to report the episode return measured at 100M env steps, corresponding to about 100 days of in-game time. Atari100k[¹⁷](#page-12-2) includes 26 tasks with a budget of 400K env steps, 100K after action repeat. Prior work has used various environment settings, summarized in [Table 10,](#page-31-0) and we chose the environments as originally introduced. Visual Control[46](#page-14-0)[,21](#page-12-6) spans 20 tasks with an action repeat of 2. Proprioceptive Control follows the same protocol but we exclude the two quadruped tasks because of baseline availability in prior work[⁴⁷](#page-14-1) .

<span id="page-18-0"></span>

Table 2: Benchmark overview. All agents were trained on a single Nvidia A100 GPU each.

Environment instances In earlier experiments, we observed that the performance of both Dreamer and PPO is robust to the number of environment instances. Based on the CPU resources available on our training machines, we use 16 environment instance by default. For BSuite, the benchmark requires using a single environment instance. We also use a single environment instance for Atari100K because the benchmark has a budget of 400K env steps whereas the maximum episode length in Atari is in principle 432K env steps. For Minecraft, we use 64 environments using remote CPU workers to speed up experiments because the environment is slower to step.

Seeds and error bars We run 5 seeds for each Dreamer and PPO per benchmark, with the exception of 1 seed for ProcGen due to computational constraints, 10 seeds for BSuite as required by the benchmark, and 10 seeds for Minecraft to reliably report the fraction of runs that achieve diamonds. All curves show the mean over seeds with one standard deviation shaded.

Computational choices All Dreamer and PPO agents in this paper were trained on a single Nvidia A100 GPU each. Dreamer uses the 200M model size by default. On the two control suitse, Dreamer the same performance using the substantially faster 12M model, making it more accessible to researchers. The replay ratio control the trade-off between computational cost and data efficiency as analyzed in [Figure 6](#page-8-0) and is chosen to fit the step budget of each benchmark.

Model sizes

To accommodate different computational budgets and analyze robustness to different model sizes, we define a range of models ranging from 12M to 400M parameters shown in [Table 3.](#page-19-0) The sizes are parameterized by the model dimension, which approximately increases in multiples of 1.5, alternating between powers of two and power of two scaled by 1.5. This yields tensor shapes that are multiples of 8 as required for hardware efficiency. Sizes of different network components derive from the model dimension. The MLPs have the model dimension as the number of hidden units. The sequence model has 8 times the number of recurrent units, split into 8 blocks of the same size as the MLPs. The convolutional encoder and decoder layers closest to the data use 16× fewer channels than the model dimension. Each latent also uses 16× fewer codes than the model dimension. The number of hidden layers and number of latents is fixed across model sizes. All hyperparamters, including the learning rate and batch size, are fixed across model sizes.

<span id="page-19-0"></span>

Table 3: Dreamer model sizes. The number of MLP hidden units defines the model dimension, from which recurrent units, convolutional channels, and number of codes per latent are derived. The number of layers and latents is constant across model sizes.

Previous Dreamer generations

We present the third generation of the Dreamer line of work. Where the distinction is useful, we refer to this algorithm as DreamerV3. The DreamerV1 algorithm[²¹](#page-12-6) was limited to continuous control, the DreamerV2 algorithm[²²](#page-12-7) surpassed human performance on Atari, and the DreamerV3 algorithm enables out-of-the-box learning across diverse benchmarks.

We summarize the changes that DreamerV3 introduces as follows:

• Robustness techniques: Observation symlog, KL balance and free bits, 1% unimix for all categoricals, percentile return normalization, symexp twohot loss for the reward head and critic.
• Network architecture: Block GRU, RMSNorm normalization, SiLu activation.
• Optimizer: Adaptive gradient clipping (AGC), LaProp (RMSProp followed by momentum).
• Replay buffer: Larger capacity, online queue, storing and updating latent states.

Hyperparameters

Table 4: Dreamer hyperparameters. The same values are used across all benchmarks, including proprioceptive and visual inputs, continuous and discrete actions, and 2D and 3D domains. We do not use any hyperparameter annealing, prioritized replay, weight decay, or dropout.

Minecraft

Game description With 100M monthly active users, Minecraft is one of the most popular video games worldwide. Minecraft features a procedurally generated 3D world of different biomes, including plains, forests, jungles, mountains, deserts, taiga, snowy tundra, ice spikes, swamps, savannahs, badlands, beaches, stone shores, rivers, and oceans. The world consists of 1 meter sized blocks that the player and break and place. There are about 30 different creatures that the player can interact with or fight. From gathered resources, the player can use over 350 recipes to craft new items and progress through the technology tree, all while ensuring safety and food supply to survive. There are many conceivable tasks in Minecraft and as a first step, the research community has focused on the salient task of obtaining a diamonds, a rare item found deep underground and requires progressing through the technology tree.

Learning environment We built the Minecraft Diamond environment on top of MineRL to define a flat categorical action space and fix issues we discovered with the original environments via human play testing. For example, when breaking diamond ore, the item sometimes jumps into the inventory and sometimes needs to be collected from the ground. The original environment terminates episodes when breaking diamond ore so that many successful episodes end before collecting the item and thus without the reward. We remove this early termination condition and end episodes when the player dies or after 36000 steps, corresponding to 30 minutes at the control frequency of 20Hz. Another issue is that the jump action has to be held for longer than one control step to trigger a jump, which we solve by keeping the key pressed in the background for 200ms. We built the environment on top of MineRL v0.4.4[¹⁸](#page-12-3), which offers abstract crafting actions. The Minecraft version is 1.11.2.

Observations and rewards The agent observes a 64 × 64 × 3 first-person image, an inventory count vector for the over 400 items, a vector of maximum inventory counts since episode begin to tell the agent which milestones it has achieved, a one-hot vector indicating the equipped item, and scalar inputs for the health, hunger, and breath levels. We follow the sparse reward structure of the MineRL competition environment[¹⁸](#page-12-3) that rewards 12 milestones leading up to the diamond, for obtaining the items log, plank, stick, crafting table, wooden pickaxe, cobblestone, stone pickaxe, iron ore, furnace, iron ingot, iron pickaxe, and diamond. The reward for each item is only given once per episode, and the agent has to learn to collect certain items multiple times to achieve the next milestone. To make the return easy to interpret, we give a reward of +1 for each milestone instead of scaling rewards based on how valuable each item is. Additionally, we give −0.01 for each lost heart and +0.01 for each restored heart, but did not investigate whether this is helpful.

Minecraft video predictions


Figure 7: Multi-step predictions on Minecraft. The world model receives the first 5 frames as context input and the predicts 45 steps into the future given the action sequence and without access to intermediate images.

Minecraft additional results

Table 5: Minecraft Diamond scores at 100M environment steps.


Figure 8: Minecraft learning curves.


Figure 9: Item success rates as a percentage of episodes. Dreamer obtains items at substantially higher rates than the baselines and continues to improve until the 100M step budget. At the budget, Dreamer obtains diamonds in 0.4% of episodes, leaving a challenge for future research. This metric differs from [Figure 5,](#page-7-0) which shows that over the course of training, 100% of Dreamer agents obtain one or more diamonds regardless of episode boundaries, compared to 0% of the baseline agents.

Atari learning curves


Figure 10: Atari learning curves.

Atari scores

<span id="page-25-0"></span>

Table 6: Atari scores.

ProcGen learning curves


Figure 11: ProcGen learning curves.

ProcGen scores

Table 7: ProcGen scores. The PPO implementation we use throughout our paper under fixed hyperparameters performs on par or better than the original PPO, which its authors describe as highly tuned with near optimal hyperparameters[³⁷](#page-13-6) .

DMLab learning curves


Figure 12: DMLab learning curves.

DMLab scores

Table 8: DMLab scores at 100M environment steps and larger budgets. The IMPALA agent corresponds to "IMPALA (deep)" presented by Kapturowski et al. [³⁵](#page-13-4) who made the learning curves available.

Atari100k learning curves


Figure 13: Atari100k learning curves.

Atari100k scores

Table 9: Atari100k scores at 400K environment steps, corresponding to 100k agent steps.

<span id="page-31-0"></span>

Table 10: Evaluation protocols for the Atari 100k benchmark. Computational resources are converted to A100 GPU days. EfficientMuZero[⁴⁴](#page-13-13) achieves the highest scores but changed the environment configuration from the standard[¹⁷](#page-12-2). IRIS uses a separate hyperparameter for its exploration strength on Freeway.

Proprioceptive control learning curves


Figure 14: DeepMind Control Suite learning curves under proprioceptive inputs.

Proprioceptive control scores

Table 11: DeepMind Control Suite scores under proprioceptive inputs.

Visual control learning curves


Figure 15: DeepMind Control Suite learning curves under visual inputs.

Visual control scores

Table 12: DeepMind Control Suite scores under visual inputs.

BSuite performance spectrum


Figure 16: BSuite scores visualized by category[⁴⁸](#page-14-2). Dreamer exceeds previous methods in the categories scale and memory. The scale category measure robustness to reward scales.

BSuite scores

Table 13: BSuite scores for each task averaged over environment configurations, as well as aggregated performance by category and over all tasks.

Robustness ablations


Figure 17: Individual learning curves for the robustness ablation experiment. All robustness techniques contribute to the overall performance of Dreamer, although each individual technique may only improve the performance on a subset of the tasks.

Learning signal ablations


Figure 18: Individual learning curves for the learning signal ablation experiment. Dreamer relies predominantly on the undersupervised reconstruction objective of its world model and additional reward and value gradients further improve performance on a subset of tasks.