2022-06-08 · NeurIPS · 145 citations · club pick

Deep Hierarchical Planning from Pixels

Danijar Hafner, Kuang-Huei Lee, Ian Fischer, Pieter Abbeel

Published at NeurIPS (the arXiv record still lists it as a preprint). 145 citations, 13 of them influential, as of the last refresh.

Abstract

Intelligent agents need to select long sequences of actions to solve complex tasks. While humans easily break down tasks into subgoals and reach them through millions of muscle commands, current artificial intelligence is limited to tasks with horizons of a few hundred decisions, despite large compute budgets. Research on hierarchical reinforcement learning aims to overcome this limitation but has proven to be challenging, current methods rely on manually specified goal spaces or subtasks, and no general solution exists. We introduce Director, a practical method for learning hierarchical behaviors directly from pixels by planning inside the latent space of a learned world model. The high-level policy maximizes task and exploration rewards by selecting latent goals and the low-level policy learns to achieve the goals. Despite operating in latent space, the decisions are interpretable because the world model can decode goals into images for visualization. Director outperforms exploration methods on tasks with sparse rewards, including 3D maze traversal with a quadruped robot from an egocentric camera and proprioception, without access to the global position or top-down view that was used by prior work. Director also learns successful behaviors across a wide range of environments, including visual control, Atari games, and DMLab levels.

arXiv comment: Website: https://danijar.com/director

Ten-minute slide kit

Six slides is the whole talk: what was broken, what people tried, what these authors did, what the numbers say, where it falls over, and the sentence people should remember.

SLIDE 1

The problem

Artificial agents have achieved remarkable performance on reactive video games \(Mnih et al., 2015; Badia et al., 2020\) or board games that last for a few hundred moves \(Silver et al., 2017\). However, solving complex control problems can require millions of time steps. For example, consider a robot that needs to navigate along the sidewalk and cross streets to buy groceries and then return home and cook a meal with those

SLIDE 2

What came before

These approaches are described in more detail in Appendix I. Relatively few works have demonstrated successful learning of hierarchical behaviors directly from pixels without domain-specific knowledge, such as global XY positions, manually specified pretraining tasks, or precollected diverse experience datasets. HSD-3 \(Gehring et al., 2021\) showed transfer benefits for low-dimensional control

SLIDE 3

The method

Director learns a world model that compresses the history of observations into a compact feature space and enables planning in this space \(Watter et al., 2015; Zhang et al., 2019\). We use the Recurrent State Space Model (RSSM) model of PlaNet \(Hafner et al., 2018\), which we briefly review here to introduce notation. The dynamics model predicts future states without the corresponding observations. The decoder reconstructs observations to provide a rich learning

SLIDE 4

What they measured

We evaluate Director on two challenging benchmark suites with visual inputs and very sparse rewards, which we expect to be challenging to solve using a flat policy without hierarchy \(Section 3.1\). We further evaluate Director on a wide range of standard tasks from the literature to demonstrate its generality and ensure that the hierarchy is not harmful in simple settings \(Section 3.2\). We use a fixed set of hyperparameters not only across tasks but also across domains, detailed in Table F.1. Finally, we offer…

SLIDE 5

Where it breaks

We present Director, a reinforcement learning agent that learns hierarchical behaviors from pixels by planning in the latent space of a learned world model. To simplify the control problem for the manager, we compress goal representations into compact discrete codes. Our experiments demonstrate the effectiveness of Director on two benchmark suites with very sparse rewards from

SLIDE 6

One-line takeaway

Hierarchical latent planning for long-horizon decision making

Assembled from the paper's own PDF, parsed with its layout intact, 77,341 characters of it, then split on the paper's own section headings. Extractive, not generated: every sentence here is lifted from the paper. The takeaway line is the club's own one-liner from its reading list. Check it before you present it.

Figures worth putting on a slide

Figure 1: Director on Ant Maze M from egocentric camera inputs. The top row shows agent inputs. The bottom row shows the internal subgoals of the agent. The goals are latent vector
Figure 2: Director is based on the world model of PlaNet [\(Hafner et al.,](#page-10-1) [2018\)](#page-10-1) that predicts ahead in a compact representation space. The world model
Figure 3: Comparison of Ant Mazes in the literature and this paper. HIRO [\(Nachum et al.,](#page-10-4) [2018a\)](#page-10-4) provided global XY coordinates of the goal and robot p
Figure 4: Egocentric Ant Maze benchmark. A quadruped robot is controlled through joint torques to navigate to a fixed location in a 3D maze, given only first-person camera and prop
Figure 3.](#page-4-0) In this paper, we instead attempt learning directly from firstperson camera inputs, without privileged information, and a single sparse reward that the agent
Figure 5: Visual Pin Pad benchmark. The agent controls the black square to move in four directions. Each environment has a different number of pads that can be activated by walking

Presented at

Saturday, May 9, 2026

Saturday Robotics, session 7

Robotics & World Models Reading Club 07: Learning to Dream: World Models, Imagination, Path to Foundation Models for Control — Los Altos

Listed on the event page as “Director (2022)”. The arXiv title above is the record.

Why the club picked it. Hierarchical latent planning for long-horizon decision making