2023-07-31 · ICML · 88 citations · club pick

Learning to Model the World with Language

Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, Pieter Abbeel, Dan Klein, Anca Dragan

Published at ICML (the arXiv record still lists it as a preprint). 88 citations, 8 of them influential, as of the last refresh.

Abstract

To interact with humans and act in the world, agents need to understand the range of language that people use and relate it to the visual world. While current agents can learn to execute simple language instructions, we aim to build agents that leverage diverse language -- language like "this button turns on the TV" or "I put the bowls away" -- that conveys general knowledge, describes the state of the world, provides interactive feedback, and more. Our key idea is that agents should interpret such diverse language as a signal that helps them predict the future: what they will observe, how the world will behave, and which situations will be rewarded. This perspective unifies language understanding with future prediction as a powerful self-supervised learning objective. We instantiate this in Dynalang, an agent that learns a multimodal world model to predict future text and image representations, and learns to act from imagined model rollouts. While current methods that learn language-conditioned policies degrade in performance with more diverse types of language, we show that Dynalang learns to leverage environment descriptions, game rules, and instructions to excel on tasks ranging from game-playing to navigating photorealistic home scans. Finally, we show that our method enables additional capabilities due to learning a generative model: Dynalang can be pretrained on text-only data, enabling learning from offline datasets, and generate language grounded in an environment.

arXiv comment: ICML 2024. Website: https://dynalang.github.io/

Ten-minute slide kit

Six slides is the whole talk: what was broken, what people tried, what these authors did, what the numbers say, where it falls over, and the sentence people should remember.

SLIDE 1

The problem

A long-standing goal of artificial intelligence is to develop agents that can use language to interact naturally with people in the physical world \(Winograd, 1972\). Current embodied agents can follow basic instructions like "bring me the apple" \(Driess et al., 2023\). However, the full potential of language affords much richer communication beyond task

SLIDE 2

What came before

Much work has focused on teaching reinforcement learning agents to utilize language to solve tasks by directly conditioning policies on language \(Lynch & Sermanet, 2021; Shridhar et al., 2022; Abramson et al., 2020\). More similar to our work, recent work proposes text-conditioning a video model trained on expert demonstrations and using the model for planning \(Du et al., 2023b; Yang et al., 2023\). However, language in these settings has thus far been limited to short instructions, and only a few works…

SLIDE 3

The method

Jessy Lin 1 Yuqing Du 1 Olivia Watkins 1 Danijar Hafner 1 Pieter Abbeel 1 Dan Klein 1 Anca Dragan 1 Step environment rt, ct, xt, lt ← env(at−1). Encode observations zt ∼ enc(xt, lt, ht). Execute action at ∼ π(at | ht, zt). Add transition (rt, ct, xt, lt, at) to replay

SLIDE 4

What they measured

Our experiments test the following hypotheses: H1) Aligning image and language as single (image, token) pairs per timestep outperforms other methods for incorporating language into DreamerV3 \(Section 4.1\). H2) Dynalang can better utilize diverse types of language to improve task performance over language-conditioned policies. To test this, we investigate whether Dynalang performance improves when provided with different kinds of language hints in HomeGrid \(Section 4.2\) and game manuals in Messenger \(Section…

SLIDE 5

Where it breaks

Not recoverable from the parsed text. Read this section in the paper yourself.

SLIDE 6

One-line takeaway

Adds language conditioning to world models

Assembled from the paper's own PDF, parsed with its layout intact, 105,529 characters of it, then split on the paper's own section headings. Extractive, not generated: every sentence here is lifted from the paper. The takeaway line is the club's own one-liner from its reading list. Check it before you present it.

Figures worth putting on a slide

Figure 1. Dynalang learns to use language to make predictions about future (text + image) observations and rewards. Here, we show real model predictions in the HomeGrid environment
Figure 2. We consider a range of environments that feature visual inputs and diverse types of language. We introduce HomeGrid, a challenging visual gridworld with instructions and
Figure 3. During world model learning, the model compresses observations of image frames and text to a latent representation. The model is trained to predict the next representatio
Figure 4. Comparison of ways to equip the world model with language inputs on Messenger S1. We compare ways of conditioning DreamerV3 on language and find that Dynalang substantial
Figure 5. HomeGrid performance after 50M steps (2 seeds). Dynalang learns to use all types of language hints to score higher than when just provided with the task information, outp
Figure 6. Messenger training performance (2 seeds). Dynalang outperforms language-conditioned IMPALA and R2D2, as well as the task-specific EMMA architecture, fitting the most comp

Presented at

Saturday, May 9, 2026

Saturday Robotics, session 7

Robotics & World Models Reading Club 07: Learning to Dream: World Models, Imagination, Path to Foundation Models for Control — Los Altos

Listed on the event page as “Dynalang (2023)”. The arXiv title above is the record.

Why the club picked it. Adds language conditioning to world models