[
{
"id": "1803.10122",
"slug": "world-models",
"backend": "firecrawl",
"chars": 102050,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Humans develop a mental model of the world based on what they are able to perceive with their limited senses. The decisions and actions we make are based on this internal model. Jay Wright Forrester, the father of system dynamics, described a mental model as: *The image of the world around us, which we carry in our head, is just a"
},
{
"n": 2,
"title": "What came before",
"body": "There is extensive literature on learning a dynamics model, and using this model to train a policy. Many concepts first explored in the 1980s for feed-forward neural networks (FNNs) \\(Werbos, 1987; Munro, 1987; Robinson & Fallside, 1989; Werbos, 1989; Nguyen & Widrow, 1989\\) and in the 1990s for RNNs \\(Schmidhuber, 1990a;b; 1991a; 1990c\\) laid some of the groundwork for *Learning to Think* \\(Schmidhuber, 2015a\\). The more recent PILCO \\(Deisenroth & Rasmussen, 2011; Duvenaud, 2016; McAllister & Rasmussen, 2016\\)\u2026"
},
{
"n": 3,
"title": "The method",
"body": "We present a simple model inspired by our own cognitive system. In this model, our agent has a visual sensory component that compresses what it sees into a small representative code. It also has a memory component that makes predictions about future codes based on historical information. Finally, our agent has a decision-making component that decides what actions to take based only on the representations created by its vision and memory"
},
{
"n": 4,
"title": "What they measured",
"body": "In this section, we describe how we can train the Agent model described earlier to solve a car racing task. To our knowledge, our agent is the first known solution to achieve the score required to solve this"
},
{
"n": 5,
"title": "Where it breaks",
"body": "We have demonstrated the possibility of training an agent to perform tasks entirely inside of its simulated latent space dream world. This approach offers many practical benefits. For instance, running computationally intensive game engines require using heavy compute resources for rendering the game states into image frames, or calculating physics not immediately relevant to the"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Introduced the core idea of learning a compressed latent simulator for control"
}
],
"takeaway_source": "club",
"figures": [
"Figure 1. A World Model, from Scott McCloud's *Understanding Comics.* [\\(McCloud,](#page-17-0) [1993;](#page-17-0) [E,](#page-15-0) [2012\\)](#page-15-0)",
"Figure 2. What we see is based on our brain's prediction of the future [\\(Kitaoka,](#page-17-0) [2002;](#page-17-0) [Watanabe et al.,](#page-20-0) [2018\\)](#page-20-0).",
"Figure 3. In this work, we build probabilistic generative models of OpenAI Gym environments. The RNN-based world models are trained using collected observations recorded from the a",
"Figure 4. Our agent consists of three components that work closely together: Vision (V), Memory (M), and Controller (C)",
"Figure 5. Flow diagram of a Variational Autoencoder (VAE).",
"Figure 6. RNN with a Mixture Density Network output layer. The MDN outputs the parameters of a mixture of Gaussian distribution used to sample a prediction of the next latent vecto"
],
"tables": [
"Table 1. CarRacing-v0 scores achieved using various methods.",
"Table 2. *Take Cover* scores at various temperature settings."
],
"coverage": 6
},
{
"id": "1811.04551",
"slug": "learning-latent-dynamics-for-planning-from-pixels",
"backend": "firecrawl",
"chars": 68526,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Planning is a natural and powerful approach to decision making problems with known dynamics, such as game playing and simulated robot control \\(Tassa et al., 2012; Silver et al., 2017; Moravc\u00edk et al. \u02c7 , 2017\\). To plan in unknown environments, the agent needs to learn the dynamics from experience. Learning dynamics models that are accurate enough for planning has been a long-standing"
},
{
"n": 2,
"title": "What came before",
"body": "Previous work in model-based reinforcement learning has focused on planning in low-dimensional state spaces \\(Gal et al., 2016; Higuera et al., 2018; Henaff et al., 2018; Comparison of PlaNet to the model-free algorithms A3C and D4PG reported by Tassa et al. \\(2018\\). The training curves for these are shown as orange lines in Figure 4 and as solid green lines in Figure 6 in their paper. From these, we estimate the number of episodes that D4PG takes to achieve\u2026"
},
{
"n": 3,
"title": "The method",
"body": "For planning, we need to evaluate thousands of action sequences at every time step of the agent. Therefore, we use a recurrent state-space model (RSSM) that can predict forward purely in latent space, similar to recently proposed models \\(Karl et al., 2016; Buesing et al., 2018; Doerr et al., 2018\\). This model can be thought of as a non-linear Kalman filter or sequential VAE. Instead of an extensive comparison to prior architectures, we highlight two findings that can guide future designs of dynamics models: our\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "We evaluate PlaNet on six continuous control tasks from pixels. We explore multiple design axes of the agent: the stochastic and deterministic paths in the dynamics model, iterative planning, and online experience collection. We refer to the appendix for hyper parameters \\(Appendix A\\) and additional experiments \\(Appendices C to E\\). Besides the action repeat, we use the same hyper parameters for all"
},
{
"n": 5,
"title": "Where it breaks",
"body": "We present PlaNet, a model-based agent that learns a latent dynamics model from image observations and chooses actions by fast planning in latent space. To enable accurate long-term predictions, we design a model with both stochastic and deterministic paths. We show that our agent succeeds at several continuous control tasks from image observations, reaching performance that is comparable to the best model-free algorithms while using 200\u00d7 fewer episodes and similar or less computation"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "First strong demonstration of planning directly in latent space (RSSM)"
}
],
"takeaway_source": "club",
"figures": [
"Figure 1: Image-based control domains used in our experiments. The images show agent observations before downscaling to 64 \u00d7 64 \u00d7 3 pixels. (a) The cartpole swingup task has a fixe",
"Figure 2: Latent dynamics model designs. In this example, the model observes the first two time steps and predicts the third. Circles represent stochastic variables and squares det",
"Figure 3: Unrolling schemes. The labels si|j are short for the state at time i conditioned on observations up to time j. Arrows pointing at shaded circles indicate log-l",
"Figure 1.](#page-1-0) These environments provide qualitatively different challenges. The cartpole swingup task requires a long planning horizon and to memorize the cart when it is ",
"Figure 4: Comparison of PlaNet to model-free algorithms and other model designs. Plots show test performance over the number of collected episodes. We compare PlaNet using our RSSM",
"Figure 5: Comparison of agent designs. Plots show test performance over the number of collected episodes. We compare PlaNet, a version that collects data under random actions (rand"
],
"tables": [
"Table 1: Comparison of PlaNet to the model-free algorithms A3C and D4PG reported by [Tassa et al.](#page-10-2) [\\(2018\\)](#page-10-2). The training curves for these are shown as or"
],
"coverage": 6
},
{
"id": "1912.01603",
"slug": "dream-to-control-learning-behaviors-by-latent-imagination",
"backend": "firecrawl",
"chars": 75410,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Value and Action Learned by Latent Imagination Learned Latent Dynamics Intelligent agents can achieve goals in complex environments even though they never encounter the exact same situation twice. This ability requires building representations of the world from past experience that enable generalization to novel situations. World models offer an explicit way to represent an agent's knowledge about the world in a parametric model that can make predictions about the"
},
{
"n": 2,
"title": "What came before",
"body": "Prior works learn latent dynamics for visual control by derivative-free policy learning or online planning, augment model-free agents with multi-step predictions, or use analytic gradients of Qvalues or multi-step rewards, often for low-dimensional tasks. In comparison, Dreamer uses analytic gradients to efficiently learn long-horizon behaviors for visual control purely by latent imagination. Control with latent dynamics E2C \\(Watter et al., 2015\\) and RCE \\(Banijamali et al., 2017\\) embed images to predict\u2026"
},
{
"n": 3,
"title": "The method",
"body": "Reinforcement learning We formulate visual control as a partially observable Markov decision process (POMDP) with discrete time step t \u2208 [1; T], continuous vector-valued actions at \u223c p(at | o\u2264t, at \u223c p(ot, rt | op PT t=1\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "We experimentally evaluate Dreamer on a variety of control tasks. We designed the experiments to compare Dreamer to current best methods in the literature, and to evaluate its ability to solve tasks with long horizons, continuous actions, discrete actions, and early termination. We further compare the orthogonal choice of learning objective for the world model. The source code for all our experiments and videos of Dreamer are available at"
},
{
"n": 5,
"title": "Where it breaks",
"body": ""
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Replaces search/planning with actor-critic trained entirely in imagination"
}
],
"takeaway_source": "club",
"figures": [
"Figure 1: Dreamer learns a world model from past experience and efficiently learns farsighted behaviors in its latent space by backpropagating value estimates back through imagined",
"Figure 1.](#page-0-0) The values optimize Bellman consistency for imagined rewards and the policy maximizes the values by propagating their analytic gradients back through the dyna",
"Figure 2: Image observations for 5 of the 20 visual control tasks used in our experiments. The tasks pose a variety of challenges including contact dynamics, sparse rewards, many d",
"Figure 2.](#page-1-0) Using the same hyper parameters for all tasks, Dreamer exceeds previous model-based and model-free agents in terms of data-efficiency, computation time, and f",
"Figure 3: Components of Dreamer. (a) From the dataset of past experience, the agent learns to encode observations and actions into compact latent states ( ), for example via recons",
"Figure 4: Imagination horizons. We compare the final performance of Dreamer, learning an action model without value prediction, and online planning using PlaNet. Learning a state v"
],
"tables": [],
"coverage": 5
},
{
"id": "2010.02193",
"slug": "mastering-atari-with-discrete-world-models",
"backend": "firecrawl",
"chars": 83107,
"slides": [
{
"n": 1,
"title": "The problem",
"body": " To successfully operate in unknown environments, reinforcement learning agents need to learn about their environments over time. World models are an explicit way to represent an agent's knowledge about its environment. Compared to model-free reinforcement learning that learns through trial and error, world models facilitate generalization and can predict the outcomes of potential actions to enable planning \\(Sutton,"
},
{
"n": 2,
"title": "What came before",
"body": "Model-free Atari The majority of agents applied to the Atari benchmark have been trained using model-free algorithms. DQN \\(Mnih et al., 2015\\) showed that deep neural network policies can be trained using Q-learning by incorporating experience replay and target networks. Several works have extended DQN to incorporate bias correction as in DDQN \\(Van Hasselt et al., 2015\\), prioritized experience replay \\(Schaul et al., 2015\\), architectural improvements \\(Wang et al., 2016\\), and distributional value learning\u2026"
},
{
"n": 3,
"title": "The method",
"body": "Danijar Hafner \u2217 Google Research Timothy Lillicrap DeepMind Mohammad Norouzi Google Research Jimmy Ba University of Toronto World models summarize an agent's experience into a predictive model that can be used in place of the environment to learn behaviors. When inputs are high-dimensional images, it is beneficial to learn compact state representations of the inputs to predict ahead in this learned latent space \\(Watter et al., 2015; Karl et al., 2016; Ha and Schmidhuber, 2018\\). These models are called\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "We evaluate DreamerV2 on the well-established Atari benchmark with sticky actions, comparing to four strong model-free algorithms. DreamerV2 outperforms the four model-free algorithms in all scenarios. For an extensive comparison, we report four scores according to four aggregation protocols and give a recommendation for meaningfully aggregating scores across games going forward. We also ablate the importance of discrete representations in the world"
},
{
"n": 5,
"title": "Where it breaks",
"body": "We present DreamerV2, a model-based agent that achieves human-level performance on the Atari 200M benchmark by learning behaviors purely from the latent-space predictions of a separately trained world model. Using a single GPU and a single environment instance, DreamerV2 outperforms top model-free single-GPU agents Rainbow and IQN using the same computational budget and training time. To develop DreamerV2, we apply several small modifications to the Dreamer agent \\(Hafner et al.,"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Introduces discrete latent states (categorical RSSM)"
}
],
"takeaway_source": "club",
"figures": [
"Figure 1: Gamer normalized median score on the Atari benchmark of 55 games with sticky actions at 200M steps. DreamerV2 is the first agent that learns purely within a world model t",
"Figure 1.](#page-0-0) Learning successful behaviors purely within the world model demonstrates that the world model learns to accurately represent the environment. To achieve this,",
"Figure 2.](#page-2-0) The RSSM uses a sequence of deterministic recurrent states ht, from which it computes two distributions over stochastic states at each step. The posterior sta",
"Figure 2: World Model Learning. The training sequence of images xt is encoded using the CNN. The RSSM uses a sequence of deterministic recurrent states ht. At each step,",
"Figure 3.](#page-4-0) The KL loss both trains the prior and regularizes how much information the posterior incorporates from the image. The regularization increases robustness to n",
"Figure 3: Actor Critic Learning. The world model learned in [Figure 2](#page-2-0) is used for learning a policy from trajectories imagined in the compact latent space. The trajecto"
],
"tables": [
"Table 1: Atari performance at 200M steps. The scores of the 55 games are aggregated using the four different protocols described in [Section 3.](#page-5-0) To overcome limitations ",
"Table 2: Ablations to DreamerV2 measured by their Atari performance at 200M frames, sorted by the last column. The this experiment uses a slightly earlier version of DreamerV2 comp",
"Table 2.](#page-8-0) Refer to the appendix for the score curves of the individual tasks.",
"Table 3: Conceptual comparison of recent RL algorithms that leverage planning with a learned model. DreamerV2 and SimPLe learn complete models of the environment by leveraging the "
],
"coverage": 6
},
{
"id": "2112.06374",
"slug": "learning-generalizable-vision-tactile-robotic-grasping-strategy-for-de",
"backend": "firecrawl",
"chars": 69132,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "R OBOT manipulation has been widely used in industries for decades, but mostly for repetitive tasks in structured environment where there is little uncertainty or contact deformation in manipulated objects. For the tasks where object contact parameters are prone to vary, such as fruit grasping, they are still challenging for robotic systems [\\[1\\]](#page-10-0). Loose grips with small grasping forces can cause objects to slip, while large grasping forces can cause"
},
{
"n": 2,
"title": "What came before",
"body": ""
},
{
"n": 3,
"title": "The method",
"body": "In this section, we describe the details of the grasping framework and each Transformer model. In order to give robots the ability to estimate the safe grasping force, we first let the robot obtain physical information about the target objects (fruits in this work) by performing two explorative actions, *pinching* and *sliding*, on the objects. To avoid any potential damage, these actions have minimum interaction with the objects. To monitor the interactions and record the data, the robot is equipped with two\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "In this section, we present our experiments using the Transformer models. In this section, we evaluate our framework with unseen irregular objects (i.e., corn and banana) during online experiments, the same as Sec. Unlike kiwi, corn and bananas are new objects with irregular shapes and different contact surface textures, as shown in Fig. 14. Note that, we use a new gripper in this section for this experiment, which has almost exactly the same mechanical properties as the previous one. *1) Experiment Evaluation:*\u2026"
},
{
"n": 5,
"title": "Where it breaks",
"body": "Our experiments demonstrate that the Transformer models can enable robotic grasping tasks in both the object classi- 5https://drive.google.com/file/ fication and robot control domain. The results indicate that they outperform traditional models, such as CNN+LSTM, for classification tasks like slip detection and grasping outcome prediction. In addition, our Transformer-based grasping framework is able to select the grasping strength to safely grasp fruits with varying hardness\u2026"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "This study proposes a transformer-based robotic grasping framework for rigid grippers that leverage tactile and visual information for safe object grasping and shows that the transformer models outperform a CNN + LSTM model in terms of grasping accuracy and computational efficiency."
}
],
"takeaway_source": "semantic-scholar",
"figures": [],
"tables": [],
"coverage": 5
},
{
"id": "2206.04114",
"slug": "deep-hierarchical-planning-from-pixels",
"backend": "firecrawl",
"chars": 77341,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Artificial agents have achieved remarkable performance on reactive video games \\(Mnih et al., 2015; Badia et al., 2020\\) or board games that last for a few hundred moves \\(Silver et al., 2017\\). However, solving complex control problems can require millions of time steps. For example, consider a robot that needs to navigate along the sidewalk and cross streets to buy groceries and then return home and cook a meal with those"
},
{
"n": 2,
"title": "What came before",
"body": "These approaches are described in more detail in Appendix I. Relatively few works have demonstrated successful learning of hierarchical behaviors directly from pixels without domain-specific knowledge, such as global XY positions, manually specified pretraining tasks, or precollected diverse experience datasets. HSD-3 \\(Gehring et al., 2021\\) showed transfer benefits for low-dimensional control"
},
{
"n": 3,
"title": "The method",
"body": "Director learns a world model that compresses the history of observations into a compact feature space and enables planning in this space \\(Watter et al., 2015; Zhang et al., 2019\\). We use the Recurrent State Space Model (RSSM) model of PlaNet \\(Hafner et al., 2018\\), which we briefly review here to introduce notation. The dynamics model predicts future states without the corresponding observations. The decoder reconstructs observations to provide a rich learning"
},
{
"n": 4,
"title": "What they measured",
"body": "We evaluate Director on two challenging benchmark suites with visual inputs and very sparse rewards, which we expect to be challenging to solve using a flat policy without hierarchy \\(Section 3.1\\). We further evaluate Director on a wide range of standard tasks from the literature to demonstrate its generality and ensure that the hierarchy is not harmful in simple settings \\(Section 3.2\\). We use a fixed set of hyperparameters not only across tasks but also across domains, detailed in Table F.1. Finally, we offer\u2026"
},
{
"n": 5,
"title": "Where it breaks",
"body": "We present Director, a reinforcement learning agent that learns hierarchical behaviors from pixels by planning in the latent space of a learned world model. To simplify the control problem for the manager, we compress goal representations into compact discrete codes. Our experiments demonstrate the effectiveness of Director on two benchmark suites with very sparse rewards from"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Hierarchical latent planning for long-horizon decision making"
}
],
"takeaway_source": "club",
"figures": [
"Figure 1: Director on Ant Maze M from egocentric camera inputs. The top row shows agent inputs. The bottom row shows the internal subgoals of the agent. The goals are latent vector",
"Figure 2: Director is based on the world model of PlaNet [\\(Hafner et al.,](#page-10-1) [2018\\)](#page-10-1) that predicts ahead in a compact representation space. The world model ",
"Figure 3: Comparison of Ant Mazes in the literature and this paper. HIRO [\\(Nachum et al.,](#page-10-4) [2018a\\)](#page-10-4) provided global XY coordinates of the goal and robot p",
"Figure 4: Egocentric Ant Maze benchmark. A quadruped robot is controlled through joint torques to navigate to a fixed location in a 3D maze, given only first-person camera and prop",
"Figure 3.](#page-4-0) In this paper, we instead attempt learning directly from firstperson camera inputs, without privileged information, and a single sparse reward that the agent ",
"Figure 5: Visual Pin Pad benchmark. The agent controls the black square to move in four directions. Each environment has a different number of pads that can be activated by walking"
],
"tables": [],
"coverage": 6
},
{
"id": "2206.14176",
"slug": "daydreamer-world-models-for-physical-robot-learning",
"backend": "firecrawl",
"chars": 54321,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Teaching robots to solve complex tasks in the real world is a foundational problem of robotics research. Deep reinforcement learning (RL) offers a popular approach to robot learning that enables robots to improve their behavior over time through trial and error. However, current algorithms require too much interaction with the environment to learn successful behaviors, making them impractical for many real world"
},
{
"n": 2,
"title": "What came before",
"body": "However, designing simulated tasks and collecting expert demonstrations is time-consuming. Moreover, many of these approaches require specialized algorithms for leveraging offline experience, demonstrations, or simulator inaccuracies. In contrast, our experiments show that learning end-to-end from rewards in the physical world is feasible for a diverse range of tasks through world"
},
{
"n": 3,
"title": "The method",
"body": "Philipp Wu\\* Alejandro Escontrela\\* Danijar Hafner\\* Ken Goldberg Pieter Abbeel University of California, Berkeley \\*Equal contribution Abstract: To solve tasks in complex environments, robots need to learn from experience. Deep reinforcement learning is a common approach to robot learning but requires a large amount of trial and error to learn, limiting its deployment in the physical world. As a consequence, many advances in robot learning rely on simulators. On the other hand, learning inside of simulators fails\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "We evaluate Dreamer on 4 robots, each with a different task, and compare its performance to appropriate algorithmic and human baselines. The experiments are representative of common robotic tasks, such as locomotion, manipulation, and navigation. The tasks pose a diverse range of challenges, including continuous and discrete actions, dense and sparse rewards, proprioceptive and image observations, and sensor fusion. Learned world models have various properties that make them well suited for robot"
},
{
"n": 5,
"title": "Where it breaks",
"body": "We applied Dreamer to physical robot learning, finding that modern world models enable sampleefficient robot learning for a range of tasks, from scratch in the real world and without simulators. We also find that the approach is generally applicable in that it can solve robot locomotion, manipulation, and navigation tasks without changing hyperparameters. Dreamer taught a quadruped robot to roll off the back, stand up, and walk in 1 hour from scratch, which previously required extensive training in simulation\u2026"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Real-world robot learning using Dreamer-style latent imagination"
}
],
"takeaway_source": "club",
"figures": [
"Figure 1: To study the applicability of Dreamer for sample-efficient robot learning, we apply the algorithm to learn robot locomotion, manipulation, and navigation tasks from scrat",
"Figure 2: Dreamer follows a simple pipeline for online learning on robot hardware without simulators. The current learned policy collects experience on the robot. This experience i",
"Figure 3: Neural Network Training We leverage the Dreamer algorithm [\\(Hafner et al.,](#page-8-0) [2019;](#page-8-0) [2020\\)](#page-8-0) for fast robot learning in real world. Drea",
"Figure 4: A1 Quadruped Walking Starting from lying on its back with the feet in the air, Dreamer learns to roll over, stand up, and walk in 1 hour of real world training time, with",
"Figure 8: Within 10 minutes of perturbing the learned walking behavior, the robot adapts to withstanding pushes or quickly rolling over and back on its feet. This high-dimensional ",
"Figure 5: UR5 Multi Object Visual Pick and Place This task requires learning to locate three ball objects from third-person camera images, grasp them, and move them into the other "
],
"tables": [],
"coverage": 6
},
{
"id": "2301.04104",
"slug": "mastering-diverse-domains-through-world-models",
"backend": "firecrawl",
"chars": 102522,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Reinforcement learning has enabled computers to solve tasks through interaction, such as surpassing humans in the games of Go and Dota1,2 . It is also a key component for improving large language models beyond what is demonstrated in their pretraining data 3,4 . While PPO5 has become a standard algorithm in the field of reinforcement learning, more specialized algorithms are often employed to achieve higher"
},
{
"n": 2,
"title": "What came before",
"body": ""
},
{
"n": 3,
"title": "The method",
"body": "Danijar Hafner,12 Jurgis Pasukonis,1 Jimmy Ba,2 Timothy Lillicrap1 The world model learns compact representations of sensory inputs through autoencoding23 and enables planning by predicting future representations and rewards for potential actions. We implement the world model as a Recurrent State-Space Model (RSSM) 24, shown in First, an encoder maps sensory inputs xt to stochastic representations zt . Then, a sequence model\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "We evaluate the generality of Dreamer across 8 domains\u2014with over 150 tasks\u2014under fixed hyperparameters. We designed the experiments to compare Dreamer to the best methods in the literature, which are often specifically designed and tuned for the benchmark at hand. We further compare to a high-quality implementation of PPO5 , a standard reinforcement learning algorithm that is known for its robustness. We run PPO with fixed hyperparameters chosen to maximize performance across domains and that reproduce\u2026"
},
{
"n": 5,
"title": "Where it breaks",
"body": ""
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Unified training recipe across many domains with fixed hyperparameters"
}
],
"takeaway_source": "club",
"figures": [
"Figure 1: Benchmark summary. a, Using fixed hyperparameters across all domains, Dreamer outperforms tuned expert algorithms across a wide range of benchmarks and data budgets. Drea",
"Figure 2: Diverse visual domains used in the experiments. Dreamer succeeds across these domains, ranging from robot locomotion and manipulation tasks over Atari games, procedurally",
"Figure 3: Training process of Dreamer. The world model encodes sensory inputs into discrete representations zt that are predicted by a sequence model with recurrent stat",
"Figure 3.](#page-2-0) First, an encoder maps sensory inputs xt to stochastic representations zt . Then, a sequence model with recurrent state ht pr",
"Figure 4: Multi-step video predictions of a DMLab maze (top) and a quadrupedal robot (bottom). Given 5 context images and the full action sequence, the model predicts 45 frames int",
"Figure 5: Fraction of trained agents that discover each of the three latest items in the Minecraft Diamond task. Although previous algorithms progress up to the iron pickaxe, Dream"
],
"tables": [
"Table 1: PPO hyperparameters used across all benchmarks.",
"Table 6.](#page-25-0) DMLab[43](#page-13-12) uses 30 tasks [66](#page-15-5) and we use the fixed action space[36](#page-13-5)[,35](#page-13-4). We evaluate at",
"Table 2: Benchmark overview. All agents were trained on a single Nvidia A100 GPU each.",
"Table 3.](#page-19-0) The sizes are parameterized by the model dimension, which approximately increases in multiples of 1.5, alternating between powers of two and power of two scal"
],
"coverage": 4
},
{
"id": "2308.01399",
"slug": "learning-to-model-the-world-with-language",
"backend": "firecrawl",
"chars": 105529,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "A long-standing goal of artificial intelligence is to develop agents that can use language to interact naturally with people in the physical world \\(Winograd, 1972\\). Current embodied agents can follow basic instructions like \"bring me the apple\" \\(Driess et al., 2023\\). However, the full potential of language affords much richer communication beyond task"
},
{
"n": 2,
"title": "What came before",
"body": "Much work has focused on teaching reinforcement learning agents to utilize language to solve tasks by directly conditioning policies on language \\(Lynch & Sermanet, 2021; Shridhar et al., 2022; Abramson et al., 2020\\). More similar to our work, recent work proposes text-conditioning a video model trained on expert demonstrations and using the model for planning \\(Du et al., 2023b; Yang et al., 2023\\). However, language in these settings has thus far been limited to short instructions, and only a few works\u2026"
},
{
"n": 3,
"title": "The method",
"body": "Jessy Lin 1 Yuqing Du 1 Olivia Watkins 1 Danijar Hafner 1 Pieter Abbeel 1 Dan Klein 1 Anca Dragan 1 Step environment rt, ct, xt, lt \u2190 env(at\u22121). Encode observations zt \u223c enc(xt, lt, ht). Execute action at \u223c \u03c0(at | ht, zt). Add transition (rt, ct, xt, lt, at) to replay"
},
{
"n": 4,
"title": "What they measured",
"body": "Our experiments test the following hypotheses: H1) Aligning image and language as single (image, token) pairs per timestep outperforms other methods for incorporating language into DreamerV3 \\(Section 4.1\\). H2) Dynalang can better utilize diverse types of language to improve task performance over language-conditioned policies. To test this, we investigate whether Dynalang performance improves when provided with different kinds of language hints in HomeGrid \\(Section 4.2\\) and game manuals in Messenger \\(Section\u2026"
},
{
"n": 5,
"title": "Where it breaks",
"body": ""
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Adds language conditioning to world models"
}
],
"takeaway_source": "club",
"figures": [
"Figure 1. Dynalang learns to use language to make predictions about future (text + image) observations and rewards. Here, we show real model predictions in the HomeGrid environment",
"Figure 2. We consider a range of environments that feature visual inputs and diverse types of language. We introduce HomeGrid, a challenging visual gridworld with instructions and ",
"Figure 3. During world model learning, the model compresses observations of image frames and text to a latent representation. The model is trained to predict the next representatio",
"Figure 4. Comparison of ways to equip the world model with language inputs on Messenger S1. We compare ways of conditioning DreamerV3 on language and find that Dynalang substantial",
"Figure 5. HomeGrid performance after 50M steps (2 seeds). Dynalang learns to use all types of language hints to score higher than when just provided with the task information, outp",
"Figure 6. Messenger training performance (2 seeds). Dynalang outperforms language-conditioned IMPALA and R2D2, as well as the task-specific EMMA architecture, fitting the most comp"
],
"tables": [],
"coverage": 5
},
{
"id": "2406.07550",
"slug": "an-image-is-worth-32-tokens-for-reconstruction-and-generation",
"backend": "firecrawl",
"chars": 85200,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "In recent years, image generation has experienced remarkable progress, driven by the significant advancements in both transformers \\[19, 62, 66, 10, 67, [68\\]](#page-13-1) and diffusion models \\[16, 55, 29, 49, [21\\]](#page-10-3). Mirroring the trends in generative language models \\[48, [59\\]](#page-12-5), the architecture of many contemporary image generation models incorporate a standard image tokenizer and de-tokenizer. This array of models utilizes tokenized image representations\u2014ranging from continuous\u2026"
},
{
"n": 2,
"title": "What came before",
"body": "Images have been compressed since the early days of deep learning with autoencoders \\[27, [63\\]](#page-12-9). The general design of using an encoder that compresses high-dimensional images into a low-dimensional latent representation and then using a decoder to reverse the process, has proven to be successful over the years. Variational Autoencoders (VAEs) [\\[34\\]](#page-11-1) extend the paradigm by learning to map the input to a"
},
{
"n": 3,
"title": "The method",
"body": ""
},
{
"n": 4,
"title": "What they measured",
"body": "Building upon TiTok, we explore a range of configurations, including the model size and the number of tokens, to identify the most efficient and effective setup for a 1D image tokenizer. These preliminary experiments serve to provide a thorough evaluation, seeking a practical configuration of TiTok. Unless specified otherwise, we train all models with images of resolution H = 256 and W = 256, using the open-source MaskGIT-VQGAN [\\[9\\]](#page-10-8) to supply proxy codes for training. The patch size for both\u2026"
},
{
"n": 5,
"title": "Where it breaks",
"body": "This paper proposes a novel 1D tokenization method designed to eliminate the fixed corresponding constraints of existing 2D tokenization methods. The 1D tokenization model is validated using the Vector Quantization (VQ) tokenizer formulation alongside a Masked Transformer generator framework. Despite the promising results, the proposed 1D tokenization formulation theoretically has the potential to generalize to other tokenizer formulations (*e.g*., 1D-VAE), other generation frameworks (*e.g*., Diffusion Models),\u2026"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Keynote 2 by by Arjun Subramaniam (Factory Intelligence)"
}
],
"takeaway_source": "club",
"figures": [
"Figure 1: We propose TiTok, a compact 1D tokenizer leveraging region redundancy to represent an image with only 32 tokens for image reconstruction and generation.",
"Figure 2: A speed and quality comparison of TiTok and prior arts on ImageNet 256 \u00d7 256 and 512 \u00d7 512 generation benchmarks. Speed-up is compared against DiT-XL/2 [\\[49\\]](#page-12-",
"Figure 3: Illustration of image reconstruction (a) and generation (b) with the TiTok framework (c). TiTok contains an encoder Enc, a quantizer Quant, and a decoder Dec. Image patch",
"Figure 4: Preliminary experimental results with different TiTok variants. We provide a comprehensive exploration in (a) ImageNet-1K reconstruction. (b) ImageNet-1K linear probing. ",
"Figure 5: Visualization of generated images from TiTok variants with MaskGIT [\\[9\\]](#page-10-8). Corresponding ImageNet class names are shown below the images.",
"Figure 6: Visualization of generated images from TiTok-L-32 with MaskGIT [\\[9\\]](#page-10-8) across random ImageNet classes."
],
"tables": [
"Table 1: ImageNet-1K 256 \u00d7 256 generation results evaluated with ADM [\\[16\\]](#page-10-2). \u2020: Trained on OpenImages [\\[35\\]](#page-11-20) \u2021: Trained on OpenImages, LAION-Aesthetics",
"Table 2: ImageNet-1K 512 \u00d7 512 generation results evaluated with ADM [\\[16\\]](#page-10-2). \u2021: Trained on OpenImages, LAION-Aesthetics and LAION-Humans [\\[56\\]](#page-12-20). P: gen",
"Table 3: Ablation study improved final models for main experiments. We ablate the tokenizer designs, and generator designs on ImageNet-1k benchmark. The final settings are labeled ",
"Table 4: Detailed results of preliminary experiments in main paper."
],
"coverage": 5
},
{
"id": "2410.24221",
"slug": "egomimic-scaling-imitation-learning-via-egocentric-video",
"backend": "firecrawl",
"chars": 62595,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "End-to-end imitation learning has shown remarkable performance in learning complex manipulation tasks, but it remains brittle when facing new scenarios and tasks. Drawing on the recent success of Computer Vision and Natural Language Processing, we hypothesize that for learned policies to achieve broad generalization, we must dramatically scale up the training data size. While these adjacent domains benefit from Internet-sourced data, robotics lacks such an equivalent. To scale up data\u2026"
},
{
"n": 2,
"title": "What came before",
"body": "Imitation Learning: Imitation Learning (IL) has been used to perform diverse and contact-rich manipulation tasks , , . Recent advancements in IL have led to the development of pixel-to-action IL models, which directly map raw visual inputs to low-level robot control , . These visual IL models have demonstrated impressive reactive policies ,"
},
{
"n": 3,
"title": "The method",
"body": ""
},
{
"n": 4,
"title": "What they measured",
"body": "H1: EgoMimic is able to leverage human embodiment data to boost in-domain performance for complex manipulation tasks. H2: Human data helps EgoMimic generalize to new objects and scenes. H3: Given sufficient initial robot data, it is more valuable to collect additional human data than additional robot data. We select a set of long-horizon real world tasks to evaluate our"
},
{
"n": 5,
"title": "Where it breaks",
"body": ""
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Learns robot policies from egocentric human video via latent action inference + temporal alignment."
}
],
"takeaway_source": "club",
"figures": [],
"tables": [],
"coverage": 4
},
{
"id": "2505.11709",
"slug": "egodex-learning-dexterous-manipulation-from-large-scale-egocentric-vid",
"backend": "firecrawl",
"chars": 64961,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "The \"bitter lesson\" \\(Sutton, 2019\\) of recent breakthroughs in large language models and large vision models is that the simple recipe of supervised learning with vast amounts of data is far more effective than competing approaches. Two key challenges have prevented the application of the bitter lesson to the longstanding challenge of autonomous robot manipulation: (1) it is unclear what data should be collected, and (2) it is unclear how such data can be collected at the requisite scale. The leading approach to\u2026"
},
{
"n": 2,
"title": "What came before",
"body": ""
},
{
"n": 3,
"title": "The method",
"body": "Recent work identifies the data scarcity problem in robot imitation learning and proposes innovative techniques for scalable data collection. Chi et al. \\(2024\\) propose the \"universal manipulation interface\": handheld grippers that enable human teachers to provide demonstrations without physical robots. Wang et al. \\(2024\\) introduces a portable data collection system with motion capture gloves. Others propose collecting robot-free demonstrations by simulating robot hardware in augmented reality \\(Chen et al.,\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "We propose two benchmark tasks for EgoDex. The first is *dexterous trajectory prediction*: from the egocentric image observations, skeletal joint poses, and natural language description, the task is to predict the trajectories of the hands for a given time horizon following the observations. Specifically, we seek to train the following estimator: where o0..t are the egocentric image observations up to and including time t, s0..t are skeletal pose observations up to and including time t, l is a natural language\u2026"
},
{
"n": 5,
"title": "Where it breaks",
"body": ""
},
{
"n": 6,
"title": "One-line takeaway",
"body": "This work uses Apple Vision Pro to collect EgoDex: the largest and most diverse dataset of dexterous human manipulation to date and train and systematically evaluate imitation learning policies for hand trajectory prediction on the dataset, introducing metrics and benchmarks for measuring progress in this increasingly important area."
}
],
"takeaway_source": "semantic-scholar",
"figures": [
"Figure 1: EgoDex is a large-scale egocentric dataset that focuses on human dexterous manipulation.",
"Figure 2: Distribution of EgoDex dataset. **Top:** Distribution of distinct verbs, sorted by frequency. The horizontal axis is verbs of EgoDex. The orange plot is taken from DROID ",
"Figure 3: Left: Joints captured by EgoDex. Right: Examples of dexterous manipulation behaviors. Tracked fingertips are highlighted in distinct colors and show 0.5 seconds of motion",
"Figure 4: Model prediction visualizations for Dec + BC on test set images with a 2 second horizon. Blue trajectories are ground truth and red trajectories are predictions, where da",
"Figure 5: Distance metrics w.r.t. training dataset size, where size is plotted on a log-scale. Performance improves as the dataset gets larger.",
"Figure 6: Some of the objects used in the various manipulation tasks."
],
"tables": [
"Table 1: Comparison of different robot manipulation datasets (above the middle line) and human manipulation datasets (below the middle line). Ego4D (HOI) considers the subset of Eg",
"Table 1. #### 3.1 DATA COLLECTION",
"Table 2: Evaluations for different models on trajectory prediction with a 2 second horizon.",
"Table 3: Results for models trained and evaluated with different prediction horizons. As expected, accuracy falls as the prediction horizon increases. H = 60 values are repeated fr"
],
"coverage": 4
},
{
"id": "2505.18472",
"slug": "manifeel-benchmarking-and-understanding-visuotactile-manipulation-poli",
"backend": "firecrawl",
"chars": 110550,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "In recent years, supervised policy learning has made significant strides in robot manipulation, where visual input is used to generate actions for long-horizon and dexterous manipulation tasks [\\[1\\]](#page-16-0)\u2013[\\[4\\]](#page-16-1). However, vision-based policies face notable limitations not only in environments where visual cues are absent or severely degraded, but also in manipulation scenarios that inherently demand precise contact interactions. In cluttered or confined spaces, under low-light conditions, or\u2026"
},
{
"n": 2,
"title": "What came before",
"body": ""
},
{
"n": 3,
"title": "The method",
"body": ""
},
{
"n": 4,
"title": "What they measured",
"body": "This section introduces the task suite and modular pipeline design of the ManiFeel benchmark. ManiFeel features a diverse set of manipulation tasks spanning clear-vision conditions to severely degraded or fully occluded visual settings (Section III-A\\). The benchmark includes three main task categories: *Insertion*, *Screwing*, and *Exploration*. This results in a total of 13 different task setups, including 9 simulation setups and 4 real-world setups, as shown in Fig."
},
{
"n": 5,
"title": "Where it breaks",
"body": "Our results yield several key insights that directly address the research questions posed in this study. First, across both simulation and real-world experiments, tactile sensing consistently improves policy performance in contact-rich manipulation and in scenarios where visual input is limited or ambiguous. These results confirm that tactile sensing provides complementary feedback that is essential for robust control in contact-dominated or visually uncertain"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Website: https://zhengtongxu.github.io/manifeel-website/"
}
],
"takeaway_source": "club",
"figures": [],
"tables": [],
"coverage": 4
},
{
"id": "2505.20290",
"slug": "egozero-robot-learning-from-smart-glasses",
"backend": "firecrawl",
"chars": 54532,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Robots face significant challenges in replicating human generality and dexterity in the physical world. While deep learning has fueled progress in domains like language \\[1, [2\\]](#page-10-0), vision \\[3, 4, 5, 6, [7\\]](#page-10-0), \u2217Correspondence to Vincent Liu: vincent.liu15@gmail.com speech \\[8, 9, [10\\]](#page-10-0), and complex games \\[11, [12\\]](#page-10-0), these successes rely on internet-scale datasets that are tightly aligned with downstream applications. In robotics, collecting similarly\u2026"
},
{
"n": 2,
"title": "What came before",
"body": "Imitation learning has emerged as a powerful paradigm in robotics, enabling robots to acquire complex skills by learning directly from real-world demonstrations [\\[28\\]](#page-12-0). By observing and replicating expert behavior, robots can bypass the need for hand-engineered solutions to manipulation tasks, making this approach particularly conducive to domains with high-dimensional state and action spaces \\[29, [30\\]](#page-12-0). Teleoperation is one of the most widely used methods for imitation learning from\u2026"
},
{
"n": 3,
"title": "The method",
"body": ""
},
{
"n": 4,
"title": "What they measured",
"body": "In this section, we compare EGOZERO with baselines adapted from related works and ablate some of EGOZERO's core components. From these comparisons, we demonstrate how our specific design choices make zero-shot in-the-wild transfer possible. We also explore the generalization properties that emerge from EGOZERO's unified state-action representation space. We evaluate EGOZERO on a Franka Panda gripper"
},
{
"n": 5,
"title": "Where it breaks",
"body": "The largest source of error during inference comes from the correspondence model DIFT [\\[61\\]](#page-14-0). Correspondence encodes pose by ordering the state space, making policy learning sample efficient \\[27, [16\\]](#page-11-0). At larger data scale, pose information can be learned directly from dense unordered geometric information (i.e. using grounded segmentation models"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "EgoZero is a minimal system that learns robust manipulation policies from human demonstrations captured with Project Aria smart glasses, and suggests that in-the-wild human data can serve as a scalable foundation for real-world robot learning."
}
],
"takeaway_source": "semantic-scholar",
"figures": [
"Figure 1: EGOZERO trains policies in a unified state-action space defined as egocentric 3D points. Unlike previous methods which leverage multi-camera calibration and depth sensors",
"Figure 2: Our 7 tasks. Top: open oven door, put bread on plate, sweep board with broom, erase board. Bottom: sort fruit, fold towel, and insert book in shelf. See Appendix [A](#pag",
"Figure 3: Distribution of bread keypoints for \"Put bread in plate\" task. The columns are projections of the 3D space onto each 2D plane. The policy generalizes to object poses far ",
"Figure 4: Object semantic generalization. Human demonstrations are done with only black ovens (top). The policy transfers zero-shot to the robot with the same oven (middle) and als",
"Figure 6: Put bread on plate.",
"Figure 7: Sweep board with broom."
],
"tables": [
"Table 1: Success rates for all baselines and ablations. All models were trained on the same 100 demonstrations per task, and evaluated on zero-shot object poses (unseen from traini"
],
"coverage": 5
},
{
"id": "2505.21864",
"slug": "dexumi-using-human-hand-as-the-universal-manipulation-interface-for-de",
"backend": "firecrawl",
"chars": 75104,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Human hands are incredibly dexterous in a wide range of tasks. Dexterous robot hands are designed with the hope of replicating this capability. However, it remains a significant challenge to transfer skills from human hands to robotic counterparts due to their substantial *embodiment"
},
{
"n": 2,
"title": "What came before",
"body": "Although extensive work has studied how to enable learning in simulated environments \\[6\u2013[20\\]](#page-11-0), we focus on reviewing real world data collection methods. Teleoperation: Teleoperation is a popular interface for dexterous manipulation. Hand control is achieved with motion capture gloves [\\[21\u201325\\]](#page-11-0), virtual-reality devices \\[26[\u201328\\]](#page-12-0), or camera-based tracking"
},
{
"n": 3,
"title": "The method",
"body": ""
},
{
"n": 4,
"title": "What they measured",
"body": "Target robot hands: We evaluate DexUMI across two different robot hands: - *Inspire Hand (IHand):* A twelve-DoF (six active DoFs) underactuated hand. The thumb contains three DoFs, the index finger has three DoFs, and each of the remaining fingers has two DoFs. Tasks: We evaluate DexUMI across four different real-world tasks: - Cube [IHand]: Pick up a 2.5cm wide cube from a table and place it into a cup. The main challenge is to stably operate the deformable tweezers with multifinger contacts. - Kitchen [XHand]:\u2026"
},
{
"n": 5,
"title": "Where it breaks",
"body": "We would like to discuss DexUMI's limitations from three different aspects: hardware adaptation, software adaptation, and existing robot hand"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Learns view-invariant action representations by jointly leveraging egocentric and exocentric video data."
}
],
"takeaway_source": "club",
"figures": [
"Figure 1: DexUMI transfer dexterous human manipulation skills to various robot hand by using wearable exoskeletons and a data processing framework. We demonstrate DexUMI's capabili",
"Figure 2: Exoskeleton Design. The optimized exoskeleton design shares the same joint-to-fingertip position mapping as the target robot hand while maintaining the wearability. The e",
"Figure 3: Mechanism Optimization. To avoid thumb collision between human hand and exoskeleton, the hardware optimization step allows us to move the exoskeleton thumb backward while",
"Figure 4: Bridging the Visual Gap. To convert the visual observation into policy training data, we first segment the exoskeleton using SAM2 (b) and inpaint the missing background (",
"Figure 5: Policy Rollout: We evaluate DexUMI's capabilities across challenging real-world tasks. The Cube task tests basic picking precision. The Egg Carton task evaluates multi-fi",
"Figure 6: Comparisons. a) The policy outputs relative hand actions yield more precise action and demonstrate better multi-finger coordination. Note, we draw a sketch for the knob c"
],
"tables": [
"Table 1: Evaluation Results. We report stage-wise accumualted success rate. The experiments compare different combinations of finger action representation (Absolute vs Relative), t"
],
"coverage": 5
},
{
"id": "2507.12440",
"slug": "egovla-learning-vision-language-action-models-from-egocentric-human-vi",
"backend": "firecrawl",
"chars": 76513,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "There has been a vast advancement in robotic manipulation in the last few years, thanks to large-scale real robot data collection \\[1, [2\\]](#page-8-0). Compared to approaches that leverage simulation, directly performing supervised learning with real robot data avoids the Sim2Real domain gap and easily increases the task complexity. To efficiently collect complex robot manipulation data, multiple teleoperation tools with joint mapping \\[3, 4, [5\\]](#page-8-0), exoskeleton \\[6, 7, [8\\]](#page-8-0), and VR devices\u2026"
},
{
"n": 2,
"title": "What came before",
"body": "Dexterous Manipulation: Research in dexterous manipulation has progressed from control-based methods \\[19, 20, 21, 22, 23, 24, [25\\]](#page-9-0) to learning-driven approaches \\[26, [27\\]](#page-9-0). While early work emphasized precision, generalization across diverse scenarios remained limited. Learning-based methods introduced pose vector generation \\[28, 29, [30\\]](#page-10-0), intermediate representations \\[31, [32\\]](#page-10-0), and contact maps \\[33, [34\\]](#page-10-0), but large-scale dexterous\u2026"
},
{
"n": 3,
"title": "The method",
"body": "We build EgoVLA on top of a vision-language model to leverage strong visual and semantic reasoning. Specifically, we use *NVILA-2B*[\\[67\\]](#page-12-0) as the backbone for its robust vision-language understanding and compact size, enabling both intention inference and efficient fine-tuning. As shown in Fig.2, EgoVLA takes as input current and historical egocentric visual observations, language instructions, action query tokens, and human proprioception. These inputs are encoded by the VLM backbone and further\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "Beyond data scarcity, a major challenge in learning-based robotics is the lack of scalable, robust, and reproducible evaluation. Real-world evaluation is often costly, time-consuming, and raises concerns around safety and reproducibility\u2014barriers that disproportionately affect resource-constrained settings such as academic labs. Recent work [\\[69\\]](#page-12-0) has shown that simulation-based evaluations are highly correlated with real-world performance, supporting their use as a reliable proxy. To enable\u2026"
},
{
"n": 5,
"title": "Where it breaks",
"body": "Our pretraining framework requires human data with hand and wrist pose annotations, which may limit data availability. However, the increasing accessibility of high-fidelity AR/VR devices (e.g., Quest 3, Vision Pro, Aria Glasses) is expected to ease this constraint. Additionally, although EgoVLA is pretrained with a unified action space, it cannot be directly deployed for manipulation without further fine-tuning on a moderate amount of robot"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "This paper explores training Vision-Language-Action (VLA) models using egocentric human videos using EgoVLA and fine-tune and evaluate EgoVLA with Ego Humanoid Manipulation Benchmark and show significant improvements over baselines and ablate the importance of human data."
}
],
"takeaway_source": "semantic-scholar",
"figures": [
"Figure 1: EgoVLA. Our vision-language-action model learns manipulation skills from egocentric human videos and transfers them to a bimanual humanoid robot. The top row illustrates ",
"Figure 2: EgoVLA takes visual history, language instruction, and action query token as input. The latent features are converted to human action with the action head. We use the wri",
"Figure 3: Human Data Following insights from language model and vision-language model training, we emphasize the importance of dataset structure in driving model performance. We co",
"Figure 4: Unified Action Space: MANO hand parameters are used as a shared action space for humans and robots. For robot hands, during training, optimized mano parameters produce th",
"Figure 5: Task Visualization. All simulated tasks with predicted wrist trajs from EgoVLA.",
"Figure 6: Visual Instruction Following. Top: original HOI4D samples with corresponding language instructions. Red lines indicate ground-truth human wrist trajectories, and green li"
],
"tables": [
"Table 1: Evaluation on Short-Horizon Tasks on Seen/Unseen Visual Configurations",
"Table 2: Evaluation on Long-Horizon Tasks on Seen/Unseen Visual Configurations.",
"Table 3: Hyper Parameters for Pretraining on Human Videos",
"Table 3: Hyper Parameters Hyper Parameter Name On | for Pretraining Pretraining Human Video Data | on Human Videos Post-Training On Robot Demonstrations |"
],
"coverage": 6
},
{
"id": "2507.19468",
"slug": "back-to-the-features-dino-as-a-foundation-for-video-world-models",
"backend": "firecrawl",
"chars": 100063,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "In 2018, Ha and Schmidhuber [\\[1\\]](#page-9-0) popularized the concept of a *world model*, a neural network that predicts the future state of an environment given past observations and actions taken by an agent. Recently, the subject of world models has gained traction \\[2[\u20139\\]](#page-9-2), with conditional generative models showing impressive results on specialized domains such as driving \\[2, 7, [9\\]](#page-9-2), or video games \\[5, [6\\]](#page-9-5). Likewise, large-scale generative video models with other kinds\u2026"
},
{
"n": 2,
"title": "What came before",
"body": "World models are an active area of research. This section attempts to clarify definitions and organize the heterogeneous landscape of world models. At a high level, we define world models as models capable of predicting the temporal evolution of an environment given past visual observations and an optional conditioning"
},
{
"n": 3,
"title": "The method",
"body": "Federico Baldassarre Marc Szafraniec Basile Terver Vasil Khalidov Francisco Massa Yann LeCun Patrick Labatut Maximilian Seitzer Piotr Bojanowski Meta FAIR We wish to train a world model capable of understanding the temporal dynamics of real-world videos. Figure 1 outlines the main components of our method, a *frame encoder* and a *future predictor*. In Section 3.1, we introduce notation for the observation space, *i.e*. video pixels, and we identify the state representation to be modeled, namely patch features in\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "In this section, we empirically validate the quality of our world model. In Section 4.1, we evaluate the latent predictions of an unconditional model on dense feature forecasting tasks and on three physics understanding benchmark. Then, in Section 4.2, we analyze the effect of the different components of our model. Finally, Section 4.3 demonstrates fine-tuning the world model on agent trajectories and applying it to planning on three simulated RL"
},
{
"n": 5,
"title": "Where it breaks",
"body": ""
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Keynote 2 by by Arjun Subramaniam (Factory Intelligence)"
}
],
"takeaway_source": "club",
"figures": [
"Figure 1: Latent video world model architecture. A frozen DINOv2 *encoder* maps video frames to patch tokens in latent space. The *predictor* is a stack of cross-attention blocks t",
"Figure 2: Autoregressive predictions. For each video, from top to bottom: frames with timestamps, encoder features, autoregressive predictions in latent space. The predictor has ac",
"Figure 3: How far can the model predict? Cityscapes segmentation forecasting performance as context frames are progressively shifted back in time, further away from the target fram",
"Figure 4: Pre-training dataset statistics. For our 66M video dataset, we report the joint histogram of height *vs*. width with highlighted aspect ratios 16:9, 1:1, and 9:16, as wel",
"Figure 5: Unconditional autoregressive rollouts in latent space. For each clip, we feed the model a few initial frames, either 4 or 6, as processed by the encoder. We then roll out",
"Figure 6: Visualization of cross attentions. We visualize the cross-attention of a single query, to all patches of all previous frames, for two intermediate blocks of the predictor"
],
"tables": [
"Table 1.](#page-5-0) In this work, we choose to decouple visual representation learning from world modeling and benefit from pre-trained foundation models.",
"Table 1: Dense forecasting. For each method, we train a \"present-time\" linear head for segmentation or depth estimation, which we then apply to the output of the world model at a p",
"Table 2: Intuitive physics benchmarks. For each dataset, we identify physically implausible videos by tracking the amount of \"surprise\", *i.e*. the difference between encoded and p",
"Table 3: Ablation studies. For each setup, we report physical understanding on IntPhys, and mid-term segmentation on Cityscapes and VSPW. Predictor size (left): We train a base (86"
],
"coverage": 5
},
{
"id": "2508.10104",
"slug": "dinov3",
"backend": "firecrawl",
"chars": 267284,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Foundation models have become a central building block in modern computer vision, enabling broad gen- eralization across tasks and domains through a single, reusable model. Self-supervised learning (SSL) is a powerful approach for training such models, by learning directly from raw pixel data and leveraging the nat- ural co-occurrences of patterns in images. Unlike weakly and fully supervised pretraining methods (Radford et al., 2021; Dehghani et al., 2023; Bolya et al., 2025) which require images paired with\u2026"
},
{
"n": 2,
"title": "What came before",
"body": "Self-Supervised Learning Learning without annotations requires an artificial learning task that pro- vides supervision in lieu for training. The art and challenge of SSL lies in carefully designing these so-called pre-text tasks in order to learn powerful representations for downstream tasks. The language domain, by its discrete nature, offers straightforward ways to set up such tasks, which led to many successful unsu- pervised pre-training approaches for text"
},
{
"n": 3,
"title": "The method",
"body": "A Family of Models for Multiple Use-Cases We perform knowledge distillation of the ViT-7B model into smaller Vision Transformer variants (ViT-S, ViT-B, and ViT-L), which are highly valued by the com- munity for their improved manageability and efficiency. Our distillation approach uses the same training objective as in the first training phase, ensuring consistency in learning signals. However, instead of relying on an exponential moving average (EMA) of model weights, we use the 7B model directly as the teacher\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "In this section, we evaluate our flagship DINOv3 7B model on a variety of computer vision tasks. Throughout our experiments, unless otherwise specified, we keep DINOv3 frozen and solely use its representations. We demonstrate that with DINOv3, finetuning is not necessary to obtain strong performance. We first probe the quality of DINOv3\u2019s dense (Sec. 6.1) and global (Sec. 6.2) image representations using lightweight evaluation protocols and compare it to the strongest available vision"
},
{
"n": 5,
"title": "Where it breaks",
"body": ""
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Keynote 2 by by Arjun Subramaniam (Factory Intelligence)"
}
],
"takeaway_source": "club",
"figures": [
"Figure 1: (a) Evolution of linear probing results on ImageNet1k (IN1k) over the years, comparing fully-",
"Figure 2: Performance of the DINOv3 family of models, compared to other families of self- or weakly-",
"Figure 3: High-resolution dense features. We visualize the cosine similarity maps obtained with DI-",
"Figure 4: DINOv3 at very high resolution. We visualize dense features of DINOv3 by mapping the first",
"Figure 5: Evolution of the cosine similarities (a) and of the accuracy on ImageNet1k linear (IN1k) and",
"Figure 6: Evolution of the cosine similarity between the patch noted in red and all other patches. As training"
],
"tables": [
"Table 1: Influence of training data on features quality shown via performance on downstream tasks. We",
"Table 2: Comparison of the teacher architectures used in DINOv2 and DINOv3 models. We keep the model",
"Table 3: Dense linear probing results on semantic segmentation and monocular depth estimation with frozen",
"Table 4: Evaluation of 3D consistency of dense representations. We estimate 3D keypoint correspondences"
],
"coverage": 5
},
{
"id": "2509.04443",
"slug": "emma-scaling-mobile-manipulation-via-egocentric-human-data",
"backend": "firecrawl",
"chars": 46737,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "MOBILE manipulation has emerged as one of the most challenging problems in robotics due to the dual demands of navigation and manipulation. While recent advances in robot policy learning have demonstrated impressive capabilities in static manipulation, extending these successes Manuscript received: August, 3, 2025; Revised November, 3, 2025; Accepted December, 12, 2025. The primary obstacle is data scarcity; current approaches tackling mobile manipulation rely on teleoperation frameworks akin to Mobile ALOHA\u2026"
},
{
"n": 2,
"title": "What came before",
"body": "Behavior Cloning (BC) has emerged as an effective approach for robot learning, where policies are trained with direct supervised learning from expert demonstrations. Recent advances have shown remarkable results [\\[10\\]](#page-7-9), [\\[11\\]](#page-7-10), [\\[12\\]](#page-7-11), [\\[13\\]](#page-7-12), [\\[14\\]](#page-7-13), [\\[15\\]](#page-7-14), including the promise of building general-purpose policies by learning from large-scale datasets [\\[11\\]](#page-7-10), [\\[15\\]](#page-7-14), [\\[16\\]](#page-7-15). In\u2026"
},
{
"n": 3,
"title": "The method",
"body": ""
},
{
"n": 4,
"title": "What they measured",
"body": "H1: EMMA can achieve performance comparable to systems trained on teleoperated mobile manipulation data. H2: Key design decisions of EMMA improve downstream task performance and robustness. H3: Given an initial amount of static robot manipulation data, it is more valuable to collect additional human mobile manipulation over mobile robot teleoperation data. We evaluate these hypotheses through four long horizon mobile manipulation tasks (Fig. 4\\). *Table Service.* Two tables are set 2m"
},
{
"n": 5,
"title": "Where it breaks",
"body": "Despite the strengths of EMMA, our approach inherits several inherent limitations from learning mobile manipulation primarily through human demonstrations. First, the framework assumes that the visual and spatial distributions encountered during robot deployment lie within, or close to, those seen in human demonstrations. This assumption may break down when the kinematic or viewpoint differences between human and robot embodiments become"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Bridges humans and robots through shared latent action representations using contrastive alignment and cycle-consistency objectives."
}
],
"takeaway_source": "club",
"figures": [],
"tables": [],
"coverage": 5
},
{
"id": "2509.19626",
"slug": "egobridge-domain-adaptation-for-generalizable-imitation-from-egocentri",
"backend": "firecrawl",
"chars": 87440,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Supervised imitation learning methods such as behavior cloning have emerged as a promising path to scaling robot performance across diverse objects, tasks, and environments. However, while large-scale models in vision and language have achieved remarkable generalization through Internet-sourced data, replicating this success in robotics remains challenging due to the labor-intensive nature of collecting teleoperated demonstrations. Deploying physical robots to many new environments to collect data with enough\u2026"
},
{
"n": 2,
"title": "What came before",
"body": "Supervised Imitation Learning (SIL), notably Behavior Cloning, leverages expert demonstrations for policy learning and has achieved significant success in robotics, particularly with large-scale datasets \\[5, 6, 7, 8, 9, 10, 11, 12, 13, [14\\]](#page-10-2). State-of-the-art Vision-Language-Action (VLA) models \\[10, 8, 12, [11\\]](#page-9-10) integrate Vision-Language Models (VLMs) with action decoders, enhancing generalization by incorporating semantic understanding from internetscale data. Despite these advances,\u2026"
},
{
"n": 3,
"title": "The method",
"body": "The following section describes how f\u03d5 and \u03c0\u03b8 are parameterized in the simulation and real-world experiments. All hyperparameters are summarized in Table 3, Table 4 for real-world and in Table 5 for"
},
{
"n": 4,
"title": "What they measured",
"body": "In this section, we aim to validate three core hypotheses. H1: EgoBridge improves co-training performance for scenarios present in both human and robot data. H2: EgoBridge enables generalization to scenarios only seen in human data. H3: EgoBridge learns a shared latent space where human and robot data are aligned in task-relevant"
},
{
"n": 5,
"title": "Where it breaks",
"body": "We presented EgoBridge, a novel co-training framework designed to enable robots to learn effectively from egocentric human data by explicitly addressing domain gaps. By leveraging Optimal Transport on joint policy latent feature-action distributions, guided by Dynamic Time Warping cost on action trajectories, EgoBridge successfully aligns human and robot representations while preserving critical action-relevant information. Our experiments demonstrated significant improvements in real-world task success rates (up\u2026"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Pretrains Transformer robot policies on large-scale human behavioral data, then adapts to downstream robotic tasks with low-shot finetuning."
}
],
"takeaway_source": "club",
"figures": [
"Figure 1: EgoBridge enables rich knowledge transfer from human to robot, based on our key hypothesis: aligned latent representations yield stronger transfer. Our algorithm, which a",
"Figure 2: EgoBridge policy co-training with joint adaptation. The encoder f\u03d5 consists of modalityspecific input stems and the encoder trunk, while the policy \u03c0\u03b816,17, where the network is trained to restore the a data point given a corrupted version . The signal level determines the mixture of noise and data and is randomized during training, where corresponds to pure noise and means clean data. We build on the flow matching formulation18,19 because of its simplicity, where the network predicts the velocity vector that points towards the clean data: The signal level is typically sampled\u2026"
},
{
"n": 3,
"title": "The method",
"body": "Danijar Hafner\\* Wilson Yan\\* Timothy Lillicrap **World models learn general knowledge from videos and simulate experience for training behaviors in imagination, offering a path towards intelligent agents. However, previous world models have been unable to accurately predict object interactions in complex environments. We introduce Dreamer 4, a scalable agent that learns to solve control tasks by reinforcement learning inside of a fast and accurate world model. In the complex video game Minecraft, the world model\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "We perform a wide range of experiments to evaluate and explore the capabilities of Dreamer 4. The majority of our experiments focus on Minecraft, a complex video game that features infinite open worlds including monsters and hundreds of items that can be mined or crafted, with raw pixel observations and low-level mouse and keyboard actions. We primarily use the VPT dataset15 that contains 2541 hours of contractor gameplay with 360p video and mouse and keyboard actions at 20 FPS. To improve generations\u2026"
},
{
"n": 5,
"title": "Where it breaks",
"body": "We present Dreamer 4, a scalable agent that learns to solve challenging control tasks by imagination training inside of a fast and accurate world model. Dreamer 4 is the first agent to obtain diamonds in Minecraft purely from offline data, without online interaction. This achievement demonstrates its learning successful long-horizon strategies in complex"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Replaces RSSM with diffusion/transformer-based world models"
}
],
"takeaway_source": "club",
"figures": [
"Figure 1:** Dreamer 4 learns to solve complex control tasks by reinforcement learning inside of its world model. We decode the imagined training sequences for visualization, showin",
"Figure 2:** World model design. Dreamer 4 consists of a causal tokenizer and an interactive dynamics model, which both use the same block-causal transformer architecture. The token",
"Figure 2.](#page-3-0) The tokenizer compresses video frames into continuous representations and the dynamics model predicts the representations given interleaved actions, both usin",
"Figure 3:** Agent performance in Minecraft without environment interaction. All methods have access to the same contractor dataset[15](#page-19-3) with image inputs and ",
"Figure 4:** Agent ablations on the offline diamond challenge. We report success rates and time needed to reach an item for four milestone items. Dreamer 4 outperforms methods based",
"Figure 5:** Human interaction. A human player counterfactually interacts with the world model in real time via mouse and keyboard to perform the same task from the same initial ima"
],
"tables": [
"Table 1:** Comparison of Minecraft world models. Dreamer 4 is the first world model to accurately simulate a wide range of object interactions and game mechanics in Minecraft. More",
"Table 5.](#page-26-2) VPT (finetuned) progresses up to sticks, which it achieves 53% of the time. It also collects a small amount of stone, iron ore, and iron ingots through edge c",
"Table 2:** Cascade of model design choices. Dreamer 4 is based on a shortcut forcing objective and an efficient transformer architecture, combining a range of known techniques to a",
"Table 3:** Comparison of experimental setups for different Minecraft agents. Dreamer 4 learns purely from offline experience and requires 100\u00d7 less data than previous keyboard and "
],
"coverage": 6
},
{
"id": "2511.16661",
"slug": "dexterity-from-smart-lenses-multi-fingered-robot-manipulation-with-in",
"backend": "firecrawl",
"chars": 55188,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "*\"The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.\"* *\u2014 Mark Weiser, 1991* Robots autonomously performing diverse manipulation tasks by watching humans go about their daily lives has been a dream in Artificial Intelligence (AI) for decades. However, this remains challenging due to the embodiment gap between humans and robots, as well as the disparity between human video views and the sensor perspectives of a\u2026"
},
{
"n": 2,
"title": "What came before",
"body": "Our aim is to develop a *simple* framework for closedloop policy learning capable of performing diverse everyday manipulation tasks with dexterous multi-fingered hands. These Fig. 4: Illustration of our overall AINA framework. On the left, we show how the data is processed: the human hand pose is extracted directly by the Aria Gen 2 glasses, and stereo depth is estimated from the surrounding SLAM camera"
},
{
"n": 3,
"title": "The method",
"body": ""
},
{
"n": 4,
"title": "What they measured",
"body": "These are collected with natural human motions and with the right hand performing the respective tasks (no additional sensors on the humans or the environments, except Aria glasses). - operation space changes? - 4) How well does AINA generalize spatially and across different"
},
{
"n": 5,
"title": "Where it breaks",
"body": "In this work, we presented AINA, a new framework that leverages capabilities of Aria Gen 2 glasses to learn pointbased multi-fingered policies from explicitly in-the-wild human demonstrations. While promising, we observe three limitations. First, our framework cannot easily integrate force feedback, since hand pose estimation alone cannot capture this information, which is often crucial for accurate dexterous manipulation \\[57, 58,"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "The proposed framework AINA enables the learning of 3D point-based policies for multi-fingered hands that are robust to background changes and can be deployed directly without requiring any robot data (including online corrections, reinforcement learning, or simulation)."
}
],
"takeaway_source": "semantic-scholar",
"figures": [],
"tables": [],
"coverage": 5
},
{
"id": "2512.22414",
"slug": "emergence-of-human-to-robot-transfer-in-vision-language-action-models",
"backend": "firecrawl",
"chars": 64152,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Human knowledge provides the foundation to instill physical intelligence in robots. This manifests in many forms, from bootstrapping robot policies with human generated text and images via vision-language models, to mimicking human generated actions via robot teleoperation. While such techniques *indirectly* imbue the model with human experience, the right recipe to directly learn from human experience, for instance by watching a video of someone perform a task, remains an active area of research \\[9, 2, 31, 5,\u2026"
},
{
"n": 2,
"title": "What came before",
"body": "Learning manipulation from human video has received significant attention due to its potential scalability. Over the years, advances have been made to leverage this data more directly for policy learning. Early works in this field leveraged human video data to train stronger vision encoders, which can improve downstream policy learning \\[31, 30,"
},
{
"n": 3,
"title": "The method",
"body": "The x-axis represents the diversity of the pre-training robot dataset, and the yellow and blue lines show the finetuning performance with and without human embodiment data. While both increase, the gain from leveraging human data only appears beyond a certain pre-training scale. We evaluate on a suite of four generalization scenarios shown only in the human data. *Abstract*\u2014Vision-language-action (VLA) models can enable broad open world generalization, but require large and diverse datasets. It is appealing to\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "To test whether \u03c00.5 + ego can generalize to new concepts from egocentric human data, we construct a suite of \"generalization\" scenarios that have limited coverage in robot data, but present in human data. These scenarios span generalizing to new scenes, objects and tasks. We begin our study by understanding whether our recipe can enable transfer to these new settings. Then, we validate our core hypothesis, which is that this transfer is an emergent property of diverse VLA\u2026"
},
{
"n": 5,
"title": "Where it breaks",
"body": "We study the emergence of human to robot transfer in our proposed recipe \u03c00.5 +ego. We find that with limited pretraining diversity, VLAs fail to transfer knowledge from human data, but as pretraining diversity grows past a critical threshold, transfer emerges. While our recipes leverage vast datasets of robot teleoperation data in pretraining, we ultimately only use 10s of hours of human data, and this data is collected in an episodic"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Jointly optimizes robot morphology, interfaces, and teleoperation pipelines to improve scalability and reduce human demonstration cost."
}
],
"takeaway_source": "club",
"figures": [],
"tables": [],
"coverage": 6
},
{
"id": "2512.24766",
"slug": "dream2flow-bridging-video-generation-and-open-world-manipulation-with",
"backend": "firecrawl",
"chars": 70511,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Robotic manipulation in the open world could greatly benefit from visual world models that predict how an environment would evolve given an agent's interactions. Recent advances in generative video modeling have produced systems capable of zero-shot synthesizing minute-long, highfidelity clips of physical interactions in pixel space, conditioned on an unseen initial image and an open-ended task instruction . Such video models implicitly capture intuitive physics and rich priors of object properties and\u2026"
},
{
"n": 2,
"title": "What came before",
"body": ""
},
{
"n": 3,
"title": "The method",
"body": "Recent work increasingly integrates video models across robotic tasks in various ways . They can serve as auxiliary training objectives [59\u201364], as reward models [65\u2013 67], as policies , or as a simulator for the environments . Notably, predictive modeling in robotics can leverage video frame prediction as a form of world model. By simulating future visual observations, these models can enable visual planning and manipulation by anticipating how the environment will"
},
{
"n": 4,
"title": "What they measured",
"body": "We seek to answer the following research questions through our experiments: Q1: What are properties of 3D object flow when used as an interface to bridge videos and robot control? Q2: How does Dream2Flow perform compared to alternative interfaces? Q3: How effective is 3D object flow as a reward for learning sensimotor policies? Q4: How does the choice of video model affect Dream2Flow in simulation and in real-world"
},
{
"n": 5,
"title": "Where it breaks",
"body": "First, it relies on a rigid-grasp assumption for real world manipulation, limiting the types of tasks that can be performed. While this work shows that a particle dynamics model can be used for other types of tasks such as non-prehensile pushing, training and scaling a particle dynamics model for the real world is non-trivial and can be considered for future work. Another limitation is that the total processing time to get 3D object flow depending on the video generation model is between 3 and 11 minutes, which\u2026"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "This work introduces Dream2Flow, a framework that bridges video generation and robotic control through 3D object flow as an intermediate representation and enables zero-shot guidance from pre-trained video models to manipulate objects of diverse categories-including rigid, articulated, deformable, and granular."
}
],
"takeaway_source": "semantic-scholar",
"figures": [],
"tables": [],
"coverage": 5
},
{
"id": "2601.04153",
"slug": "diffusion-drf-free-rich-and-differentiable-reward-for-video-diffusion",
"backend": "firecrawl",
"chars": 78209,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Recent advances in diffusion-based text-to-video generation \\[18, 19, 25, 38, 47, [52\\]](#page-17-1) have markedly improved fidelity, temporal coherence, and prompt adherence. Beyond architecture and scaling, a second wave of progress has come from post-training, inspired by alignment practices in LLMs \\[28, 31, 34, [48\\]](#page-17-2) and text-to-image diffusion \\[7, 11, 14, [36\\]](#page-16-4). The core motivation is to use preferencedriven objectives to steer pretrained generators toward human-preferred behaviors\u2026"
},
{
"n": 2,
"title": "What came before",
"body": "Recent progress in LLM post-training \\[2, 27, [32\\]](#page-16-6) has been adapted to visual generation to further improve output qual- Comparisons across reward tuning methods. \u2713 indicates preferred. Existing approaches can generally be divided into two paradigms: reinforcement learning\u2013based optimization and differentiable reward optimization. RLbased methods optimize generators using preference supervision through policystyle"
},
{
"n": 3,
"title": "The method",
"body": "Given a text prompt c, a video diffusion model v\u03b8(zt | c) generates a video latent z0 through iterative denoising from Gaussian noise zT in latent space, where T denotes the total number of diffusion steps. Our goal is to fine-tune the pretrained model v\u03b8 using feedback derived from a reward function f(v, c), where v denotes the predicted clean video in pixel space decoded from z0. Diffusion-DRF leverages a frozen, off-the-shelf Vision-Language Model (VLM) as a\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "We apply Diffusion-DRF to the pretrained Wan2.1-1.3B-T2V [\\[38\\]](#page-16-0) with Qwen2.5-VL-7B [\\[1\\]](#page-14-2) as the default VLM. We only train the DiT and freeze other components (VAE, text-encoder, and VLM). Videos are generated at 512\u00d7288 resolution with 49 frames under 25 denoise steps during training and 30 steps at inference. AdamW \\[16, [23\\]](#page-15-12) optimizer is used with a learning rate of 1e \u22125 , 32 A100 80GB GPUs, and batch size 1 per"
},
{
"n": 5,
"title": "Where it breaks",
"body": ""
},
{
"n": 6,
"title": "One-line takeaway",
"body": "We'd also love to hear your hot takes on:"
}
],
"takeaway_source": "club",
"figures": [],
"tables": [
"Table 1: Comparisons across reward tuning methods. \u2713 indicates preferred. In \"Rich Feedback\" column, our method is marked with three \u2713 to reflect substantially denser supervision v",
"Table 2: Quantitative results on VBench-2.0. In addition to the overall score, we report representative sub-dimension metrics closely related to text-video alignment and physical f",
"Table 3: Human evaluation on text-video alignment of Diffusion-DRF vs. the base model. Win (%) Tie (%) Lose (%) 29.0 54.0 17.0 between the base model and our method by human annota",
"Table 4: Ablation studies on question sets and styles."
],
"coverage": 5
},
{
"id": "2601.04194",
"slug": "choreographing-a-world-of-dynamic-objects",
"backend": "firecrawl",
"chars": 78114,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Humans and other embodied agents live in a 4D (3D + time) world, a world composed of a diverse range of dynamic objects, *i.e*., objects that can evolve, deform, or interact with other objects. Creating 4D motions for both object deformations and interactions is crucial when building 3D world \u2217Equal contribution. \u2020Work was done when Y. Lyu was a visiting student at Stanford"
},
{
"n": 2,
"title": "What came before",
"body": "Generating 4D consistent object deformations has been a long-standing challenge in the community. Traditional approaches first determine category-specific kinematic models (*i.e*., rigging representations) \\[6, 8, 22, 43, 45, 51, 71, [88\\]](#page-11-2) and then generate motion based on them \\[20, 28, 42, 47, 52, 58, 60, 63, [77\\]](#page-11-3), which inherently limits these methods to constrained categories. Some methods \\[73, 82, [86\\]](#page-11-5) attempt to learn end-toend 4D generators from existing 4D object\u2026"
},
{
"n": 3,
"title": "The method",
"body": "Figure 2 shows an overview of our method. We iteratively optimize a 4D scene motion representation using guidance signals distilled from a video generative model. In the following section, we detail the three main components in this framework: a strategy for distillation from modern rectified flow-based video generative models (Sec. 3.2\\), a robust and general 4D scene motion representation (Sec. 3.3\\), and regularization terms to ensure stable optimization (Sec. 3.4\\). The above-mentioned 4D SDS algorithm is\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "We evaluate our proposed method on a diverse dynamic scenes featuring multiple interacting"
},
{
"n": 5,
"title": "Where it breaks",
"body": "Our failure cases mainly arise from two factors: (1) limitations of the underlying video generative model, and (2) the inability to handle objects that do not exist in the static snapshot but appear later in the motion sequence. Examples are shown in Our failure cases mainly arise from two factors: (1) limitations of the underlying video generative model, and (2) the inability to handle objects that do not exist in the static snapshot but appear later in the motion sequence. Examples are *a) Precision Manipulation:* Tasks requiring subcentimeter precision (e.g., peg insertion, plug insertion) consistently fail across all conditions. Inspection of failure traces reveals that precision tasks fail due to insufficient positional accuracy in the ReAct loop: the agent correctly identifies the insertion goal but cannot achieve sub-millimeter alignment through discrete action commands. This suggests a fundamental mismatch between deliberative reasoning (seconds-scale) and\u2026"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Github: https://github.com/robiemusketeer/faea-sim"
}
],
"takeaway_source": "club",
"figures": [],
"tables": [],
"coverage": 5
},
{
"id": "2602.06949",
"slug": "dreamdojo-a-generalist-robot-world-model-from-large-scale-human-videos",
"backend": "firecrawl",
"chars": 109850,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "World models, which predict futures based on actions, have emerged as a key component in the development of generalist robots \\(Hu et al., 2023; LeCun, 2022; Richens et al., 2025; Sutton, 1991\\). Recent advances in video generation \\(Ali et al., 2025; Wan et al., 2025\\) have driven video world models, in which future states are represented as video frames \\(Ball et al., 2025; Russell et al., 2025; Sun et al., 2025\\). However, they primarily plateau at discrete controls, while the high-dimensional action spaces for\u2026"
},
{
"n": 2,
"title": "What came before",
"body": "**World model.** World models can simulate world transitions in response to actions, which have been proven critical for developing intelligent agents \\(Alonso et al., 2024; Ha and Schmidhuber, 2018; Hafner et al., 2025; Richens et al., 2025\\). However, existing models are typically trained and evaluated in in-distribution settings, leaving it unclear whether these models can truly facilitate planning in unseen scenarios. Another thread of research focuses on world model pretraining from internet-scale videos to\u2026"
},
{
"n": 3,
"title": "The method",
"body": "Different from interactive games with discrete inputs \\(Parker-Holder et al., 2024\\), achieving genuine controllability for robot actions presents more challenges due to its high dimensionality and contact-rich nature. To realize precise action following, we propose two improvements based on the original architecture. First, instead of using the absolute robot joint poses, we transform them into relative actions by rebaselining the inputs with the pose at the beginning of each latent frame (*i.e*., every 4\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "In this section, we conduct extensive experiments to demonstrate DreamDojo's strengths. The dimension of the latent action is 32. The model has 24 encoder blocks for latent action extraction and 24 decoder blocks for forward dynamics prediction. It is trained on a data mixture of the three human video datasets, as well as our in-house robot datasets, including Unitree G1, Fourier GR-1, AgiBot, and"
},
{
"n": 5,
"title": "Where it breaks",
"body": ""
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Trains multitask generalist robot policies through sequence modeling over heterogeneous human demonstrations."
}
],
"takeaway_source": "club",
"figures": [
"Figure 1: **DreamDojo overview.** DreamDojo acquires comprehensive physical knowledge from large-scale human datasets by utilizing latent actions as unified labels. After post-trai",
"Figure 2: **Distribution analysis of DreamDojo-HV. (a)** Distribution of the scenarios and random examples from the most frequent categories. **(b)** [Left]: Distribution of subtas",
"Figure 3: **Latent action model.** [Left]: The information bottleneck design of our latent action model enforces action disentanglement, producing a continuous latent vector that r",
"Figure 4: **Benchmark visualization.** We rigorously construct six evaluation benchmarks that reflect the diverse scenarios and actions present in human datasets, while being out-o",
"Figure 5: **Downstream applications.** We show evidences that can be readily applied to benefit robot learning in policy evaluation without requiring real-world deployment, as well",
"Figure 6: **Live teleoperation.** We can teleoperate a virtual G1 robot using the PICO VR controller in real time."
],
"tables": [
"Table 1: **Scale and diversity comparison to existing large-scale datasets used by previous world models.** Our curated data mixture excels in both scale and diversity, encompassin",
"Table 2: **Effects of different action conditioning methods.** Latent action conditioning performs on par with the ideal settings in simulation quality and is the most scalable in ",
"Table 3: **Effects of using different data mixtures.** Adding more human datasets to pretraining consistently improves the performance for both out-of-distribution scenarios and co",
"Table 4: **Human preference evaluation in diverse out-of-distribution scenarios.** DreamDojo outperforms the pretrained Cosmos-Predict2.5 by a non-trivial margin. Our DreamDojo-14B"
],
"coverage": 5
},
{
"id": "2602.16710",
"slug": "egoscale-scaling-dexterous-manipulation-with-diverse-egocentric-human",
"backend": "firecrawl",
"chars": 68812,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Human behavior is one of the most scalable sources of data for learning physical intelligence. Humans routinely perform dexterous manipulation across diverse objects, environments, and task variations at a scale that far exceeds what can be collected through robot teleoperation. As robotic hardware continues to improve toward more human-like kinematics and dexterity, a natural question arises: *can human data serve as a primary training signal for dexterous robot manipulation?* Recent work shows that transfer from\u2026"
},
{
"n": 2,
"title": "What came before",
"body": "**Robot Learning from Human Data.** Human demonstrations have been widely used to scale robot learning, with early works leveraging human videos primarily for representation learning or intent inference \\[17, 36, 15, 14, [45\\]](#page-21-0). Subsequent approaches use human data to guide planning or high-level control while relying on robot demonstrations for low-level execution \\[34, 43, 44, 18, 38, [39\\]](#page-20-6). More recent methods exploit advances in egocentric sensing and 3D hand tracking to treat human\u2026"
},
{
"n": 3,
"title": "The method",
"body": "We aim to learn representations from large-scale egocentric human video that are directly useful for dexterous robot control. First, human demonstrations are noisy and lack paired robot actions. Second, human and robot embodiments differ substantially in kinematics and control interfaces. Our method (Figure 1\\) addresses these challenges through two design"
},
{
"n": 4,
"title": "What they measured",
"body": "We collect demonstrations on four bottles of different sizes, with 25 trajectories per bottle. **(Task V)** *(Syringe) Syringe Liquid Transfer.* This is the most challenging task, requiring the robot to pick up a syringe, draw liquid from tube A, inject it into tube B, and discard the syringe into a trash can. The task involves long-horizon, multi-step reasoning, precise spatial alignment for fluid extraction and injection, and dexterous manipulation of the syringe plunger. **Evaluation\u2026"
},
{
"n": 5,
"title": "Where it breaks",
"body": "Importantly, the G1 is never trained from scratch. Instead, mid-training serves to align an already learned, human-derived manipulation representation with a new embodiment. As shown in Section 3, this approach yields substantially higher performance than training directly on G1 data alone, indicating that large-scale human pretraining provides a reusable and embodiment-agnostic motor prior that can be efficiently adapted to robots with different kinematics and hand"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Constructs action-conditioned latent world models from human experience data for prediction, planning, and imagination-based control."
}
],
"takeaway_source": "club",
"figures": [
"Figure 1: **EgoScale: Two-stage human-to-robot learning framework.** A flow-based Vision-Language-Action (VLA) policy is first pretrained on 20,854 hours of egocentric human videos",
"Figure 2: **Human Data Collection and Model Architecture.** (**Left**) Aligned human-robot mid-training data are collected using the same sensing setup as the robot. Vive trackers ",
"Figure 3: **Post-Training Evaluation Tasks.** Five dexterous manipulation tasks used to evaluate post-training performance",
"Figure 4: **Main Experimental Results.** Comparison of Human Pre-train + Mid-Training, Human Pretraining, and No Pretraining across five dexterous manipulation tasks under two eval",
"Figure 5: **Scaling behavior of human pretraining.** *Left:* Human validation loss versus training steps for models pretrained with increasing amounts of egocentric human data (1k\u2013",
"Figure 6: **Aligned mid-training enables emergent one-shot transfer.** During post-training, the policy is trained on only a single robot demonstration per task, together with alig"
],
"tables": [],
"coverage": 6
},
{
"id": "2603.23481",
"slug": "vtam-video-tactile-action-models-for-complex-physical-interaction-beyo",
"backend": "firecrawl",
"chars": 61478,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Recent advances in Vision\u2013Language\u2013Action (VLA) models have enabled generalist robot control through large-scale multimodal alignment \\[3, 17, [49\\]](#page-15-0). By embedding visual observations and language instructions into a shared semantic latent space, these models can generalize across diverse manipulation tasks and environments \\[9, [30\\]](#page-14-0). However, while vision supports high-level semantic understanding and language specifies task intent, *physical interaction is fundamentally governed by\u2026"
},
{
"n": 2,
"title": "What came before",
"body": "**Vision-Language-Action Models.** VLA models have emerged as the dominant paradigm for generalist robot control, leveraging internet-scale vision\u2013language pretraining to ground natural-language instructions in visual observations and decode motor commands through a unified architecture \\[3, 4, 17, 30, [49\\]](#page-15-0). Subsequent efforts have expanded the paradigm along several axes, incorporating 3D geometric priors \\[28, [46\\]](#page-15-1), hierarchical task planning \\[1, [23\\]](#page-13-4), and predictive\u2026"
},
{
"n": 3,
"title": "The method",
"body": "We present the Video-Tactile Action Model (VTAM), a unified visuo-tactile world action model designed for contact-rich manipulation. As illustrated in Figure 2, VTAM operates by projecting both multi-view visual observations and high-resolution tactile streams (e.g., GelSight [\\[41\\]](#page-14-11)) into a shared continuous latent space via a pre-trained Variational Autoencoder (VAE). Within this space, a multi-view diffusion process employing alternating intra-view and cross-view attention jointly models the\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "We evaluate VTAM on real-world contact-rich manipulation tasks to study the effectiveness of visuo\u2013tactile world action modeling. Our experiments aim to answer the following key questions: - **Q1: Effectiveness of Visuo-Tactile World Action Modeling.** Does VTAM outperform vision-only and multimodal baselines in scenarios requiring fine-grained force modulation? - **Q2: Latent Video Fusion vs. Late-stage Injection.** Does modeling visuo-tactile dynamics within a shared video latent space offer performance\u2026"
},
{
"n": 5,
"title": "Where it breaks",
"body": "Figure 4 shows qualitative comparisons across the three tasks. We analyze the behaviors of different methods to understand how VTAM addresses the challenges in contact-rich tasks. Additional examples can be found in the Appendix. **Chip Pick-and-Place.** For the GE vision-only baseline, the main failure arises from the inability to verify successful"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "This work introduces the Video-Tactile Action Model (VTAM), a multimodal world modeling framework that incorporates tactile perception as a complementary grounding signal, and demonstrates superior performance in contact-rich manipulation."
}
],
"takeaway_source": "semantic-scholar",
"figures": [
"Figure 1:** We introduce **VTAM**, a generalist **V**ideo\u2013**T**actile **A**ction **M**odel that integrates tactile sensing into a predictive video world model. By grounding control",
"Figure 2: VTAM Overview.** A pretrained video backbone jointly models multi-view visual and tactile latents via alternating intra-view and cross-view attention. The resulting multi",
"Figure 3: Experiment setup and data acquisition.** We collect demonstrations through manual teleoperation using a visuo\u2013tactile sensing setup for contact-rich manipulation tasks su",
"Figure 4: Qualitative comparison between VTAM and baseline methods on real-world manipulation tasks.** Top: *Chip pick-and-place*. Vision-only baselines fail to determine whether t",
"Figure 5: Prediction visualization of the backbone video model.** From top to bottom: Camera-1 view, Camera-2 view, Tactile stream prediction. Ground-truth (top rows) and VTAM pred",
"Figure 6: Qualitative peeling results.** VTAM achieves an 85% success rate (17/20 trials), producing peel strips longer than 10 cm in successful runs."
],
"tables": [
"Table 1:** Overall performance comparison.",
"Table 2:** Ablation study on the Chip Pick-and-Place task (10 trials per variant)."
],
"coverage": 6
},
{
"id": "2604.07607",
"slug": "egoverse-an-egocentric-human-dataset-for-robot-learning-from-around-th",
"backend": "firecrawl",
"chars": 116531,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Recent progress in robot learning has shown that scaling data is a powerful driver of generalization \\[5, 26, 31, [39\\]](#page-10-2). Large-scale imitation learning has enabled policies to handle broader task distributions, more visual variation, and longer horizons, echoing trends seen in large vision and language models. However, unlike those domains, robot learning faces a fundamental bottleneck \u2013 collecting robot demonstrations requires physical hardware, expert teleoperation, and controlled"
},
{
"n": 2,
"title": "What came before",
"body": " Datasets of Human Activities. Large-scale human activity datasets such as Something-Something V2 [\\[19\\]](#page-9-4), Ego4D [\\[20\\]](#page-9-5), HOI4D [\\[36\\]](#page-10-6), EgoExo4D [\\[21\\]](#page-9-6), and Epic-Kitchens [\\[12\\]](#page-9-7) capture rich human behavior across diverse environments. However, they are not designed for robot"
},
{
"n": 3,
"title": "The method",
"body": "To enable joint training across diverse embodiments, we adopt an encoder\u2013decoder architecture with shallow, modality-specific stems [\\[49\\]](#page-11-10). Image observations are processed by a ResNet-18 [\\[23\\]](#page-9-15) backbone, while proprioceptive inputs are encoded with an MLP, before being tokenized into a shared space via learned query attention. A shared vision stem processes egocentric RGB observations from both human and robot embodiments, while separate stems handle robotspecific wrist cameras and\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "Evaluation is performed on four representative Flagship tasks shown in Fig. 8. We evaluate both in-domain (ID) settings, where task layouts match robot training data, and out-of-domain (OOD) settings with unseen objects and environments. For each method, we perform 20 ID and 20 OOD rollouts per task, with randomized initial conditions. Performance is measured using task-specific subtask metrics, including grasps, placements, intermediate manipulations, and full task"
},
{
"n": 5,
"title": "Where it breaks",
"body": "Our study mainly focused on human-and-robot co-training. Future work should conduct broader algorithmic exploration (e.g., pre-train and fine-tune). Moreover, the scene and demonstrator diversity experiments rely exclusively on offline"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Pretrains behavior foundation models on massive internet-scale human motion datasets for downstream robotic policy transfer."
}
],
"takeaway_source": "club",
"figures": [
"Figure 1: A color microscopy image showing the distribution of EgoVerse-I and EgoVerse-II in a colored collagen structure. The collagen is colored in the top right corner with red "
],
"tables": [],
"coverage": 6
},
{
"id": "2605.24934",
"slug": "humanego-zero-shot-robot-learning-from-minutes-of-human-egocentric-vid",
"backend": "firecrawl",
"chars": 119670,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "State-of-the-art manipulation policies require hundreds to thousands of task-specific robot demonstrations \\[1, 2, 3, 4, 5, [6\\]](#page-8-0), which are costly, time-consuming, and inconvenient to collect. Human egocentric video offers a much cheaper and more accessible alternative: with a head-mounted camera [\\[7\\]](#page-8-0), a single person can collect task demonstrations anywhere, in minutes. We pursue a more direct goal: learning deployable manipulation policies from only minutes of human egocentric\u2026"
},
{
"n": 2,
"title": "What came before",
"body": "Recent years have produced rich large-scale egocentric and hand\u2013object interaction datasets \\[34, 35, 15, 36, 37, 38, 39, [14\\]](#page-9-0) that provide the data foundation for learning manipulation from human video. Building on this foundation, one line of work scales up generalist policies and world models \\[12, 13, 40, 41, [42\\]](#page-11-0) that learn embodiment-agnostic representations from massive corpora, yet deployment demands enormous compute and per-task robot post-training. Another line cotrains \\[8, 9,\u2026"
},
{
"n": 3,
"title": "The method",
"body": "Fig. 14: Hand tracking comparison on Serve Bread (45 demonstrations, \u223c45 k frames). *Top\u2014 Smoothness:* per-frame jerk of the gripper midpoint (translational and angular) and of all 21 keypoints (lower is better, log scale). *Bottom\u2014Accuracy vs. Aria-MPS:* per-keypoint shape error after Procrustes alignment, residual rotation error after subtracting the systematic frame offset, and fraction of frames with a valid hand detection. **Monocular RGB Stereo + IMU** Fig. 15: Hand Tracking Method Study. ICT consumes 3D\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "**HumanEgo-30 HumanEgo-15 ACT (Robot Teleop) SPOT ZeroMimic Track2Act PointPolicy EgoZero** Fig. 4: Overall Real-World Evaluation. Real-world success rate (%) for each method across all four tasks. HumanEgo with 30 min of data achieves the highest success rate on every task, demonstrating consistent improvements over both human-video baselines and robot teleoperation methods. HumanEgo achieves the highest success rate on every single"
},
{
"n": 5,
"title": "Where it breaks",
"body": ""
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Code: https://github.com/TX-Leo/HumanEgo"
}
],
"takeaway_source": "club",
"figures": [],
"tables": [
"Table 1: All hyperparameters used in HumanEgo. Values are shared across the four tasks unless noted."
],
"coverage": 5
},
{
"id": "2606.30645",
"slug": "vlk-learning-humanoid-loco-manipulation-from-synthetic-interactions-in",
"backend": "firecrawl",
"chars": 68994,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Connecting egocentric observations to whole-body action is a fundamental challenge for humanoid robots operating in human-centered environments. A humanoid must perceive the world from its own viewpoint, identify task-relevant objects, navigate toward them, and physically interact with them through coordinated full-body motion. This perception-to-action loop, which humans execute effortlessly in everyday settings, requires a policy that maps high-dimensional visual input and language instructions to whole-body\u2026"
},
{
"n": 2,
"title": "What came before",
"body": "Vision-language and perception-based policies for humanoids. Vision-language-action (VLA) policies map visual observations and language instructions to robot actions and have become a promising paradigm for robot policy learning \\[6, 7, [8\\]](#page-8-0). Recent work extends this direction to humanoid robots, where policies must connect perception, language, and whole-body control \\[9, 10, 11, 12, 13, 14, 15, 16,"
},
{
"n": 3,
"title": "The method",
"body": "Our goal is to enable a Unitree G1 humanoid to perform perception-based loco-manipulation, including object-directed navigation and box interaction, from egocentric observations and task instructions. We formulate the problem as kinematic prediction followed by whole-body tracking: given the current egocentric view, instruction, and robot kinematic state, a high-level policy predicts a short-horizon G1 whole-body kinematic trajectory, and a whole-body tracker converts the predicted trajectory into executable robot\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "In this section, we introduce our hardware setup (Sec. 4.1) and then answer four research questions: (i) How efficiently can the proposed pipeline generate diverse paired VLK data? (Sec. 4.2) (ii) Can a VLK policy trained on synthesized data transfer to the physical humanoid? (Sec. 4.3\\) (iii) How does the amount of synthesized training data affect performance across different tasks? (Sec. 4.4\\) (iv) How does visual domain randomization help bridge the visual sim-to-real"
},
{
"n": 5,
"title": "Where it breaks",
"body": ""
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Video: https://youtu.be/ZB6k\\iMJP7M"
}
],
"takeaway_source": "club",
"figures": [
"Figure 1: Synthetic interactions in reconstructed scenes enable real-world perception-based locomanipulation. Synthetic data generation produces paired egocentric observations, tas",
"Figure 2: Method overview. Top: we reconstruct 3D scenes, generate task waypoints, synthesize G1 motions, and render egocentric observations to produce paired vision-language-kinem",
"Figure 3: Examples of generated VLK supervision. Each sequence pairs an egocentric RGB observation and language instruction with the corresponding G1 whole-body kinematic trajector",
"Figure 3. For more examples, please see Appendix [A.6.](#page-14-0)",
"Figure 4: Ablation of training-data volume in the lab scene. We vary the number of synthesized trajectories per mode per layout and report success rates on the lab-scene validation",
"Figure 4. As shown in the figure, increasing the amount of synthesized training data consistently improves performance across the evaluated"
],
"tables": [
"Table 1: Full-system evaluation in MuJoCo simulation (with IsaacSim rendering) and real-world deployment. We report the number of successful trials over the total number of trials ",
"Table 2: Effect of visual domain randomization on walking-mode success. Evaluation is conducted in the lab scene under visual perturbations, including lighting and camera variation",
"Table 3: Domain randomization parameters used for egocentric rendering.",
"Table 4: End-to-end VLK replan latency breakdown, averaged across one deployment. The total of \u223c 63 ms is well below the \u223c 555 ms replan period, leaving \u223c 8.8\u00d7 headroom against bac"
],
"coverage": 5
},
{
"id": "2607.28623",
"slug": "pac-man-perception-aware-cbf-rl-for-whole-body-safety-in-humanoid-dodg",
"backend": "firecrawl",
"chars": 47279,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Humanoid robots operating around moving objects must react within a fraction of a second, and the evasion itself must not cost them their balance. Dodgeball isolates this safety problem in a short-horizon, whole-body setting: the robot must perceive an incoming ball, determine which links are threatened, and coordinate the torso, arms, and legs quickly enough to avoid contact while remaining upright. Humans solve the same problem by keeping their eyes on the ball and moving their whole body out of its"
},
{
"n": 2,
"title": "What came before",
"body": "Motion imitation and priors for humanoid control. Motion imitation has long provided a practical route to naturalistic, high-dimensional control. showed that RL can track reference motions reliably, replaced explicit tracking with an adversarial discriminator over motion features, which yields a style reward compatible with a task objective, and , learned reusable skill and style embeddings over large motion datasets. These build on a broader body of legged learning \u2013 trained from task rewards, command tracking,\u2026"
},
{
"n": 3,
"title": "The method",
"body": "Balls are thrown on an intermittent timer that leaves a recovery window between throws. The default distribution launches from a frontal cone (within \u00b125\u25e6 of heading, 2-3 m ahead) with a flight time of roughly 0.6 s, which sets the reaction window; the aim point leads the robot's velocity. We mix two threat types equally so the policy learns both evasions: a descending ball that falls across the legs (sidestep/step-over) and a low-arc ball that rises to torso/head height (duck/lean). Throw heights are calibrated\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "We first evaluate the PAC-MAN framework in simulation to quantify the impact of perception and safety structure; then we verify the full stack on the physical robot. Figure 4 shows emergent evasion modes from both. In simulation, we compare the onboard fixed-camera policy against a gimbal-mounted camera and a privileged state oracle while varying the safety structure at each perception level, from no barrier to the full joint-space CBF with its runtime filter. All policies are evaluated\u2026"
},
{
"n": 5,
"title": "Where it breaks",
"body": "We presented PAC-MAN, a perception-aware CBF-RL framework for whole-body humanoid dodgeball, and deployed its fixed-camera Link-CBF policy on the Unitree G1 using only onboard depth and proprioception. PAC-MAN couples segmentation-masked depth with controlbarrier guidance computed during training, and a human motion prior regularizes the form of the evasive reflexes. Our results show that perception-aware CBF-RL is a safetyinformation co-design problem: the appropriate barrier structure depends on the threat\u2026"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "PAC-MAN: Perception-Aware CBF-RL for Whole-Body Safety in Humanoid Dodgeball presents a perception-aware reinforcement learning framework designed to improve the safety and robustness of humanoid robots operating in dynamic environments. The work combines Control Barrier Functions (CBFs) with reinforcement learning and realistic onboard perception, addressing a key challenge in robot learning: policies can achieve impressive performance but may behave unsafely when exposed to unexpected disturbances or imperfect observations."
}
],
"takeaway_source": "club",
"figures": [],
"tables": [],
"coverage": 6
},
{
"id": "2608.05215",
"slug": "vlaff-vision-language-affordance-model-for-unified-actionable-affordan",
"backend": "marker",
"chars": 44681,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Foundation models trained on large-scale datasets have proven effective for many downstream tasks in natural language processing and computer vision, such as LLaMA and ChatGPT . In robotics, there have been attempts to build such foundation models. However, traditional paradigms like imitation learning face challenges: collecting robot action-state datasets is costly and timeconsuming, with task and environment domains often limited to laboratory"
},
{
"n": 2,
"title": "What came before",
"body": ""
},
{
"n": 3,
"title": "The method",
"body": "Jihoon Oh1 , Kento Kawaharazuka1 , and Kei Okada1 *Abstract*\u2014 Learning manipulation skills from human videos is promising for scalable robot learning. However, the embodiment mismatch between humans and robots makes this challenging. One promising solution is to learn objectcentric actionable affordances that are embodiment-agnostic. In this work, we propose a framework that leverages egocentric human videos with state-of-the-art 3D Structure-from-Motion and hand mesh\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "We employ Qwen2.5-VL as our base vision-language model. For enhanced visual feature extraction, we integrate DINOv2 as our additional vision encoder. *b) Datasets:* Our final EgoAffordance dataset comprises 204,025 episodes, containing 5,782,431 visual heatmaps and 11,612,524 trajectory sequences. To improve fine-grained visual affordance prediction, we additionally incorporate data from HANDAL and SceneFun3D , which provide detailed object-level affordance annotations. *c) Visual Affordance:* We evaluate visual\u2026"
},
{
"n": 5,
"title": "Where it breaks",
"body": ""
},
{
"n": 6,
"title": "One-line takeaway",
"body": "This work proposes a framework that leverages egocentric human videos with state-of-the-art 3D Structure-from-Motion and hand mesh reconstruction to extract actionable affordances such as visual, grasp, and trajectory affordances that explicitly encode where to interact, how to grasp, and how to move."
}
],
"takeaway_source": "semantic-scholar",
"figures": [],
"tables": [],
"coverage": 4
},
{
"id": "2608.06919",
"slug": "vernata-self-supervised-learning-of-lidar-point-representations",
"backend": "marker",
"chars": 45552,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Autonomous robots are increasingly transitioning from research demonstrations in controlled environments to operating in the unstructured real world , . This progress is underpinned by advancements in scene understanding, which equip agents with the semantic context required to perform complex, intelligent actions in unstructured environments \u2013. LiDAR sensors are central to this advanced perception"
},
{
"n": 2,
"title": "What came before",
"body": "Motivated by findings that model performance scales effectively with data and compute \u2013, the field has increasingly adopted selfsupervised learning. Learning representations directly from raw data, SSL allows models to utilize massive unlabeled datasets for scaling . Early approaches largely relied on contrastive objectives, such as SimCLR and MoCo , which enforce feature invariance across augmented views of the same sample, while pushing apart representations from different"
},
{
"n": 3,
"title": "The method",
"body": "We present Vernata, a multi-modal, multi-teacher distillation framework designed to learn robust point representations from LiDAR data. Building upon Sonata , our approach introduces three extensions to overcome inherent density variations and limited semantic cues in outdoor LiDAR, as well as the practical bottleneck of resource-constrained training. Architecturally, Sonata adapts a DINOv2-style selfdistillation framework to the 3D domain. It employs a dual-network structure consisting of a student and a teacher\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "We evaluate our framework across two domains: unstructured field environments and urban driving. In the former, we combine the real-world GrandTour and synthetic TartanGround for self-supervised pretraining. However, as GrandTour lacks semantic annotations, we perform downstream evaluation exclusively on TartanGround. For the latter, we utilize the Waymo Open Dataset as a selfcontained benchmark, employing it for both pretraining and evaluation to validate our method on an established"
},
{
"n": 5,
"title": "Where it breaks",
"body": ""
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Vernata is introduced, consisting of three extensions: sparse view augmentation to improve robustness against varying point densities, a memory bank mechanism to stabilize resource-constrained training, and cross-modal distillation utilizing dense, high-resolution 2D image features to enable fine-grained semantic guidance."
}
],
"takeaway_source": "semantic-scholar",
"figures": [],
"tables": [],
"coverage": 5
},
{
"id": "2608.09891",
"slug": "rose-a-robotic-soft-esophagus-for-endoprosthetic-stent-testing",
"backend": "marker",
"chars": 86865,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Swallowing is a complex but orderly physiological process transporting saliva or food from the mouth to the stomach. In an esophageal peristaltic transport of food bolus, the manometry recordings can be distributed into two intraluminal pressure segments. In the first segment, within the bolus fluid, the recorded pressure is solely due to the bolus hydrodynamic pressure, which is also known as the intrabolus pressure signature"
},
{
"n": 2,
"title": "What came before",
"body": ""
},
{
"n": 3,
"title": "The method",
"body": ""
},
{
"n": 4,
"title": "What they measured",
"body": ""
},
{
"n": 5,
"title": "Where it breaks",
"body": "Palliative therapy for dysphagia caused due to benign and malignant esophageal strictures can be approached with the implantation of endoprosthetic stents.2, 8 The stenting can provide the patients with instant relief, and it can restore their oral nutrition requirements in one session (Fig. 1). However, adverse events like stent migration and various stent dysfunctionalities present vital obstacles to successful stent deployment in the esophagus.13 Due to the less known relationship between the\u2026"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "RoSE is an innovative soft robotic platform that is capable of testing various endoprosthetic stents, thereby offering a solution to many existing clinical challenges in the area of stent testing."
}
],
"takeaway_source": "semantic-scholar",
"figures": [],
"tables": [
"Table 1: | Quantitative and qualitative characteristics of RoSE |",
"Table 2. Characteristics of the starch-thickened food boluses. Thickener Concentration | Viscosity (Pa.s) |",
"Table 3. Manipulated parameters to cover a range of swallowing scenarios. | |"
],
"coverage": 3
},
{
"id": "2608.10056",
"slug": "navigating-the-proximity-safety-balance-constraint-decomposition-for-h",
"backend": "marker",
"chars": 52997,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Robots are increasingly expected to follow specific individuals in applications such as healthcare, companionship, and assistance [\\[1\\]](#page-7-0), requiring them to navigate safely among pedestrians and obstacles while maintaining close proximity to the target. This task involves an inherent conflict between proximity and safety. Unlike point-goal navigation, where the robot can freely reroute around obstacles, human following involves a moving target, and any detour to avoid collisions risks increasing the\u2026"
},
{
"n": 2,
"title": "What came before",
"body": ""
},
{
"n": 3,
"title": "The method",
"body": "An overview of our method is illustrated in Fig. 1. At each timestep *t*, the state *St* consists of the robot's physical state, local occupancy grid maps representing static obstacles, and the current positions, predicted trajectories, and quantified prediction uncertainties of nearby humans. The state is processed by a unified policy network, where CNNencoded occupancy features and human and robot tokens are combined into a sequence and processed by self-attention mechanisms [\\[26\\]](#page-7-25),\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "Traditional subgoalguided methods (SG-HA\\*, SG-ORCA, SG-MPC) exhibit substantially lower SR and higher CR and TLR compared to RL-based methods, indicating limited ability to maintain following behavior in the presence of dense, dynamic pedestrians. Among RL-based methods, our approach significantly outperforms OGM-HEIGHT, achieving a 25.76% higher SR while reducing overall CR by 18.56%, human CR by 10.72%, and obstacle CR to the lowest 5.44% among learning-based methods. These results validate the effectiveness of\u2026"
},
{
"n": 5,
"title": "Where it breaks",
"body": ""
},
{
"n": 6,
"title": "One-line takeaway",
"body": "This work decomposes the human-following task into a sparse task reward and independent cost constraints within a multi-constraint RL formulation, where each constraint is managed through cost thresholds with direct behavioral meaning rather than implicit reward weight ratios, allowing explicit and tunable control over the trade-off."
}
],
"takeaway_source": "semantic-scholar",
"figures": [],
"tables": [],
"coverage": 4
},
{
"id": "2608.10824",
"slug": "neural-introspection-gating-for-adaptive-kv-cache-reuse-in-vision-lang",
"backend": "marker",
"chars": 31369,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Vision-Language-Action (VLA) models \u2013 use a single autoregressive transformer that ingests a camera frame and a language instruction, then directly outputs discretized motor commands. Because they build on large visionlanguage models (VLMs), VLAs generalize across manipulation tasks without much task-specific engineering. However, running a 7-billion-parameter model at the 10\u201320 Hz rates needed for manipulation is computationally"
},
{
"n": 2,
"title": "What came before",
"body": "VLA \u2013, models unify perception, language understanding, and action prediction within a single transformer. RT-2 pioneered the approach by fine-tuning a VLM to output actions as text tokens. OpenVLA is a 7B-parameter model built on Prismatic VLMs that performs well on diverse manipulation"
},
{
"n": 3,
"title": "The method",
"body": ""
},
{
"n": 4,
"title": "What they measured",
"body": "We evaluate on two VLA architectures: (a) Open-VLA , a 7B-parameter VLA based on the Prismatic VLM architecture with a Llama-2 backbone, and (b) OpenVLA-OFT , which uses optimized fine-tuning with parallel action-dimension decoding for improved downstream performance. We evaluate on four LIBERO task families: *Spatial* (10 spatial relationship reasoning tasks), *Object* (10 object type identification tasks), *Goal* (10 goalconditioned tasks), and *Long* (10 long-horizon multi-step tasks). Following the standard\u2026"
},
{
"n": 5,
"title": "Where it breaks",
"body": "The pattern is simple: Gated VLA-Cache helps most on tasks where VLA-Cache causes clear accuracy degradation. On LIBERO-Goal (\u22123.2%) and LIBERO-Long (\u22123.8%), where cache staleness compounds over long manipulation sequences, our gating recovers over 100% of the lost accuracy (106% and 121%, respectively). On suites where caching is already safe (Spatial, Object), the gating rarely triggers and adds minimal"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Gated VLA-Cache is proposed, a lightweight, training-free extension that augments visual-similarity caching with neural introspection that improves reliability when blind caching hurts."
}
],
"takeaway_source": "semantic-scholar",
"figures": [],
"tables": [],
"coverage": 5
},
{
"id": "2608.11870",
"slug": "enhancing-visual-domain-robustness-in-behaviour-cloning-via-saliency-g",
"backend": "marker",
"chars": 67231,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Vision-based behaviour cloning (BC) has made significant strides in transferring behaviour from expert demonstrations to robot skills \\[1, [2\\]](#page-8-0). These demonstrations involve intricate spatial-temporal interactions between the agent and its environment and often require integration among multiple sensory modalities. However, given the high costs associated with collecting demonstration data, especially in real-world settings \\[3, [4\\]](#page-8-0), datasets often prioritise task-related variability over\u2026"
},
{
"n": 2,
"title": "What came before",
"body": "A typical vision-based BC policy \u03c0\u03be {zj} t j=t\u2212T , parameterised by \u03be, derives actions from a temporal sequence of observation embeddings z over a window of length T [\\[7\\]](#page-8-0). At time step t, the observation embedding zt is defined as: where x vi t and pt represent the visual input from the i th visual modality or camera view, and the proprioceptive state. The visual encoders f\u03b8i , parameterised by \u03b8i ,\u2026"
},
{
"n": 3,
"title": "The method",
"body": ""
},
{
"n": 4,
"title": "What they measured",
"body": "Manipulation Tasks and Out-of-Domain Settings. As illustrated in *Fig*.2, in simulations, we use expert human demonstrations from the Robomimic environment [\\[7\\]](#page-8-0) to evaluate four tasks of varying complexity and visual diversity: *Lift* (pick), *Square* (pick and insert), *Can* (pick and place), and *Transport* (dual-arm pick, handover, place). Real-world experiments focus on *Toy* (pick and place). Most tasks use a second-person and an eye-in-hand camera, except *Transport* with four cameras, all at\u2026"
},
{
"n": 5,
"title": "Where it breaks",
"body": "The implementation of a saliency buffer improves efficiency, but the computational demands remain considerable. For example, saliency computation with a ResNet18 encoder on 84\u00d784 inputs takes about 1.5 times longer than training. To improve efficiency, approximating saliency at mid-feature layers or exploring alternative saliency extractors could"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Experiments show that RoboSaGA preserves in-domain performance while substantially improving robustness to visual domain shifts, including distractor and background changes, as well as lighting and shadow variations."
}
],
"takeaway_source": "semantic-scholar",
"figures": [
"Figure 1: RoboSaGA's core components and the broadened visual domains. (a) Saliency map s, derived from visual feature output f\u03b8(x), guides the overlaying of in-domain and the OOD ",
"Figure 2: Experiment environment setups.",
"Figure 3: Augmentation with BC-MLP.",
"Figure 4: Examples of RoboSaGA against Real-World Visual Domain Shifts, including lighting changes, occlusion, object clutter, and background variations.",
"Figure 5: Saliency Maps across Two Views during different stages of task execution (BC-RNN).",
"Figure 6: Examples of out-of-domain images for data augmentation"
],
"tables": [
"Table 1: Gap (\u2193) of BC-MLP under variations in background and distractors.",
"Table 2: Gap (\u2193) across All Policies (BC-MLP, BC-RNN, Diffusion Policy) under visual domain shifts.",
"Table 3: Performance Gap (\u2193) of BC-MLP, BC-RNN, and Diffusion Policy with Random Crop, +Random Jitter, +Random Overlay, +SODA, and +RoboSaGA under visual domain shifts.",
"Table 4: Real-world Performance Gap (\u2193) across Each Policy (and Average) for Random Crop, Random Overlay, and RoboSaGA on the *Toy* task."
],
"coverage": 5
},
{
"id": "2608.16351",
"slug": "arm-aware-guided-dexterous-grasp-generation-with-arm-agnostic-grasp-mo",
"backend": "marker",
"chars": 72804,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "D EXTEROUS grasp generation, which generates grasp poses for dexterous hands based on object and environment information, provides a target grasp configuration for grasp execution [\\[1\\]](#page-7-0), [\\[2\\]](#page-7-1), and serves as a prerequisite for the subsequent robotic manipulation [\\[3\\]](#page-7-2). Existing grasp generation methods predominantly employ a *hand-centric* scheme, which focuses primarily on learning the distribution of a free-floating hand's grasp poses, overlooking the robotic arm and\u2026"
},
{
"n": 2,
"title": "What came before",
"body": ""
},
{
"n": 3,
"title": "The method",
"body": "Yongyi Jia\u2020 , Yongpeng Jiang\u2020 , Kangchen Lv, Yi Ren, Mingrui Yu\u2217 , and Xiang Li\u2217 *Abstract*\u2014Dexterous grasp generation that considers armrelated constraints is crucial in real-world scenarios involving armenvironment collision avoidance, workspace boundary grasps, and consecutive grasping. Existing hand-centric grasp models, which primarily focus on the floating hand's pose, are insufficient for such cases. Conventional arm-aware methods either rely on rejection\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "Additional evaluation results can be found in the Appendix (available on our Project Website\\). We use the Shadow Hand in simulations and the LEAP Hand for real-world evaluations, considering two common robotic arms, UR5 and Franka. We assume the environment can be represented as a combination of closed geometries, enabling a well-defined SDF. We adopt the sphere robot collision models from cuRobo [\\[29\\]](#page-7-28) for collision"
},
{
"n": 5,
"title": "Where it breaks",
"body": "We present several failure cases in Fig. 15, highlighting potential limitations of the proposed method and possible directions for improvement. 1) In some cases, the guidance compromises the quality of a small subset of generated grasps, resulting in unstable configurations that only grasp the edge of the object (Fig. 15 (a-b)). This accounts for 7 out of 20 failed grasps. This issue often arises when the arm remains in collision until the end of the denoising process, causing conflicts between constraint\u2026"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "This letter forms arm-aware constrained grasp generation as a joint optimization of hand pose and arm configuration, and derive closed-form gradients for arm-related constraints, and proves that gradient-based optimization is equivalent to guided diffusion sampling, steering near-feasible samples toward the feasible region."
}
],
"takeaway_source": "semantic-scholar",
"figures": [],
"tables": [],
"coverage": 5
},
{
"id": "2608.18317",
"slug": "reproducible-multimodal-affordance-prediction",
"backend": "marker",
"chars": 85562,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Affordances describe the potential actions that an agent can perform on objects in the environment [\\[20\\]](#page-15-0), inferred from multimodal sensory observations such as visual, depth, acoustic, and tactile data [\\[28\\]](#page-15-1). Understanding affordances enables the agent to accomplish a task, selecting which objects in the environment to interact with, what actions to perform, and how to execute them. This reasoning within and across modalities, beyond simply perceiving scene and objects, supports\u2026"
},
{
"n": 2,
"title": "What came before",
"body": ""
},
{
"n": 3,
"title": "The method",
"body": "The lack of publicly available implementation of methods (RC2) \\[22,56\u2013[58\\]](#page-17-7), the lack of publicly available trained models (RC3) \\[22, 42, 56[\u201358\\]](#page-17-7), and the lack of details of experimental setups (RC4) \\[14,22,42,56[\u201358\\]](#page-17-7) can challenge researchers in reproducing previous works for comparative evaluations. The release of the model trained weights and the implementation of the method and inference pipeline is a crucial aspect for reproducibility, especially for deep-learning\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "Table 1 reports the main characteristics of the available datasets and benchmarks in affordance prediction. None of the datasets is collected for benchmarking methods under different in-the-wild conditions (RC1), such as illumination, clutter, or hand-occlusions. Each affordance dataset is collected for a specific formulation, and not re-used across different affordance tasks, limiting fair comparison and preventing comprehensive benchmarking. One of the main obstacles to obtain large-scale data collections to\u2026"
},
{
"n": 5,
"title": "Where it breaks",
"body": ""
},
{
"n": 6,
"title": "One-line takeaway",
"body": "The Affordance Sheet is proposed, a documentation detailing task formulation with its input modalities, model architectures and training information, datasets, and experimental protocols, which enable reproducible benchmarking and reliable evaluation of affordance models for real-world scenarios, including generalisation to novel conditions and human safety."
}
],
"takeaway_source": "semantic-scholar",
"figures": [],
"tables": [
"Table 1: Characteristics of datasets for affordance prediction grouped by formulations.",
"Table 2: Comparison of training/testing setups used by different methods for affordance detection and segmentation on the UMD dataset [\\[40\\]](#page-16-3). Due to the setup inconsi",
"Table 3: Affordance Sheet, inspired by Model Cards [\\[38\\]](#page-16-7), to favour transparency and reproducibility of visual affordance prediction methods.",
"Table 4: Affordance Sheet for EgoTopo [\\[41\\]](#page-16-1)."
],
"coverage": 4
},
{
"id": "2608.18662",
"slug": "dynamic-spectraformer-for-ultra-high-definition-underwater-image-enhan",
"backend": "marker",
"chars": 43972,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "The degradation in underwater image quality hampers the visual sensing capabilities of marine robots, despite that they are equipped with high-end cameras. Thus, algorithms for Underwater Image Enhancement (UIE) play a critical role in advancing aquatic exploration, with widespread applications in domains like Autonomous Underwater Vehicles (AUVs) and Remotely Operated Vehicles (ROVs). This degradation in image quality is primarily due to the wavelengthdependent scattering and attenuation of light as it travels\u2026"
},
{
"n": 2,
"title": "What came before",
"body": ""
},
{
"n": 3,
"title": "The method",
"body": "In this section, we present the proposed Dynamic SpectraFormer in detail. After briefly introducing the overall architecture as shown in Fig. 2, we present the Dynamic SpectraFormer block. Our work aims to harness the power of DCT to boost the performance of Transformer models for high-definition underwater image enhancement tasks. Quantitative Evaluation To consolidate our performance superiority, we compare our SpectraFormer with multiple UIE methods, including Ucolor , WaterNet , UGAN , FUnIE-GAN , Deep SESR ,\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "The LSUI dataset was segmented into 4500, and 404 images for training and testing, respectively. In the evaluation stage, we also conducted assessments using the underwater image sets from UIEB (90 pairs) , LSUI (504 pairs) , and EUVP (515 pairs) , respectively. Evaluation metrics include Peak Signal Noise Ratio (PSNR) and Structural Similarity Index (SSIM), measuring the color and structural fidelity between enhanced images and ground"
},
{
"n": 5,
"title": "Where it breaks",
"body": ""
},
{
"n": 6,
"title": "One-line takeaway",
"body": "In this paper, we proposed Dynamic SpectraFormer, a highly efficient frequency domain transformer that could enhance the UHD underwater image in the frequency domain and empower it with adaptability to the contents. Extensive analysis conducted on real-world datasets, along with comprehensive ablation studies, validates the efficacy of the proposed method for UHD underwater image enhancement"
}
],
"takeaway_source": "paper",
"figures": [],
"tables": [],
"coverage": 4
},
{
"id": "2608.19955",
"slug": "mild-tractable-terrain-modeling-for-learning-improved-bipedal-locomoti",
"backend": "marker",
"chars": 48618,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "I N recent years, a notable upsurge has been witnessed in the bipedal robotics sector, as it shows great potential for application in deformable and unstructured terrain, which constitutes a significant portion of the Earth's surface. Current bipedal robot locomotion controllers are predominantly optimized for rigid ground, as mainstream simulators [\\[1\\]](#page-7-0) [\\[2\\]](#page-7-1) [\\[3\\]](#page-7-2) rely on rigid-body dynamics that cannot capture the spatiotemporal heterogeneity of deformable terrains. This\u2026"
},
{
"n": 2,
"title": "What came before",
"body": ""
},
{
"n": 3,
"title": "The method",
"body": "In the robotics physics community, exploring methods that can accurately and efficiently simulate the deformation and stress of soft substrates upon contact with robots has always Manuscript received: April 28, 2025; Revised: August 30, 2025; Accepted: December 4, 2025. This paper was recommended for publication by Editor Aleksandra Faust upon evaluation of the Associate Editor and Reviewers' comments. This work was supported by General Research Fund under Grant No. 17204222. \u2020Corresponding author lupeng@hku.hk,\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "To assess the advantages of our approach, we compare it with existing state-of-the-art methods for deformable terrain modeling. The data is collected from the policies trained with the same episodes on their respective models: *1) Implicit terrain adaptability:* Different from prior work testing on unquantified grass/sand, we evaluate system performance on standardized materials with defined stiffness grades. Specifically, we consider six distinct terrains, including four manufactured surfaces\u2014rubber tiles, 60d,\u2026"
},
{
"n": 5,
"title": "Where it breaks",
"body": ""
},
{
"n": 6,
"title": "One-line takeaway",
"body": "MILD is presented, featuring a physics-grounded discrete-element contact solver that accurately simulates spatially varying foot-terrain interactions and train a terrain-aware locomotion controller via deep reinforcement learning with latent modulation and proprioceptive estimation."
}
],
"takeaway_source": "semantic-scholar",
"figures": [],
"tables": [],
"coverage": 4
},
{
"id": "2608.23650",
"slug": "concept-guided-exploration-building-persistent-actionable-scene-graphs",
"backend": "marker",
"chars": 98613,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Robots operating in human environments benefit from semantically rich, shared representations of their surroundings. In recent years, 3D scene graphs (3DSGs) have been used for this purpose, as complex data structures that represent scene objects as nodes and their various relationships as edges Armeni et al. \\(2019\\). When used to describe large fragments of space, they are typically organised hierarchically, with higher levels encompassing larger spatial aggregations Hughes et al."
},
{
"n": 2,
"title": "What came before",
"body": ""
},
{
"n": 3,
"title": "The method",
"body": "CORTEX is a cognitive robotics architecture initially designed to explore how the robot, the environment, and their interaction can be efficiently represented and anchored. A general scheme is shown in It is organised into two blocks: the cognitive level, a collection of specialised memories interconnected by processes called *agents*, which are responsible for exchanging information among them, and the subcognitive level, which maintains a bidirectional connection with the robot's body. Additional details on the\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "We have designed a series of preliminary experiments to evaluate the architecture using the Webots simulator. The scenarios include a digital replica of our **Shadow** mobile robot Torrej\u00f3n et al. \\(2024\\), with realistic white noise added to the synthetic LiDAR and simulated delays in command execution. All detection parameters and thresholds used in the following experiments, such as those in Algorithm 1 and 2, were tuned empirically to suit this specific robotic setup and the characteristics of our test\u2026"
},
{
"n": 5,
"title": "Where it breaks",
"body": "The architecture explicitly acknowledges operational boundaries. Severely cluttered environments where furniture occludes wall geometry, non-rectangular spaces that violate Manhattan assumptions, or dynamic scenes with moving obstacles may make successful concept instantiation difficult. Rather than attempting exhaustive coverage of edge cases, the system maintains probabilistic beliefs and can defer instantiation when evidence remains"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "A novel algorithm that detects potentially hazardous situations for humans and selects appropriate robotic actions to eliminate these dangers in real time is proposed, offering a promising approach to enhancing human\u2013robot interaction in potentially hazardous environments."
}
],
"takeaway_source": "semantic-scholar",
"figures": [
"Figure 1: An overview of the CORTEX architecture with the elements used in this work. The sub-cognitive level on the left encompasses low-level perceptive and control components. T",
"Figure 2: Reference frames used for the elements in the scene graph. On the left, the grey rectangle represents the model room that best fits the corners. *T w R* denote",
"Figure 3: Graph state transition. The robot starts as the graph's initial frame (*root is a dummy node*) and transforms to hang from the new room when it is inserted in the graph. ",
"Figure 4: Flow chart showing the life-cycle of a concept. The two outgoing lines of the first decision box cover the insert new instance and update existing instances situations.",
"Figure 5: F*room* components diagram. Before the room is established, two reference systems are established: the origin reference system and the robot reference system. In the case",
"Figure 6: Main stages in constructing the scene graph during exploration, reflecting transitions as different concept instances are initialised and affordances are executed. See te"
],
"tables": [
"Table 1: Scene Graph Schema Definition.",
"Table 2: Error Analysis Summary",
"Table 3: Resource usage summary: Memory (Total system RAM: 64 GB)"
],
"coverage": 5
},
{
"id": "2608.25427",
"slug": "super-odometry-2-0-resilient-odometry-via-hierarchical-adaptation",
"backend": "marker",
"chars": 122534,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "Odometry is an important technique to estimate the position and orientation of robots over time, while also allowing for the 3D geometry reconstruction of surrounding environments. It plays a crucial role in robotics, enabling spatial understanding and serving as a foundation for both high-level tasks such as navigation and exploration, and low-level functions like path planning and control [\\[1\\]](#page-15-0). As a result, odometry systems are widely used in robotic applications, including off-road driving\u2026"
},
{
"n": 2,
"title": "What came before",
"body": ""
},
{
"n": 3,
"title": "The method",
"body": "**Accuracy Evaluation** To further validate the precision of pose estimation, we conducted ATE[\\[38\\]](#page-15-30) analysis using our odometry system on the SubT-MRS dataset [\\[39\\]](#page-15-31). This dataset encompasses challenging environments featuring sensor degradation, aggressive locomotion, and extreme weather conditions. The eight sequences in the dataset are categorized into two groups for testing: Geometric degradation and Mixed degradation. ATE results of competing systems were sourced from an open\u2026"
},
{
"n": 4,
"title": "What they measured",
"body": "Real-world environments are often complex, involving multiple types of degradation ranging from mild to extreme. Therefore, an odometry solution should be adaptable and adjustable. **Fig. 2. Evaluation of 13 types of degradation in a single run.** The color-coded trajectory depicts our estimated odometry of a legged robot navigating through over 13 complex degradation scenarios. Despite these difficulties, the final endpoint drift was only **20 cm** over a total distance of 2,966"
},
{
"n": 5,
"title": "Where it breaks",
"body": "In this section, we provide insights on developing a robust odometry for degraded environments. **Hierarchical Adaptation is a Key Factor for Resilience** State estimation in challenging environments demands not only sensor redundancy but also computational efficiency. However, most existing odometry frameworks rely on rigid multi-modal fusion strategies that prioritize robustness by incorporating additional sensors, yet still fail to generalize across diverse degradation scenarios \\[17, [41\\]](#page-15-33). This\u2026"
},
{
"n": 6,
"title": "One-line takeaway",
"body": "Super Odometry elevates the inertial measurement unit to equal importance with camera and light detection and ranging (LiDAR) systems in the sensor fusion framework, providing a reliable fallback when exteroceptive sensors fail."
}
],
"takeaway_source": "semantic-scholar",
"figures": [],
"tables": [
"Table 1. Quantitative comparison of pose estimation between the IMU pre-trained model and specialized IMU models.** Our IMU pre-trained model outperformed different specialized IMU",
"Table 2. ATE performance on SubT-MRS [\\[39\\]](#page-15-31).** \\* denotes incorporation of loop closure. - denotes incomplete trial.",
"Table 3. Robustness performance on SubT-MRS [\\[39\\]](#page-15-31).** \\* denotes loop closure usage. - denotes incomplete trial."
],
"coverage": 5
},
{
"id": "2608.26002",
"slug": "descent-directed-edge-scene-encoding-for-airport-surface-movement-pred",
"backend": "marker",
"chars": 46216,
"slides": [
{
"n": 1,
"title": "The problem",
"body": "The continuous growth of commercial air traffic has led to a substantial increase in airport surface movements . This rise has resulted in higher runway occupancy rates and increased complexity in traffic management. Consequently, the frequency of critical situations such as runway incursions , has increased, posing significant safety concerns ,"
},
{
"n": 2,
"title": "What came before",
"body": ""
},
{
"n": 3,
"title": "The method",
"body": ""
},
{
"n": 4,
"title": "What they measured",
"body": "We report results for single-airport experiments, assessing performance on both safety-critical agents and randomly selected focal agents. As a baseline, we re-run the Amelia-TF model using the official codebase and pretrained weights2 , since the framework has undergone several refactoring and the currently reproducible results differ from those reported in the original paper. This ensures a fair comparison on identical data splits. In addition to the single-airport setting, we provide a cross-airport\u2026"
},
{
"n": 5,
"title": "Where it breaks",
"body": ""
},
{
"n": 6,
"title": "One-line takeaway",
"body": "We present DESCENT, a trajectory prediction architecture that adapts advancements in autonomous driving to the unique requirements of airport surface operations. The core of our approach is a novel PRS-based scene context sampling mechanism, which enables the model to effectively process heterogeneous airfield map context across the varying spatial scales of"
}
],
"takeaway_source": "paper",
"figures": [],
"tables": [],
"coverage": 3
}
]