2026-01-28 · IROS 2026 · 2 citations · club pick

Demonstration-Free Robotic Control via LLM Agents

Brian Y. Tsui, Alan Y. Fang, Tiffany J. Hwu

Published at IROS 2026. 2 citations, as of the last refresh.

Abstract

Robotic manipulation has increasingly adopted vision-language-action (VLA) models, which achieve strong performance but typically require task-specific demonstrations and fine-tuning, and often generalize poorly under domain shift. We investigate whether general-purpose large language model (LLM) agent frameworks, originally developed for software engineering, can serve as an alternative control paradigm for embodied manipulation. We introduce FAEA (Frontier Agent as Embodied Agent), which applies an LLM agent framework directly to embodied manipulation without modification. Using the same iterative reasoning that enables software agents to debug code, FAEA enables embodied agents to reason through manipulation strategies. We evaluate an unmodified frontier agent, Claude Agent SDK, across the LIBERO, ManiSkill3, and MetaWorld benchmarks. With privileged environment state access, FAEA achieves success rates of 84.9%, 85.7%, and 96%, respectively. This level of task success approaches that of VLA models trained with less than 100 demonstrations per task, without requiring demonstrations or fine-tuning. With one round of human feedback as an optional optimization, performance increases to 88.2% on LIBERO. This demonstration-free capability has immediate practical value: FAEA can autonomously explore novel scenarios in simulation and generate successful trajectories for training data augmentation in embodied learning. Our results indicate that general-purpose agents are sufficient for a class of manipulation tasks dominated by deliberative, task-level planning. This opens a path for robotics systems to leverage actively maintained agent infrastructure and benefit directly from ongoing advances in frontier models. Code is available at https://github.com/robiemusketeer/faea-sim

arXiv comment: Accepted to IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2026

Ten-minute slide kit

Six slides is the whole talk: what was broken, what people tried, what these authors did, what the numbers say, where it falls over, and the sentence people should remember.

SLIDE 1

The problem

Learning-based robotic manipulation policies often require task-specific demonstrations, fine-tuning, and specialized execution pipelines. Despite the effectiveness of these policies, these requirements create substantial data-collection and engineering costs when adapting robots to each new task or environment. We therefore ask whether a general-purpose language model agent can instead perform manipulation using existing reasoning and tool-use capabilities, without demonstrations or robotics-specific

SLIDE 2

What came before

Not recoverable from the parsed text. Read this section in the paper yourself.

SLIDE 3

The method

The key insight of FAEA is that general-purpose agent frameworks designed for software engineering tasks can transfer to manipulation tasks in simulation without modifying the agent framework. We demonstrate this by using the Claude Agent SDK, the same infrastructure that enables Claude to write and debug code, to control robots in simulation environments. As discussed in Section I-B, we leverage the capability asymmetry between high-level reasoning and lowlevel control by delegating manipulation primitives to…

SLIDE 4

What they measured

We evaluate FAEA on LIBERO to establish native agent capability, then test whether human coaching provides additional optimization. We then apply the same agent loop and prompt to ManiSkill (domain randomization) and MetaWorld (different robot arm and environment). This tests transfer under a shared privileged-state and Cartesian-control abstraction, not other agent frameworks or physical hardware. *a) Simulation Interface:* Upon ingesting the FAEA prompt, Claude Agent autonomously discovers simulation APIs by…

SLIDE 5

Where it breaks

<span id="page-5-2"></span>*a) Precision Manipulation:* Tasks requiring subcentimeter precision (e.g., peg insertion, plug insertion) consistently fail across all conditions. Inspection of failure traces reveals that precision tasks fail due to insufficient positional accuracy in the ReAct loop: the agent correctly identifies the insertion goal but cannot achieve sub-millimeter alignment through discrete action commands. This suggests a fundamental mismatch between deliberative reasoning (seconds-scale) and…

SLIDE 6

One-line takeaway

Github: https://github.com/robiemusketeer/faea-sim

Assembled from the paper's own PDF, parsed with its layout intact, 50,948 characters of it, then split on the paper's own section headings. Extractive, not generated: every sentence here is lifted from the paper. The takeaway line is the club's own one-liner from its reading list. Check it before you present it.

Presented at

Saturday, July 4, 2026

Saturday Robotics, session 16

Robotics & World Models Reading Club 16: The Embodied AI Hardware Stack — Supply Chain, Sensors, and the Data Flywheel — SF 07/04

Listed on the event page as “Pre-reading:”. The arXiv title above is the record.

San Francisco, CA

Why the club picked it. Github: https://github.com/robiemusketeer/faea-sim