Anthropic’s strongest result is not that a language model can control a robot. It is that the control interface can change the apparent capability almost as much as the model.
In its new Embody benchmark report, Anthropic connected multiple frontier models to classic control tasks, simulated quadrupeds and humanoids, a robot arm, and a real Unitree Go2. The models received four kinds of authority: direct low-level actions, Python controller generation, reinforcement-learning supervision, or high-level commands to a pretrained policy.
Direct manipulation remained unreliable, with reported full-task success rates from 0% to 5.5%. Higher-level interfaces produced much more useful behavior. The immediate engineering lesson is to keep the LLM above the real-time control loop.
Embody separates the model from the interface
Robot benchmarks often collapse the complete stack into one score. Embody varies both the model and the mechanism through which it acts.
Anthropic’s composite score stacks performance by interface. Mythos Preview is highest in the published chart at 0.389, while Opus 4 is lowest at 0.115. Source: Anthropic.
The four interfaces test different abilities.
- Direct control asks the model to choose forces, torques, or actions repeatedly.
- Programmatic control asks it to write a controller that maps observations to actions.
- RL supervision asks it to configure and train a policy before deployment.
- Policy control lets it issue higher-level instructions to a pretrained controller.
These are not interchangeable tests of “robot intelligence.” Direct control punishes latency and numerical instability. Code control rewards abstraction and debugging. RL adds reward design and experiment management. Policy control depends heavily on the quality and coverage of the underlying policy.
Anthropic paused simulated environments between LLM calls during direct-control tests so inference latency would not create a trivial failure. That makes the result closer to a best-case reasoning test than a real-time deployment measurement.
Models improved most when they could revise a strategy
Anthropic reports broad generational improvement on classic control tasks, especially after the first attempt. Initial performance was often similar across model generations; newer systems gained more from observing the outcome and changing their approach.
The report compares direct control, generated Python controllers, and learned policies across pendulum, hopper, and TwinFlipper tasks. Source: Anthropic.
That pattern resembles coding agents. The first plan is not necessarily much better, but the stronger model uses test output more effectively. In robotics, however, every failed attempt can consume time, damage equipment, or create unsafe motion. Iteration needs a simulator, bounded test space, and explicit stop conditions.
The TwinFlipper task is particularly useful because Anthropic designed it as a new chaotic environment that should not appear in training data. GPT-5.4 was the only model the report says consistently learned a competent RL policy for that task, despite weaker performance through other interfaces. This is another warning against using one aggregate score to choose a system.
Direct manipulation is still far from dependable
On LIBERO robot-arm tasks, complete direct-control success remained between 0% and 5.5%. The models made more progress on intermediate subgoals such as reaching or making contact, but finishing a multi-stage manipulation task was rare.
Mythos Preview reaches the highest reported direct-manipulation success rate, while several evaluated models remain at zero. Source: Anthropic.
This gap matters more than a compelling demonstration. A robot can look purposeful while repeatedly failing at grasp stability, object state, or spatial memory. Production metrics should therefore include subgoal completion, intervention rate, recovery behavior, unsafe-contact rate, and task completion under environmental variation.
Anthropic also reports that models struggle with stable spatial memory, self-localization, and long open-loop plans. Those weaknesses argue for external state estimation and short action horizons rather than asking a conversational model to remember the physical world from dialogue history.
Better perception tools did not uniformly solve control
The researchers supplied aids including depth, segmentation, and cursor-style orientation. Their value varied by model and task. On manipulation, depth and segmentation were roughly neutral in aggregate, while some simpler orientation tools helped selected models.
Additional visual structure did not create a universal improvement across models. Source: Anthropic.
More reasoning was not a universal fix either. High reasoning budgets increased latency, and the report’s aid-uplift analysis suggests reasoning alone does not guarantee that a model will use a visual aid effectively.
The deployment implication is straightforward: evaluate the model-interface-tool combination. A robotics team should not assume that the best text benchmark model, the largest reasoning budget, or the richest visual representation will produce the best control system.
The production pattern is hierarchical control
Today’s useful architecture places an LLM at the slow, semantic layer. It interprets the task, selects a skill, checks progress, requests recovery, and explains uncertainty. A pretrained motion or vision-language-action policy handles coordinated movement. A deterministic safety controller enforces limits beneath both.
The hierarchy should look like this:
- The LLM converts the user goal into a bounded task plan.
- A skill or VLA policy proposes short-horizon motion.
- State estimation tracks the world independently of conversation memory.
- A safety controller enforces speed, force, workspace, and collision constraints.
- A monitor stops execution when confidence or state validity drops.
- The LLM receives structured results and chooses whether to retry, recover, or escalate.
This arrangement accepts what the benchmark shows. Language models are improving at adaptation and high-level supervision, but they remain poor substitutes for a fast, stable controller.
The benchmark is valuable, with limits
Anthropic tested a broad set of bodies and interfaces, disclosed substantial methodological detail, and says the code will be published at github.com/safety-research/embody. At the time of the report, the wording was future tense, so reproducibility should be reassessed when the public repository is available.
The results also mix simulated and real systems, different tool access, and multiple model families. The composite score is useful for orientation but should not be read as a universal robotics ranking. Task-specific tables are more informative for an engineering decision.
The benchmark nevertheless establishes a strong evaluation principle: report the body, interface, policy scaffold, perception tools, reasoning budget, latency treatment, and safety layer alongside the model name. Without those details, a “robotics score” hides too much of the system that produced it.
Interface design is only one layer of physical-AI reliability. The one-camera Robostral safety analysis covers sensing and fallback, and the Omniverse agent toolkit review covers the simulation assets needed before deployment.
