AI coding agents build a robotics simulation benchmark

Creating robotics simulations demands expert work. Pranjal Aggarwal and Sean Welleck propose using coding agents to build and audit simulations, then use the resulting environments to test AI models.

What the study found

In their arXiv preprint, the authors introduce Gym-Anything for Robotics, a framework in which coding agents create simulations. They say they selected simulations using U.S. GDP data to cover economically valuable physical work, and used an adversarial creation-and-audit loop to check simulations and tasks against defined criteria.

The resulting RoboWorld collection contains 25 simulations, with 175 training and 25 test reinforcement-learning environments of varying difficulty. The authors report that even the best frontier models failed every test task, with an average reward of 0.06.

They also report substantial performance improvements from several approaches: learning in context over repeated attempts; a teacher with access to simulator state instructing a student; learning from a model’s own privileged experience; and using code policies to generate demonstrations for training vision-language-action models.

Why it matters

Broader simulations could offer more ways to train and evaluate embodied AI without requiring experts to handcraft every environment. RoboWorld also provides a test of the gap between current models and the simulated tasks its authors selected. Whether skills learned in these environments transfer to physical robots remains an open question.

Limits and unknowns

The abstract does not specify the individual tasks, audit criteria, models tested, reward scale or numerical gains from each improvement method. It also does not describe real-world robot experiments or establish how faithfully the simulations reproduce physical conditions. The reported results are preprint findings, and their interpretation depends on details beyond the abstract.

Read the research

Authors: Pranjal Aggarwal, Sean Welleck

Published: 2026-10-29

Source basis: Based on the abstract only

Original abstract

Realistic, fast robotics simulations could enable large-scale training of embodied AI models that transfer to the real world. However, despite decades of progress, available simulations cover only a narrow range of real-world tasks and physical phenomena. The core bottleneck is that creating better simulations requires significant expert effort and therefore does not scale. To address this, we introduce Gym-Anything for Robotics, a framework in which coding agents create realistic robotics simulations. Our core insight is that simulation creation is a coding agent task, so improvements in coding agents translate directly into richer and more faithful simulations. We select simulations based on U.S. GDP data to cover economically valuable physical work, define what makes a good simulation and a good task, and develop an adversarial creation-audit loop that checks each simulation and task against these definitions. Using this pipeline, we build RoboWorld, a set of 25 simulations with 25 test and 175 training RL environments of varying difficulty. Even the best frontier models fail on every RoboWorld test task, achieving an average reward of only 0.06. Building on RoboWorld, we introduce several new ways for frontier models to improve, including in-context learning across repeated attempts, a privileged teacher that reads the simulator state and teaches a student in context, self-assisted learning from the model's own privileged experience, and code policies that generate demonstrations across varied scenes to train vision-language-action models (VLAs) such as $π_{0.5}$. Each of these substantially improves performance. We release our pipeline, RL environments, and benchmark to facilitate future research. Code and data are available at https://gym-anything.github.io/robotics

This article is an AI-generated adaptation, not a substitute for the original research. Preprints may not have been peer reviewed; AI can make mistakes.