What If AI Builds the Robot Instead of Controlling It?
Last year I wrote about robotics moving from bespoke machines toward reusable foundation models, essentially an attempt to build a general-purpose robot brain. More recently, I argued that models are getting easier to buy, while the harder work lies in making data useful. A newer development complicates that story. What if we can automate more of the work required to engineer the robot itself?
Appreciate this newsletter? Consider becoming a paid supporter 🙏
The Engineering Bottleneck in Robotics
A large class of robotics problems sits between rigid factory automation and the general-purpose robots we see in demos. The robot and workspace may stay mostly the same, but packages arrive in different positions, groceries come in different shapes, or cables bend differently every time. Roboticists sometimes call this variational automation. These jobs are common in warehouses, factories, and food-processing facilities, but they remain expensive to automate.
There are several bottlenecks. Robots lack the vast stores of physical experience available to language models, and each new application still requires engineers to assemble perception, grasping, motion planning, and recovery systems. Simulation reduces physical testing, but building one that captures the properties that determine success is itself labor intensive. The commercial bar is also high. A robot has to work quickly and reliably, not just succeed in a carefully selected demo.
Four Ways to Build a Robot
The traditional approach combines well-understood components for perception, motion planning, and control. These systems can be fast and dependable, but they require substantial engineering for each application. More recently, Vision-Language-Action models, or VLAs, have tried to replace some of this work with learning. Give the model images and an instruction, and it produces actions. The promise is broader generalization, but several VLAs struggled when object positions or task instructions changed.
Simulation offers another path. A robot can practice thousands of times without tying up hardware or requiring someone to reset the scene after each failure. The hard part is making the virtual environment predictive of the real one. Small errors in friction, weight, stiffness, or contact behavior can make a policy that succeeds virtually fail on hardware. This is the familiar sim-to-real problem.
World models are increasingly being positioned as part of the answer. Some learn from large collections of video, then provide a foundation for robot policies. Others reconstruct real environments, generate additional experience, or test what might happen after an action. The most interesting systems are already hybrids, combining learned representations with simulation, conventional software, and feedback from real failures. The line between a world model and a traditional simulator is becoming less clear.
What If the AI Builds the Robot Program?
This is why a recent post from Ken Goldberg caught my attention. His group is exploring what it calls Agentic Robotics. Instead of asking one large neural network to control the robot directly, coding agents assemble a program from reusable robotics skills. Their Graph-as-Policy system represents the program as connected modules for tasks like perception, grasping, and motion planning. The agents rehearse it in simulation, find where it fails, and modify the relevant piece rather than retraining the whole system.
The more surprising step is automating the simulator too. Goldberg describes giving a frontier model a short video of a robot wiping ink off a metal bar with a sponge, along with access to a physics engine. The model built a simulation, then agents used it to improve the robot program before running the code on the real robot. Not every trial worked. In one, the robot pushed over the bar because its mass could not be reliably inferred from the video. That failure shows why testing on real hardware still matters. This reminded me of a recent conversation about AI-assisted chip design. Hardware engineers are using foundation models to write and optimize kernels, the specialized programs that make models run efficiently on particular hardware. In both cases, AI is helping experts with engineering work that has traditionally been detailed, specialized, and time consuming.
There are reasons to take this seriously, but also reasons not to get ahead of the evidence. In one benchmark, the agent-generated system came close to a graph built by robotics experts. In another example, Ambi Robotics reports using agents to analyze 500 production events and develop three candidate fixes in about ten hours. The company then tested all three on production robots in customer warehouses for ten days. Two reduced throughput. The third increased it by 4.2 packages per hour, and Ambi says it deployed that fix across 30% of its U.S. fleet. These are still bounded industrial tasks, and open-ended manipulation and humanoids are much harder. The interesting result is not that the agent is always right. It is that generating, testing, and improving highly specific engineering solutions may become much cheaper.
World Models May Have to Earn Their Place
What changes my view is how far Agentic Robotics can get without a learned world model. A general-purpose foundation model interprets the task, helps build a task-specific simulation, and writes the robot program, while a conventional physics engine handles the dynamics. For bounded industrial tasks, that suggests learned world models are not always necessary. Companies developing them need to show where a learned model adds something that a foundation model, existing robotics software, and a physics engine cannot provide as well.
I would not count world models out. They might reconstruct an environment, fill in physical behavior that is hard to infer from a short video, or generate variations for stress testing. A VLA might still be the best way to handle a manipulation skill that is hard to program. The bigger shift is that none of these models necessarily has to become the robot brain. The general intelligence can sit in an engineering agent that chooses the models and tools needed for a particular job.
This also changes where I would look for durable advantage. Individual models can be replaced as better ones arrive. Calibrated simulations, reusable skills, evaluation environments, failure histories, and production data are harder to reproduce. This should look familiar to enterprise AI teams. The foundation model gets easier to buy, while the surrounding system becomes the asset. For robotics teams, the practical question may shift from “How do we train the best robot brain?” to “How quickly can we assemble a system that turns observations of the real world into behavior we can trust?”
Book Recommendations
- Broken China: How the Economic Miracle Shattered and What it Means for the World. Most media coverage of China focuses on its export and manufacturing prowess. Yet when I talk with visitors and friends just back from the country, what they most want to discuss is the bleak job market, especially for young people. For anyone who wants to understand China’s economy, this book is essential reading. Drawing on decades of on-the-ground analysis, it is rigorous in both its arguments and its evidence.
- Kobuk the Destroyer: And Other Tales from the Wild, Unseen World of Test Engineering. I enjoyed this entertaining look at the often overlooked world of test engineering. As AI applications and agents become more capable and complex, I think AI teams have a lot to learn from decades of experience testing how systems fail, behave under real-world conditions, and handle unexpected situations.
- Money to Burn: The Unvarnished Truth About Leon Black, Apollo, and the Rise of a New Wall Street. This book gave me a much better understanding of private equity through the story of Apollo, a firm born out of Drexel Burnham Lambert’s collapse. I found Apollo’s evolution toward private credit especially interesting, particularly its ties to insurers and annuities, which are becoming increasingly important to financing the AI buildout.
- The Everywhere Millionaire: Who Is Really Rich in America and How They Got There. If you want to better understand America’s small and midsize businesses, this book is a goldmine. I learned a lot about how the owners of seemingly ordinary businesses build wealth and wield economic influence. Written by two academic economists, it’s also a surprisingly fun and engaging read.
The $2.1 Trillion Compute Landlord
