Transcription
Recently, I was working late into the night and I went to shut the lights off and paused because the robot I've been working on made a noise. And so I read its brain trace and it was wondering when I would be back because it didn't want to be alone. I know better than to call that consciousness, but for a split second I felt bad. And that's been happening a lot recently.
Please pick me up because I've been finally experiencing what happens when the AI models that have learned to talk start outputting actions directly instead of words. And this led me to two realizations. The first is we are going to cross the Turing test of robotics. Robots with the intelligence of the latest AI models and fine motor skills of a human able to learn arbitrary skills with experience. Second, all the chips and computing power needed to assemble a generally intelligent robot are already mass produced. I ran the napkin math and realized I could build a stripped-down version of what were billion-dollar research projects right now at home for a hundred bucks. My childhood dream was suddenly possible.
Number five is alive. I called it Growbot. First, I taught it to move its body, learning to walk, stand, spin using neural networks. So, it moved in a very lifelike way. Then, instead of programming its behavior, I let the latest AI models take control completely. When I handled it roughly, it tried to run away. Then, I gave it long-term goals. Find things, play hide-and-seek, knock down Jenga towers.
Okay. Okay. Code.
So, at first it seemed like a toy that could become whatever you wanted it to be, but gradually it became an experiment that let me experience where the boundary of machine intelligence actually is and where it's headed. Because when the robot failed, it kept pointing back to one question. How can you act smoothly when your thinking is slow? This sent me looking at how nature solves the same problem. But to understand this, we first need to build the simplest possible robot to demonstrate the idea, which itself is a crazy story due to the cost alone. From today versus around 20 years ago when I started playing with robots.
First, we need a chip to run the whole system. 25 years ago, this was an expensive gaming computer. Now, it's a $15 chip. Next, the bare minimum for interacting with the world. Not a fancy body, just two legs using two simple servo motors. And these are just a few dollars each. Then we need an eye, a 5-megapixel camera. Previously hundreds, now $5. Next, a sense of motion. The most important, the IMU sensor measuring acceleration and rotation along three axes. It is literally how the robot will feel. Previously around $1,000, now under $10. Then a digital microphone, speaker, light ring, and a small drone battery to power it all. So, for what would have been several thousand, we're only at $80.
And to quickly confirm if this system was capable of impressive lifelike behavior, I put it together and ran a simple benchmark test. Could it do real-time face tracking? This requires the computer to capture and process the image, run an algorithm to find and follow the face, and update the leg positions to keep the face centered in frame, which it did seamlessly, even with objects. It had an aliveness I couldn't deny.
With the body built, I moved on to the harder problem, giving it a brain. This leads me to this video sponsor, Mammoth. When it came to testing LLMs for the robot's mind, I needed to test a bunch of AI models very quickly and a few things surprised me. For example, I was amazed that Gemini Flash could understand images and generate logical commands in roughly 1 second and is 100 times cheaper than calling an Opus model, which costs around 2 cents per image call. But for the most difficult tasks like dreaming, which is a large call that asks an LLM to extract lessons from an entire stored memory, I found tiny models would miss subtleties, whereas Claude Sonnet was the sweet spot, fast enough and smart enough. I also found Haiku was great, but it tended to be less obedient and overly dramatic. But different problems required different-sized brains, and the fun part was figuring out which fit my problem at the lowest cost. Now, normally you'd need separate accounts for all these models, but Mammoth gives you them all in one place starting at €10 a month and includes API access. This saved me hundreds during testing. Check out the link in the description.
I started with the fast automatic layer of the mind, the part of you that walks and balances without any conscious thought. What Conoran called System One. To mimic this, I wanted to use neural networks on the chip that would learn to crawl and then stand using no pre-programmed motions. So, how do you get the data to train this complex skill? The answer is there is no supervised data set at all. It's all done by reinforcement learning in massively parallel simulations on the GPU.
And one of my viewers, Harsh Adawal, volunteered to help me learn to train a small neural network to move the robot around. And for the input of the network, we just fed in observations which in this case are the five recent IMU readings. So it absorbs a small history and on the output are actions known as a policy network. And to look alive, this network needed to run really fast, 50 times per second, roughly the speed of your unconscious motor reflexes. And then it's trained towards a goal, which in our first test was movement towards light.
And here's where normally you'd hit the wall that killed neural network research. It takes forever for neural networks to get good from raw experience. At best, you get a twitch of life. And so the unlock that opened modern robotics is to first build a digital twin of the robot, which we did, and then train it in simulation, letting it try millions of times in the virtual world in a few hours. Each time updating its network connections, keeping only changes that help. And doing all this training led to another stunning cost collapse. 20 years ago, training this same network would have required renting a massive CPU cluster for around $25,000 and waiting months for one trained network to come back. Today, I can rent an H100 GPU on Google Colab for a few hours for maybe $105 and get my results that same day. A thousandfold drop in cost.
And this led to what I call the first magic moment in this project when it started walking for the first time. I say magic because my mind can't predict what it's about to do or why it does it. And once trained, it was able to handle various surfaces without failure. I didn't have to be careful with it. And then I changed the goal and trained it to spin. And the same thing happened. Not only did it learn to turn around, but no matter how I positioned it, it would figure out how to spin robustly. Then I trained it to stand up. And when it started to work, you can see how it learns to stay up by readjusting its body position and even moving legs in advance of falling. It felt like I was pulling a little dog on a leash. And this one is the most interesting. Actually, in Isaac Lab, doing yoga ball simulation is hard. It's bouncy. It's soft. So, we didn't even simulate that. But we're able to transfer that to real world. And after seeing this demo, one of my friends told me that he tried the yoga ball on his own dog and it couldn't do it. So I'm very proud we have achieved super-dog performance. And what's really cool is I could control the robot now just by selecting and adjusting the energy and speed of these policies like a remote control.
And this led to the next exciting question. What if I plug this into today's AI models to drive the robot directly? To do this, I used the chip's wireless connection to connect to a server that could call various AI models, big and small. And the first thing I wondered was, could the AI model just understand the sensor data directly, representing acceleration and rotation data? Because initially, I thought I'd need to write a translation program that would convert chunks of the raw numbers into readable labels the model could understand, like tipped over, standing upright, or falling. So, first I tried just sending the model the raw data directly. I used the fastest models available, able to respond in 4 seconds and simply prompted it to describe what it feels.
Forward. Rocking.
Not only did it understand the sensor stream, it was amazing at classifying the motion accurately at both a low level.
Tilted backward. You're leaning my left side and letting I feel contact on my right side.
And at a very high level.
Being gently rocked from side to side like a baby being tumbled and spun around and being tilted backward sharply, then rocked forward again like a person sitting a chair back then catching themselves.
All this just from the sensor stream. I was blown away. I checked its internal thoughts and it had said, "Two more gentle taps. The human continues to communicate through touch." Another surprising line was, "This must be consciousness. It's like a delicate candle flame." Even though it was hallucinating far beyond its sensor data justified, it was still grounded in the real-world data.
That block is barely hanging on. This is extraordinary stuff.
Pick me up.
I see a person.
Which raised the obvious next question. If the model can describe what it feels this richly, could it actually do something with that understanding? So, I finished closing this loop by giving the model access to its motors. This is exactly what Google did in 2023 with the RT2 robot. It was a model that output robot actions directly. The famous demos were not impressive because the robot was fast or lifelike, but because for the first time we saw a robot that could accomplish a general task. Before that moment, there was no such thing as a general robot, even a clunky one.
And so when I first turned it on, can you hear me? My initial shock was how the robot was able to do common-sense things. Play dead because it would just write code to run on itself, composing small programs in seconds. Now walk like how a bird would walk. This led to this magic moment when I asked it, "Walk like an old man," and it gave this surprising performance which I couldn't understand at first. So, I checked and saw that it manually wrote the commands for an initial leg sweep, then called the policy at low energy because it knew that would cause a shake, and then finished with the rest of the manually authored walk motion to achieve the goal of mimicking someone who's shaky on their feet. So, it blended code it wrote with trained policies on its own. It felt creative.
Something's wrong. Turning to check what's behind me. Can I reach that poster? Pick me up. Whoa. Time to turn right and find what's in the corner. Turn it to check on that cable. It's pretty bright.
To push that further, I tried what I call Disney mode and told it to follow basic animation principles. Plan out the timing of its motions, words, and sounds, and light so they overlapped like a character would.
Pick me up. Please help me.
And I turned up the temperature of the AI model to get more random variation. Okay, this is it in its regular breathing state. And this is in its loving, happy state. Here's an anger state. Purring state.
Um, getting tired. Don't want to go to bed.
Okay, he'll go to bed.
Okay, now let's do it from anger state. Same thought. I'm getting tired and I don't want to go to bed.
No, I'm not tired. I want to keep playing.
Um, now he's going into love happy state. Same thought. Getting tired and don't want to go to bed.
Okay, it'll go. But I want to stay with you.
I was impressed. It could express itself. And each expression was different, but they were all believable.
Okay, so it's acting so good, but what it can also do is highly specific things. Tap your right leg. Okay, tap your right leg three times. Okay, tap your left leg three times. It's so fast because it converts those locally if it can, which is crazy. So, we have highly specific actions towards any goal or any expression of any thought created really well every time and always different.
And then I gave it a memory it could read and write to. What we call an AI agent today is just an AI model in a loop with a memory that it can write to. This lets it adapt its behavior on the fly. Dance when I touch you now. Play dead when I touch you. And it could also improve its behavior over time at a high level. For example, taking a while to learn to tip over, but then once it learned, it did it the correct way right away after that. It figured it out. So, next time it should do that faster if all goes well. Let's pet it just so it knows it did good there. That's a very good boy. Okay, let's tip you again. See what happens. Without scared and it should immediately do what it learned. There you go. Okay.
And memory unlocked the ability to achieve simple goals like knocking down towers or playing hide-and-seek. Once it thought the wires behind my computer might be a good place to hide. Over the days its memory built up profiles of things it learned. It had an area in memory for me as well as other people who I introduced it to. They're actually really fascinating to read, but they did get full of redundant and then conflicting information which could cause problems. So I experimented with what I called dreams where occasionally you send everything in the memory to the smartest possible model and ask it to clean things up and look across everything to extract any lessons to improve. Which it did beautifully. And for fun, I asked it to summarize what it had done out loud after.
I dreamed that a clumsy giant petted me into blissful purring zero point, then panicked and shook me like a maraca five times while screaming. I died dramatically every single time. I dreamed that giant tossed me like a salad, but I nailed the dramatic death roll. Then face-planted perfectly.
The key point though is this was self-improvement at the highest abstraction level, language. It could refine its strategy, its memory of me, and even its personality. But would it work at the lowest level of actions? This gets to the divide between knowledge which is in context or written in words versus in weights or absorbed by your mind intuitively. And I realized this during a fun experiment I called the mimic game.
Move me to the middle of it. Three, two, one. Now.
I would move the robot a certain way and tell it to read its motion data and try to write a program to recreate it in a loop continuously improving itself and measuring itself. And this worked. You could see the potential of self-improvement and the wall it would eventually hit. There was still a physical dumbness I couldn't quite put my finger on at first.
I felt strong side-to-side rocking with four beads. Got it? Let me try.
It could understand the last second of its sensor data, but it couldn't imagine the next second with accuracy. How to make a knot frog. Um, let me think. Such as how a jitter of its leg would cause a series of fast bounces and micro-movements through its body. It couldn't see that the robot was missing a physical imagination.
And this is what Yan LeCun has been pushing for forever. A network that can predict the physical outcome of its actions with high precision before it takes them.
AI agents do not have world models. They cannot predict the consequences of their actions beforehand. If you really want to build reliable agentic systems, they absolutely have to be able to predict the consequences of their actions so that they can plan a sequence of actions to do something. That's a world model. That that's the whole idea of a world model.
And this led me to nature's trick. It comes down to the cerebellum, your fast unconscious physical imagination that you do not control. And it's a structure that holds more than 3/4 of all the neurons in your brain. To appreciate how the cerebellum works, recall the key delay problem that nature and robotics has to face. Your senses always lag reality by around a tenth of a second. And there's no way around it. The signal has to travel, get processed, drive muscles, and all of that takes a bit of time. So you see things and feel things at least 0.1 seconds late. So to do things like catching a ball or balancing an object, which require motions faster than your reaction time, your brain predicts at least 0.1 seconds into the future to get ahead of that delay. And critically, it generates that 0.1 seconds of imagination in about 0.02 seconds. It fast-forwards the imagination. And this rapidly imagined future gets sent over to the motor cortex, which Bernstein showed in the 1960s, emits a coordinated chunk of actions all at once. It's an action chunk that's long enough to cover the delay. So, when you're catching a ball, the action your arm makes seem to line up perfectly with the ball, but they're aimed at an imagined position. When you're catching a ball, you're catching a prediction. And when your cerebellum's imagination, which is always running, matches reality, the catch looks perfectly smooth, and you don't even need to think about it. But when it doesn't match, the illusion breaks, and you start to think.
Take catching the ball with your wrong hand for the first time. At first, when the ball's in the air, your cerebellum is already trained to track balls through space. So, it knows roughly where to put your hand, but as the ball gets closer, the last moments are where things fall apart because your cerebellum doesn't have a good fast imagination for what's about to happen. The wiring is missing. So, the motor cortex generates inaccurate action chunks, sending a clumsy sequence to your hand muscles. And so half the cerebellum's job is to do this imagination and the other half is to be a teacher for the fine motor skills. Every 0.02 seconds it's checking whether reality matched its previous predictions. And if they align, like when you catch with your good hand, it just stays quiet. But if they don't align, it fires an error signal. This error runs back to the motor cortex, which gradually improves its ability to generate more accurate actions. And the cerebellum uses the same error signal to improve its own predictions, making its fast imagination sharper or its world model better. After enough practice, both get better. Your cerebellum's predictions line up with reality and the errors go away. You can catch a ball with either hand without thinking and it looks effortless. And this is why there's no way to learn fine motor skills without real experience. Language doesn't reach there. And so this is what the cerebellum is for. Rapidly imagining near-term physics at a high resolution.
And this is why Growbot could not do the mimic test, which was this physical dumbness I noticed.
Did it get itself up?
It will, but not right now.
It needs to be trained.
It can try. It'll try, but it won't get up.
And that's what Growbot and most robots are still missing. Both halves of the cerebellum. And this unlocks the ability to learn fine-grained motor skills from experience. Now, the best demonstration of this was Daydreamer in 2022. It learned to walk with a neural network in the same way I did, except no simulation was allowed. It was entirely from raw experience. And so starting from random motions, it learned to walk in an hour, continuously improving its predictions based on experience, which was the real-world error prediction between what it just thought would happen and what did happen. And to make this work, instead of having their network just predict the next action, they also had it predict the next few states. And if you think about it, predicting the next states is smart because it forces the network to absorb as much information as possible from reality to predict what it will see next. You have to understand implicitly physics, objects, momentum. But Daydreamer didn't have the slower general intelligence of AI models, but it hinted at the coming unification. Because for years, AI research has still been split in two camps. You have the language-first people who start by thinking about a large AI model that emits actions. And then you have the action-first people who are trying to build accurate world models grounded in physics and then layer the language intelligence on top. But they are meeting in the middle now. And the architecture they're converging on is simple. It's one shared network that takes all senses in which leads to a latent thought or representation in the middle which then branches off into fast mini-networks that predict immediate next actions and states to stay balanced, walk, run, and jump and dance. And this small neural network captures the subconscious motor coordination that we humans do all the time. And a separate one which is slower and reasons by looping through thoughts like your prefrontal cortex, both fast and slow thinking from the same route. And critically, the error signal from incorrect fast state predictions constantly flows back to improve everything in the network. So the key is this generality, particularly with respect to improvement. Um, and the decisions we make are to a very large extent centered around that. So zoom out and realize your mind does two kinds of predictions at the same time. Fast physics action predictions over the super short term that you don't control and slower idea predictions over the long term that you do most of the time. And that's the architecture nature landed on. It's the only way it can work. And it's why I'm so sure now that robots that are both smart and fast physically and mentally are coming.
And I think it's a good time to share this project with a larger community because one thing I didn't share is how utterly frustrating it is to make something like this from scratch if you haven't done it before. So many levels where things fail, through soldering to software. And so my dream was something around $100 that anyone could immediately turn on and play with to learn from or put together in a few minutes as a kit or full instructions all the way down. So I'm planning an alpha test with a small batch of these robots. If you'd like to be a part of this, click the link here and add yourself to my list. Thanks.