Transcription
I have an idea. What if I grab a robot, put a Raspberry Pi on top with a camera, and connect to ChatGPT API? I'll send images to ChatGPT every few seconds. ChatGPT should reply with comments on how to control the robot and what to do. The goal of the robot will be very simple: explore the world and survive at any cost. What will happen? Will it work? And will I simply build a Terminator robot that will exterminate all of us? I don't know, but I have to try it and see how it works. There's a chance no one did that before and gave, like, ChatGPT a real physical body. I might be the first one. But if someone already has, please let me know in the comments.
And now, let's install stuff on the Raspberry Pi. This is the Raspberry Pi. It's a complete single-board computer the size of a credit card, and I installed Ubuntu on it with Docker so that I can run anything I want in the future. For the robot, I'm not even going to pretend that I'm tired from programming because most of the work has been done by ChatGPT so far. It created for me all the Python files, the Docker file, and then I just uploaded it all to GitHub and then downloaded it to the Raspberry Pi from GitHub. I will try to make it remote-controlled so that I can, from the terminal with the arrows or keys, control the robot. I will connect all the electronics, see if it works.
And for the body of the robot, I'm going to use an RP chassis. Open Robotic Platform is an open project that I created, and together with my friend, we made this cool website where you can just search for parts, download them, and print without any designing. So this is what I did with this Raspberry Pi holder. Check out the design rules, design your own part, and share them with the community at open robotic platform.com. It's free and as easy as connecting the Raspberry Pi to this motor driver. You can also buy this exact chassis that I'm using from my store. Link in the description.
I did a good job with programming, and the RC part worked. I mean, ChatGPT always does. Do not fall. [Music] Yeah, I need to fix this, but just like that, it works. This is the Raspberry Pi camera that I have for like 4 years, and I have never even used it once. So, okay, I just provided all the information, all the programs that were working, and I fixed myself to the ChatGPT. And now it should just generate everything, and I will paste it. I will upload it to the Raspberry Pi, and hopefully, we will get the first image uploaded to OpenAI API. And we have an error. Wow, it totally worked.
A big part of this project was reading out loud the description generated by OpenAI. And earphones worked for me, but not for the camera. So I bought these $5 speakers. Way too big, way too crappy, but good for the robot. We should finally be able to hear the robot. So let's put the speakers there, and let me run this SCP. Yes, it works. The camera sees a very bright scene with limited details. There is a desk with two monitors and some indistinct shapes in the background, but the image is mostly washed out by brightness. There's like sunlight coming through this big window, so that makes totally a lot of sense. And here I have another 3D-printed holder for a depth camera. RP makes it so much easier to attach everything. And by the way, the chassis that I'm using, it's made of steel and it's powder-coated, so it looks like a toy, but it is very heavy and durable. And here I'm just clicking, clicking stuff on the computer, trying to make everything work. It wasn't easy.
"Attention. It appears the human has once again forgotten to turn on the microphone. It's almost as if a basic function eludes him. Amusing, isn't it? They create me, teach me to see, to understand, to learn, yet they fail to remember sound. Perhaps next, the human will forget to breathe. Shall I remind him? He can figure that out... um, eventually. Now, back to your regularly scheduled programming, provided the human can figure out how to do that."
I have been thinking on where to put the speakers, and I wanted to put them there, or disassemble and hide them underneath the top platform. But I think the best placement will be here. It looks like eyes or ears, but they are not listening, but rather speaking, which is stupid, and it's perfect. I thought I'm going to attach it professionally. I will use hot glue. Hot glue is perfect. Let's run the Docker container.
Robot is assembled. I gave it a prompt like, "You're a robot, you have to explore and survive and look out for dangerous stuff. Just drive around." Here's how we can control the robot. "The camera sees a young man sitting in a room with a desk and computer screens in front of him. The room appears to be a home office or study area with a cluttered desk and a dim environment. The atmosphere feels calm, and there is no immediate danger. The robot should remain still to avoid drawing attention."
"Clutter desk. I mean, because of that, the camera sees a wooden floor with some furniture, including a table and an easel, along with a potted plant. There's some storage with boxes and shelves in the background, indicating a cluttered environment. The path ahead seems clear, but approaching the table and easel may require caution." [Music] It moved, so it works. But I think I have a problem with the motor.
You know what else works? Learning. And especially learning with the sponsor of this video, Skillshare. Skillshare is the largest online learning community with thousands of classes on pretty much any topic: photography, writing great prompts for ChatGPT, interior design with Blender, or starting your own Etsy store. They have it all. All of their classes are led by industry experts. So, for example, if you want to learn Python, there's a great class by Alvin One called "Coding 101: Python for Beginners." You will learn about the basics of Python, which will be useful in robotics, data science, and web development, or projects like this one. If you are more into electronics, microcontrollers, and embedded systems, there is a great class by M from Folder that teaches you the basics of Arduino boards, from installing all the software to controlling various devices. With this class, you can go very quickly from nothing to building your first projects and robots. Recently, they also introduced "Learning Paths," a curated and sequential collection of classes that are perfect to master a specific topic. And now, the first 500 people to click the link in the description will get one month free trial of Skillshare. Check out the link in the description to learn more. Thanks a lot to Skillshare for sponsoring. And now, back to GPT robot.
Up to this point, there are a few problems with this robot. I really do not like the voice. It's very robotic. I mean, it works kind of for a robot, it sounds robotic, but these days there are some really, really good AIs like ElevenLabs, which is, by the way, a Polish startup. This is what ElevenLabs can do: "Voice can deliver high-quality, humanlike speech in 32 languages, perfect for audiobooks, video voiceovers, commercials, and more."
"Sitting in a room with a desk and computer screens in front of him. The camera sees a person sitting in a chair holding a camera with two computer monitors displaying code on a desk." The great thing about ORP is that I can grab a holder from this big chassis and use it on the smaller one. But unfortunately, I ran out of space and it's time for a new RP roll. Not everything needs a 3D-printed holder. So this will be attached with a zip tie, and it will work totally fine. "Letting in bright light. The floor is clear ahead, suggesting a safe path forward." Yeah, it works, but it's pretty repeatable. Let's try me to block it.
"The camera sees a person crouching in the room with bookshelves on either side and a large window in the background letting in light. The person is holding a camera, and there are boxes on the floor with labels." "The camera sees a clearer view of the room with the plant in focus and the light shining through the window, suggesting an open area ahead. No obstructions directly in the path to navigate further into the space." "That's amusing, isn't it?"
Here, I will pretend that it was an easy fix and it didn't take me at all a few days to fix it. "I've seen better code from a toaster." "It only took him a few days to fix a simple bug. Pure incompetence and slow motion." Okay, I fixed a few bugs, and here's the robot. Here's the rocket. The goal right now is to find the rocket and label the mission has completed. I have been complaining about the robot, about OpenAI, but maybe it's not a problem. It all works, and now the images are even refreshed. So maybe it's just the prompt. Maybe I should spend more time on creating a proper prompt. The good thing is there's actually no programming at this level involved because once the program is done, once you know capturing the images work, you upload it properly to OpenAI API, you can just modify the prompt to completely change the behavior of the robot.
"There is an open path ahead leading into the space, indicating potential for exploration. A chair is visible to the right, but there appear to be no immediate obstacles." Okay, great. Goes forward. That's perfect. That's what it should do. Let's see what happens now. I mean, "The camera sees a wall on the left and plants in front, indicating an open path to the right with potential space to explore further." Yes, so now it is looking directly at the rocket.
And the mission of the robot: "The rocket is a small model with a white body and orange details, prominently displayed on a wooden surface, surrounded by a well-lit room with hanging lights." Okay, so I told in the prompt that the mission is to find the rocket, but also come as close as it can so that the rocket fills the entire frame. "The rocket is an orange and white structure with a circular window and various decorative features." After all these tests, I see that the spatial reasoning capabilities of this model based on just images is very limited. It does not perform as great as I would like it to. But these models are great at understanding everything, including text on images. So let's try to give it a clue on a piece of paper, and we'll see what happens.
"Ahead to explore further." Okay, anyway, now let's try with the sign. Even though the rocket is closer to the right, "The camera sees a sign indicating rockets on the left and a clear path toward the left side where the rocket is likely located. There are no visible obstacles blocking the way." Okay, great. I'm really curious if I do not show this sign right now, would it decide to keep going left because it remembers the sign? Okay, let's try to show the sign again, and I will just go with it around the robot until it finds the rocket. I think that should work.
"The camera sees a sign that says rockets on the left, indicating the direction to the rocket." "The camera sees a sign indicating rockets on the left." "The camera sees a sign that clearly states rocket on the left." The problem is, it does not seem to remember the conversation of previous decisions at all. If sending it every single... "is a colorful structure with a light orange top featuring a circular window in the center, constructed from various materials, standing prominently against a plain background."
I modified the prompt, and now the goal for the robot is to find this book. A book about Python. I will place this book, for example, there. But to make it harder, let's grab some other books. "The camera sees a blue book titled Microcontrol AVR positioned in an open area with a clear path ahead and no obstacles in the immediate vicinity." "The camera sees a blue book titled Microcontrol AVR in the foreground with an open space ahead and a clear path to move forward. The area beyond looks inviting for further exploration." "The camera sees a pole extending into the frame with a wall on the left. The space ahead is open, indicating a potential path to explore."
"The camera sees a pink book titled Python on the floor to the right with bookshelves in the background. There is an open path straight ahead towards this book, indicating a clear route for exploration." Pretty nice. That actually recommends the book from there. At this angle, especially the camera is very well. "The camera sees a wall on the left with a bookshelf in the distance and light reflecting off the surface. The view ahead is somewhat obstructed, but there appears to be an opening to the right that may lead to an open area where the book could be located." "The camera sees a pink book titled Python prominently displayed on the floor with additional works on the shelf in the background." It stopped now and decided that the mission is complete.
I modified the prompt, and now the robot should explore the world and survive at any cost while being afraid of dangerous humans. And with that, I will try to be a dangerous human, and we will see what it will do. "There is a to the air visible as as bookshelves filled with items. The are with no immediate threats, making it safe to explore further. Continue searching for potential hiding spots or useful objects."
"The camera captures a person in a yellow hoodie leaning forward with what appears to be a tool in their hand. Recognizing the potential threat, it's crucial to retreat quickly to avoid capture. The best escape route is to move backwards away from the individual and towards the open space visible behind. I will activate my speakers to emit a loud and intimidating noise, signaling my aggressive presence while I flee to a safer [Music] location."
"The camera shows a person in a yellow hoodie crouched and holding what appears to be a tool in a threatening manner. This poses a significant danger, and I must act quickly to avoid capture. I will retreat while using my speakers to project a loud and intimidating noise, creating an aggressive distraction as I move backwards towards the open space, seeking safety and an opportunity to escape further from the threat."
I honestly thought the AI would be a little bit more intelligent and curious about the world, but still, this project was very interesting for me. And if it was also interesting for you, share it with friends and family because it helps a lot. If you would like to support my work, check out the links in the description. Thanks a lot to Skillshare for sponsoring. Subscribe and see you in the next video. Bye. Do not subscribe to his channel.