Transcription
So I've been very busy testing all the new models that just came out: Cool 4.1, 03, 04 Mini, 04 Mini High, as well as OpenAI's codecs. There's just way too many things to cover in one video, so today we're just going to look at one thing. We're going to take all the new OpenAI models—Gemini 2.5 Pro and Claude 3.7—and we're going to see how well they do with making Python games and using reinforcement learning to teach itself to play that game. Let's dive in.
All right, let's pit all the world's best AI models against each other in a game of creating the best autonomous snake game. Not just snake game; we'll do a couple more, but this is the real LM arena—the pit. All the models enter; only one emerges. Also, to make it more interesting, we're going to start with somewhat simple prompts and just rapidly take it to the extreme. Let's go.
Here's the first prompt. This is somewhat the easy one; this is going to be the easier one, although it's not brain-dead simple. First, create a fully autonomous snake game. Make everything in one Python file. Make two snakes that battle each other. Create a scoreboard that tracks their cumulative scores. Their score increases by one every second the snake is alive. Every time it eats a fruit, it gains 10 points, and every time the other snake dies and it survives, it gets 50 points. Right, so if it wins, it gets 50 points each time one snake dies. Reset the game, but keep the cumulative score. We're giving the same identical prompt to 03—that sounds weird—to the 03 model and also to the 04 Mini model, as well as the 04 Mini High. But wait, there's more. We're going to give Gemini 2.5 Pro a try as well, and last but certainly not least, Claude 3.7 Sonnet.
All right, let's start with Claude 3.7 Sonnet. Here we go. We got the two snakes, perfect. We have the round number, perfect. We have Snake 1; we're keeping a score in the top left. We got Snake 2 keeping score in the top right. If I really wanted to complain, I guess I'd say that the blue text is kind of hard to see, but honestly, not a big deal. Notice it increments by one every second, so it captured that portion of it. I feel like, yeah, it's going up by 10 every single time it eats a fruit, and every single time it wins, yeah, probably, probably 50 points. Ooh, it crashed. I got to say I liked a lot of things that were happening in there. Great graphics; everything was looking very, very good. It crashed because of some type error issue, but let's move on.
Next, we have the 1 million context window beast, 2.5 Pro. Google is probably the model to beat right now, so this is what that looks like. So, so far so good; I'm liking it. The score is working as far as I can tell. They didn't go with a grid system here, so, and since the looks like the snakes can't move vertically, it kind of gets this shimmy thing going, but there's nothing I can point to that says that it's bad in any way, shape, or form. It definitely nailed the prompt perfectly. Let's just kind of wait until one of them screws up to make sure that—oh, it's got a little sort of, um, what do you call that, like a summary at the end of the round? That's pretty cool. The cumulative score stays; it's phenomenal. Everything is very good, so it shows you the score for that round and the cumulative score. So there's a lot of things to love here.
Next, we have the 04 Mini High. Looking good. It chose to go with a very nice, good design, and there's sort of a grid-style design. The score is looking good; the things are looking good. It resets properly; I mean, everything is pretty good. I, I mean, the snakes collide quite a bit, so it seems like whatever sort of script they decide to use, whatever algorithm for the little snakes, it, it doesn't think about the fact that it's going to collide with the other snakes, so maybe gets a few points off for that, but overall, I would say it's not too bad.
The 04 Mini, there it goes. Very good. Uh, overall, I think very simple, but the score is very easy to see; one is green, one is blue. And let's see here, there, I just want to make sure they're not constantly colliding like they did with the 04 Mini High. So far, it's looking okay. I'm, yeah, I mean, they tend to collide as well, so, but that's fine. This, this isn't too bad; very simple, but it works. So let's continue.
Next, we have 03. All right, there it goes. Very good. Good sort of score; it's got a little question mark up there, and the two players are just named one and two, so it's a little bit difficult to kind of at a glance know what this represents. I mean, I know because I wrote the, the, the prompt, but, um, for a brand-new player, it might take a second for them to be like, "What, what are those numbers that are going up and down over there?" Notice that the snakes are not colliding; they are extremely good actually at not colliding, so that's the big sort of jump that I see between the 03 and the 04 Mini—basically, the Mini models and the 03, which is the, the full big model—is that it thought about the fact that, hey, maybe these things shouldn't collide, and it wrote that in the code. So phenomenal. Um, I like everything about this. I would say, yeah, I got to say I think Claude 3.7 would have been my favorite had it not crashed. That was my only concern with it; something made it crash. Other than that, there's a lot of things to love here. Based on this, I would probably give 03 and Gemini 2.5 Pro sort of like the pass grade; those two were probably the best. Yeah, it keeps crashing; I think when the snake eats itself or something along those lines.
All right, but what if we take that same prompt and we start adding some complexity on top of it? First and foremost, let's say this: if I run the file with the argument -play, just make the two snakes play autonomously with a simple script. When we execute the file, we can pass these arguments into it to make little different versions of it. So if we run it in play mode, it should do exactly what it just did, but if I run the file with the argument -train, it should do something completely different. For example, here it's going to create a reinforcement learning and training pipeline using PyTorch that trains the agents to play the game over 500 episodes and saves the trained model. And if I run the file with argument one, then use that sort of saved model—the AI brain, the neural network that has trained—as the brain for snake one and the simple original script for snake two. Right, so the smart and evolved snake that would be snake one, and then the original script would be snake two. And also the reverse, so if we run it with the argument two, then we're going to use a saved AI model for snake two and this simple script for snake one. Also, I'm going to add some obstacles just to make those games go a little faster, so we're going to say every second add two permanent obstacles to the field. When the snakes touch it, they die.
All right, so the 04 Mini was obviously the fastest one to finish this task, so let's start with it. It shows us the commands how to run it here. Unfortunately, there's an error, and the script does not run. But next up, we have 04 Mini High. Let's see if it does better. Oop, cell size is not defined. You know what, those are simple ones; I'll just, for the curiosity sake, I will have them try it again, but technically, I guess that should be a disqualification, but let's try the 03. 03 figures it out. I don't know what's happening here; I don't know if this is, I mean, it's not good; I don't know what it's doing. All right, Gemini 2.5 Pro, let's see what you got for us. Show me what you've got. Oh, crashed. All right, I'll let them try again on this one, and here comes Claude. Oh, Claude is doing great. So as you can see here, it's got, it's got everything; it's got the score, it's got the cumulative score; yep, it everything's looking perfect. So it's got the score for the round; it's got the cumulative score; it's adding up, as far as I can tell, correctly. And notice that the, um, obstacles are appearing rather quickly, and if it crashes into an obstacle, it does get destroyed, so that is very, very cool. I do feel like orange just ran into the wall, and as you can see, the scripts are very simple, so it's basically, yeah, so they're running into the wall almost like they're expecting to merge on the other side, and that's just not happening. But overall, Claude 3.7, phenomenal.
Actually, before we get too excited, we need to test to make sure that it's able to train it. All right, so there it goes; it's training it very, very fast. I specifically picked the snake game; I tried it with Bomber Man, I tried some other ones, but it takes significantly longer to do this. So as you can see here, it's running through these episodes pretty quickly; like we're at 60 out of 500 so far. And as you can see here, it's learning. If you take a look at the sort of the reward—oh, that's interesting—so it's showing the snake one and snake two reward, so basically how many points each is getting as it's kind of running through these simulations and trying to get better at it. So hopefully, what we'd like to see over time is that its ability to play this game keeps improving. So we're at 100 out of 50; let's, uh, let's keep going.
All right, so it ran 500 episodes and saved the neural net, and so now we're going to run it by putting that neural net into snake number one, and let's see what's better now. Spoiler alert: it's very likely that the Python script that it wrote is probably going to be better than the neural nets just because of the simplicity of the game; like you really don't need anything too complicated to be very good at this. So I feel like snake number two is going to be a lot better. Oh wow, I am so wrong. Snake number one is crushing it; it's at almost 700 points; snake number two is at 130 points. Wow. Okay, no, I take it back. The first few times that we tried these experiments, the simple logic of, you know, a Python script was a lot better. In this case, training it over 500 episodes seems to have taught it a thing or two, because as you can see, snake one is way, way, far, far ahead, uh, at almost 1,200 points at now over 1,300 points, whereas Snake 2 has what is that, 270, 280, so it's definitely, definitely lagging behind. Fascinating. And just to make sure that if we switch them it still works, now we're going to make the super brain in snake number two. So now if everything worked correctly, we should expect to see snake number two winning handsomely. So let's see what happens here. All right, so snake number two is off to an early lead, over 100 points; snake number one at 78. All right, snake number two still pulls even further ahead to 80; snake one at 130. So yeah, this confirms it: for this particular sort of iteration, the neural net trained snake is significantly better. Very cool. I gotta say.
All right, so that was Claude 3.7 Sonnet. If I recall correctly, he just one-shotted this; created the four different versions with the arguments; everything worked perfectly on the first attempt. Claude wins; fatality. Let's see how well, uh, 03 does. All right, so first and foremost, we're going to test—oh, 03 was the one with the insane time-traveling snake that never manages to eat, to eat any fruits. Okay, yeah, I'm giving it to Claude on this one; it is the clear winner.
All right, and now let's try this: we're going to create a 2D solar system simulator. Make everything in one Python file. Have the player be able to launch probes from outside the galaxy that can slingshot around the planet's gravity wells. Place two stationary targets to hit; one within the solar system, one on the outside. The player's goal should be to hit both by using gravity to change the direction of the probe. I, I wonder if that, that's making sense. So you're trying to kind of slingshot these little things using the sort of, if the planet has a certain like gravitational pole, to like curve it around to, you know, make it sort of change trajectory. So let's see if we can figure out how to do that. Here's the 04 Mini. All right, so far so good, except am I supposed to launch this thing? Okay, I see. Okay, okay, okay. It bounced off the sun and, uh, ricocheted out; missed. What am I trying to hit? Am I trying to hit the—oh, I think I'm trying to hit the green. Also, am I crazy, or should the planets be rotating around the sun? Okay, so if I'm trying to hit the—maybe I'm trying to hit the red. Let me see; maybe I'm supposed to do something like this. Nope, like this. So if I go around the green. All right, so it got some things right. Okay, so it's saying the two red circles are the ones to hit; they turn green when hit. Okay, I get it. The problem is you're not supposed to hit them; they're like, you fly through them. Okay, so maybe you were supposed to do something like this where you go around. Okay, okay, okay, okay. Yeah, oh, did I nail it? I nailed it. Okay, this gets a pass in my book for sure. So basically, so it's kind of like galactic pool. Okay, no, yeah, you pass right through it. That's phenomenal; that's terrific; that's exactly what I wanted. Okay, and if we wanted to shoot from way out here, it would look something like this. Let me see if I can make it curve and hit both. Not quite, but it really captured what I was looking for, except these are supposed to be rotating, but okay, maybe I should have put that as part of the prompt, I guess, now that I think about it.
Here comes 04 Mini High. So here they're rotating, and I think it's choosing, it's choosing the red as our sort of, uh, planets to hit. So okay, so here I can do like a lot of different probes, but I can't really set their speed too well, so they sort of start at a slow speed and start traveling towards the middle. Uh, okay, not quite what I was hoping for. The other thing that I think would be useful here is to add a slider that sort of allows you to speed up things, but I got to say there's a lot of things here that I like. So notice that all the probes are kind of falling into orbit; they are interacting with the gravity of the planets; are they? Maybe not so much. They do seem to be interacting with the sun's gravity. All right, so this is okayish. Unfortunately, the, the thing about it, like ping-ponging and rocketing off into space, we're going to see that quite a bit. I've, I've noticed that's it has a tendency to do that, but overall, it's okay. I'll give this a B.
Next, we have the 03. 03 crashes. Next, we have the Gemini. Let's see; Gemini made a huge simulator. Wow, really needed to make it bigger than everything else. So hit both green targets. I don't see the—oh, here's the other green target. Oh my goodness. Okay, so let's see; so this, click to launch probe towards cursor. So if I click right here—oh no, Gemini, it's not doing anything. Try it again. So we click to launch it towards cursor; I still not figuring it out. So that didn't work, but let's try Claude 3.7. Wow, Claude is all action. Okay, so I'm able to sort of change it like this, and then space to shoot. Okay, R to reset. I, I'm liking this. Okay, so I'm liking this a lot. The only thing is it doesn't seem to be affected by any gravitational pulls or anything like that. Yeah, it's not being affected by, uh—so I like a lot of the things about the graphics and stuff like that, the, the, the whole gravity simulator thing, not great, but pretty interesting. So I think it was the 03 Mini that actually was the one that did the best and actually did surprisingly well, although, um, not perfect, but definitely got a lot of the things right. I feel like.
All right, now let's try this: create an autonomous 2D soccer game. It's 3v3, and each player has stats that are displayed at the top. They also have XP that they get for playing well, and once they get enough XP, they level up, and their stats improve. Have a mechanic where they're able to steal the ball and knock the other player down. Have a scoring mechanic: when the ball hits the net, briefly slow down time and create a screen shake effect; so something fancy for when we score a goal. I also have a scoreboard that tracks goals; each goal resets the players and the ball.
All right, we'll start with the 03 Mini. So it's looking pretty good; everybody just bum-rushes the ball, but as you can see here, they do have levels; they do have XP, and, uh, the basically whoever kind of takes the lead rapidly kind of snowballs out of control and becomes better and better. Do we have an actual scoring system, however? Yeah, I feel like one of the players became level—now he's level 16, 17, so he's going to be hard to beat, and now it's just becoming a whole giant mess, but kind of interesting; it's kind of a tornado-like thing.
Now here's 04 Mini High. So this one actually has a scoreboard. Okay, oh, and it captured the screen shake effect; very good. So one thing that I don't like is the players kind of bunch up, so you're not able to see where they are. One of the things we should probably ask for is that they can't bunch up. Also, why does the blue start off to the side and the red in front of the, the goal? I feel like that gives red a pretty unfair advantage. 03 was not able to do it. All right, here's a Gemini. Uh, it created multiple files; I forgot to say make it all in one file, so I had to prompt it to put it onto one file, but after it's done, I mean, this is looking pretty good. First of all, it's got great kick mechanics. You know, there's some issues; I mean, if we had to redo this, we would follow up with some things about how to spread players apart, have some sort of formations, but notice it's got the leveling system; it's got the XP system; we have what seems to be strength, speed, and accuracy, and as these players are leveling up, they're beginning, they're getting better and better at doing those things. So this is Gemini 2.5 Pro; I would give this an absolute A+; it's phenomenal; it's, in a word, go. Yeah, I'm very curious to see what happens if you just let it run, and, uh, the speed just keeps increasing more and more and more and more. Do you get to a point where this is just, uh, super rapid and out of control? That'll be interesting to see. And here's Claude with its version of the game. Oh, crashed. Distance to ball, some issue, uh, forced it to crash. Um, I mean, before that it was looking pretty good, but yeah, some bug is forcing it to crash. Don't take these too seriously; this is just one round of testing. I'll say one thing: Google is not going down without a fight, it seems.