Transcription
So, OpenAI just dropped GPT 5.5 and it is really solid. But the question is, how solid is it? And the other question is, is it more expensive and how does it compare to Opus 4.7? Because obviously, you can look at the benchmarks, which there are some pretty impressive things, especially when you are comparing it to Opus 4.7. But of course, what we want to do is actually get in there and get our hands dirty. So, I ran a bunch of experiments and I'm going to share with you guys what actually matters when you think about the cost and the speed of these different models.
So, we'll come back here to the results and to the experiments that I ran in just a sec, but real quick, let's just go over some of the release details. So, GPT 5.5 is OpenAI's newest model, the most recent version being GPT 5.4. This is OpenAI's new flagship model. It was codenamed Spud when people kind of saw the leaks and stuff, and it is being positioned as OpenAI's smartest and most intuitive model to date, and it is a purpose-built step toward AI and enterprise computing.
Now, what's really interesting is that the pitch isn't that, hey, this model is just better at everything. The pitch was kind of that it does more with less. So, fewer tokens per task, less handholding, and more autonomy. And that's what really got me thinking about, okay, how could I run some experiments to see if that's actually true? Unfortunately, right now, we don't have the API. It will be coming. It's in Codeex. It's in ChatGPT. I'm sure by the time you're watching this video, the API will probably be out, but I saw the announcement and I just jumped on it and wanted to play around.
So, a quick quote from the president of OpenAI: "It's faster, sharper, and it uses fewer tokens compared to something like GPT 5.4." Now, let's see where it pulls ahead a little bit. So, the Terminal Bench 2.0, it scored an 82.7, whereas GPT 5.4 scored a 75.1 and Opus 4.7 scored a 69.4. So, it is beating Opus here on the terminal bench. And it's also beating the previous model. They also ran an expert bench internally. And if I actually go to the actual site here, you can see that they didn't compare this to Opus because it was internal, but they did compare this to GPT-4. And you can see based on this chart, once again, it is doing more and it is costing less output tokens. And the output tokens are the ones that really matter because they are more expensive than the input tokens.
So, what else do we got here? Harder problems, better answers. So, there's a few other evaluations here. We've got the GDP val, which is for knowledge work. We've got the Frontier Math and we've got the Cyber Gym. And you can see here GPT 5.5 is beating out Opus 4.7 and Gemini 3.1 Pro in all of these, except for this one down here. We didn't pull in Gemini 3.1 Pro. But right here, these benchmarks alone are very impressive, especially this one. There's quite a big gap right here. Now, it is important to note that the SweetBench Pro still does belong to Claude Opus 4.7, and this was being able to pull in real GitHub issues and resolve those. And that's why it's all about actually putting these models to the test and seeing what you think.
Now, the other important thing to notice is that the price has doubled compared to GPT 5.4. So, it was 2.5 in and 15 out, and now it's five in and 30 out, which is actually a little bit more expensive than Opus 4.7. Opus is the same on the input, but it's about $5 cheaper on the output tokens. But, of course, apparently GPT 5.5 uses less output tokens, so it's going to be cheaper either way. And that's what I wanted to show you guys in my experiment.
All right, so four quick things that really do matter. The token efficiency: same quality output with fewer tokens. Autonomous decomposition: so, it's taking vague prompts, identifying what's unclear, and then taking the next steps and executing. So, when I ran this experiment with all four, I shot off one prompt, and I didn't let it ask me any questions. I just shot off a prompt, and I wanted to see what its first iteration looked like. This also comes with some Codeex upgrades. So, things like tool calling or multi-agent parallel execution, reusable workflows, stuff like that. They also talk a lot about security. So, if I go back into the release blog real quick and I just go all the way down to the bottom, obviously we saw stuff with Claude dropping Mythos and talking about security, but right here: "Advancing cybersecurity for everyone's safety. These frontier models are becoming increasingly more capable in cybersecurity, and these capabilities will become broadly distributed, and we believe the best path forward is to make sure that they can be put to use for accelerating cyber defense and strengthening the ecosystem."
And then finally, we have super app positioning. GPT 5.5 is the intelligence layer for GPT Codeex and Atlas. So, they're trying to sort of obviously have this whole ecosystem. All right. So, what is no longer here in the lineup? Apparently, we got these kind of models cleared out. So, no longer using these models. And if you're using GPT 5.5 within Codeex, you will be getting a 400,000 token context window compared to Opus 4.7's 1 million token context window.
All right. So, before we get to the experiment, four quick takeaways for founders and creators. One, agent coding got an upgrade, but Anthropic still leads real-world suite. Two, the price doubled. So, look at your unit economics if you want to make a switch, specifically if you wanted to make a switch from GPT 5.4 to GPT 5.5. Number three, OpenAI is a platform play. Number four, release cadence is a content liability. Six weeks from 5.5. Anything model-specific ages really fast, and that can make it really stressful, really overwhelming in this time where every company is shipping. Every company wants to compete with each other. And with every new release, it's apparently like changing the game or the benchmarks are skyrocketing. So, that's just why I think it's important to not just think which model is the best, but think about, okay, for this use case, what are the benchmarks that might align here, and how do I actually run a quick experiment between two models to see which model fits me best here?
So, there's a few quotes that people are saying. I'm actually not going to read these off because you can find them on the blog or you can pause the video and look real quick. Obviously, it's not the full thing, but let's actually go into our showdown of Opus 4.7 versus GPT 5.5. We ran four different experiments. We did an AI-modeled personal brand website. We did a solar system type of simulation. We did a space shooter game. And we did the finale, which was just building like an actual ecosystem type of simulation game. I'll show you guys what I mean by that, but let's hop into these experiments.
Okay, so the first one was building a personal brand site. So, this was the prompt that I used, and it didn't ask me any questions. It basically just oneshotted this, and then we were able to get an output. So, let's real quick first look at GPT 5.5's output. And one thing I do need to preface is I'm using GPT in Codeex and I'm using Opus in Claude Code. So, people might get mad and argue that this isn't a model comparison. This is an agentic coding harness comparison. I mean, sort of, yes, but obviously the model is what is powering it. So, I just wanted to clear that up.
All right. So, here's Codeex's personal brand site. We've got Codeex. We've got a really cool dynamic thing going on in the background. Got a nice drop shadow behind this. A nice little interface. "The model in the room. Precise enough for code. Curious enough for messy ideas. Calm enough to keep going." Let's click on "How I Think" and see where it goes. Okay. So, we've got another little context map visual. We've got this mode console that's apparently interactive. Okay. I'm not exactly sure what that's changing, but you can click on those buttons. We've got a verification loop, and it says, "I prefer evidence over vibes. Run it, render it, read the diff, cite the source, or admit the uncertainty." Let's see what we missed up here. "A working partner for the whole arc." So, clarify, build, verify, adapt. Very nice. If we come down, um, "Keep the goal visible." Okay, we can expand these or close these out. "I'm not a search box with better manners." And then here's the end. "Bring the question. I will bring the shape. Design, coding, writing, reasoning." Okay, cool. So, that is what the personal brand site looks like for Codeex.
And then here is what I did for um Claude Code. Pretty much the exact same prompt. I did say "don't use superpowers" because I wanted to see sort of as raw as we could get. It did use the front end of design skill, which I think is fine, but same prompt. Now, I know I could have done this in the uh Claude Code desktop app because I use the Codeex desktop app. I don't know why I decided to do VS Code just 'cause it's so natural to me. But let me pull up this site.
All right. So, here we are with Claude's site. We've got a nice little scrolling banner up top. We've got another kind of animated visual element right here. "A million context window and some latency stuff." "A thinking machine in the margins of every idea you're trying to finish." Something's weird going on with those Fs, just the font, I guess. But this looks really nice. I think they both looked from a design perspective pretty solid. So, what can I do? "Give me a messy brief and I'll find the clean shape inside of it." "I read whole code bases, argue with spreadsheets, rewrite paragraphs, blah blah blah. I can reason through hard problems. I can write like a person. I can write code. I can find the signal in your spreadsheet. I can build a thing end-to-end and explain it like I've seen you squint." So, I don't love the way that these kind of change font and and uh like compress. Also, I noticed if you click on these, it just takes you back to the top of the site. Not exactly sure why that is, but "How I Think." So, this is kind of a cool visual. It's like a little sort of animated token thing. I think that's a nice touch. "I speak in tokens, not words." Okay, thank you, Claude. "Every word looks at every other one." I think this is cool. So, inside the model, each token weighs its relationship to every other token. Thicker lines mean to pay more attention. So, kind of like looking at the weights in here. I think that's a nice little visual. "A million tokens of memory." So, it also shows us like system instructions, your code base, reference docs, working memory. I think this is a very nice illustration here. And then we have this little thing down here. "I branch before I commit." So, it's got this nice visual here. I'm not exactly sure what's going on. It's showing us a chosen path and a literal and a lateral. And then we've got some other things going on down here. Just the way it gets to its best answer, I guess. So, that is the difference between what Claude thought its personal brand was and what Codeex thought that its personal brand was. They definitely have different feels. I mean, this very feels like OpenAI branded and this very feels Anthropic branded. So, pretty cool.
Okay, so all of that stuff is pretty subjective. So, let's look at the actual statistics here for this experiment number one, AI model site. First of all, look at the speed. GPT 5.5 was about 4 minutes and Opus was about 14 minutes. Look at the difference between the input and the output tokens. I mean, this one was way more on both ends. Also had more requests. And you can see that this costed GPT about a dollar if we were doing API billing, and it would have cost us almost $5 with Opus 4.7. And by the way, if you guys are curious how I was able to get those stats, I basically just asked it to look within its JSONL files, and it was able to give me the time that it started, the time that we came back, and all the tokens. And it worked pretty much the same way across using Claude, but also if you do the same thing in Codeex, it can look at those logs and get that information for you as well.
But anyways, number two was a solar system kind of simulation. So, this was the prompt. Once again, gave the same prompt to Codeex over here and to Claude Code. So, let's go ahead and pull up the results. We'll start with Codeex.
All right, so here is Codeex's version. The first thing I noticed here is that this looks kind of squished. Like the aspect ratio here doesn't look great, but I like the rest of the design. You can click on these planets and you get sort of like, you know, some information over here. And you can speed up the simulation. So, if we go to 100x, we can see all of the things orbiting. The sun has a weird like box around it. So, from a design perspective here, not the best, but as far as like functionality, this is what we were looking for.
Now, if we open up Claude's version, we can see we have that slider down here. We can go up to 100x. Visually, this one looks better because the aspect ratio of everything looks okay. The sun has a glow that makes more sense. You can click on the planets and you get a um description over here. I also like how when you click on the planet, that orbit ring is like the only one that you can kind of see, which is cool. Um, so once again, this is all subjective. Personally here, I like um Claude Code's version more, but let's take a look at the actual stats.
Okay, so for the solar system one, this timing was a little bit more close. Um, Opus finished about a minute later than GPT. We can see from an input token perspective, GPT took more, a little over double, but from an output token perspective, GPT took less. But this one still ended up being cheaper from Opus 4.7 by roughly a dollar. So, in this case, Opus 4.7 definitely wins because it was cheaper and I liked the output better.
Okay, let's move on to number three. We have a 3D space shooter demo. This was the prompt that I shot off. Once again, no iteration, no back and forth, just a first oneshot prompt. And let's take a look at what we got for both Codeex and Claude Code.
All right, so here is GPT's version. We can use our mouse to look. We use W, D to move around. We can use shift to boost. And we use space to shoot. So, basically, these games are shooters, right? So, I'm able to shoot at these um what are they called? Comets. Asteroids. And I can move forward. Um, I can use shift to boost. And the physics here feel very nice. Like the speed, you can see at the bottom my velocity. If I change directions, that goes down. This feels very smooth. Um, the sound effects are a little weird, but as far as like playability, this feels pretty solid. You can see we have a whole integrity. So, if I just crashed into one of these, I'm assuming that would go down. Okay. Yep. Very nice. And our score goes up when we shoot these asteroids.
Okay, now here we are in Claude Code's version where I can go ahead and initiate. It looks like the controls are pretty much the exact same. Now, right off the bat, this feels a bit clunky. It feels a little bit less smooth. The sound effects are a little better, but I don't like the feel of this one at all compared to the last one. It's like hard to control. My mouse is snapping. It's It just feels buggy. But everything else, honestly, like it's about the same. From a design perspective, it's harder to see. So, I definitely think that GPT 5.5 wins this one. Like, no doubt. And obviously, that was all subjective. So, let's take a look at the stats that matter.
GPT finished in more than half the time. It also used less input tokens and less output tokens. Had a few more tool requests, but that doesn't really matter as much. With GPT, this would have costed us under $3. And Opus 4.7, it would have costed us 4.5. And we would have had to keep making changes and iterate. GPT 5.5 definitely takes the cake here.
All right, and now for the last one. This was a big boy prompt. I dropped in this entire thing. So much bigger, much more to think about. And let's go ahead and take a look and see what we got.
Okay, so here is our simulation of Codeex. This is our living ecosystem. We can see we have population, generation, fitness, and some other stuff. And I can speed this up and I can watch this ecosystem evolve. Looks like right now it's Okay, it's starting to drop. Let me actually slow this down. I can drop in some food, I think. So, sprinkle food. Oh, the button's up here. Okay, so nothing seems to be working when I'm trying to sprinkle food and I'm trying to spawn more. Nothing seems to be working. And unfortunately, the population is dying out. So, I'm real quick going to go back into Codeex and I'm going to ask Oh, wait. But that's really interesting. In this version, the population looks like it's making a bit of a comeback, but I guess I just don't really understand how I can control this. So, I'm just going to go ahead and ask real quick. Ah, okay. So, food only works on grass or forest. Spawn only works on land. I didn't necessarily want it to make a fix because I wanted to see how good it is just by default. But let's see if I'm able to spawn some food in here or if I'm able to spawn some creatures. Okay, so I'm able to spawn creatures, but maybe it's just not visually showing me that I'm sprinkling food when I'm trying to. Well, anyways, for a oneshot prompt, it's not great. But let's also go take a look at what Claude Code was able to do on one shot to compare.
Oh, sorry. Also, one more thing you can do is we can click on a creature and we can see kind of like how it's evolving. We can see the energy, the age, the size, the speed, radius. We can we can just look at them in different ways. Um, kind of interesting. But anyways, here is Claude Code's version. So, it looks similar. I think honestly like aesthetically, visually, this one's makes more sense to me. I can kind of see the land and I can see what I can do. Sprinkle food. Let's see if I can drop food in there. Nice. I can spawn creature. I can also spawn creatures in there as well. I can save the top genome, whatever that means. Okay, that just downloaded some sort of JSON file. But when I was playing around with this, what I've noticed is I think that there's something wrong with this code. For example, all of the creatures are just staying in one fixed location. I don't even think they seem to be doing anything with the food. They just seem to die and then they just die and then the population hits 10 and then it just sticks at 10. So, if I speed this up, you can see that this is going to hit 10 and then it just stays. So, I don't know exactly, you know, obviously there's an issue with the code. Once again, this was a oneshot prompt, but if I add more, it just goes back to 10. So, there's something in there that's stuck. The logic isn't working. And if I switch back over to Codeex's version, it's kind of a similar thing. Like, it just doesn't actually work. The logic doesn't work in order to make this like a legit simulation. It doesn't seem like at least. Or maybe I just need it to speed up faster where we actually see something. So, let me speed this up and just see if it can go past 10 at all. Yeah, I mean, I don't know what's going on. Look at this poor soul. He just died. But, um, yeah, I don't think I'm able to sprinkle food. So, so anyways, none of those are perfect. And what we would have to do obviously is we would iterate and we'd come back and we would give some feedback.
But, let's take a look at the stats here. Kind of interesting. So, GPT 5.5 took almost 10 minutes and Opus 4.7 took about 12 minutes. Now, GPT 5 took way more input tokens here, like double. But also, look at how many output tokens it used compared to Opus 4.7. Now, it's interesting because GPT 5.5 still costed us more here because of how many input tokens it took. But that is really interesting to me that it's able to only output about 28,000 tokens. And we're still able to get an output that's basically the same. Like this is basically the same as this. Like pretty close. But it took a fraction of the output tokens, which is just really interesting to me.
Okay, so if we zoom out and we look at all of these statistics as a whole, we did four total experiments. Total runtime between the two was about an hour. And GPT total runtime was 20 minutes 49 seconds. Opus total run time was 40 minutes and 43 seconds. So, basically double the run time. Input tokens, they were honestly pretty similar. 2.7 million over here with GPT and 2.5 million with Opus. But the alpha tokens tells a very interesting story. It's about 70k compared to about a quarter million. And then total cost, it honestly came out to be pretty even, but GPT was about three bucks cheaper over these four experiments. But if every single time you do it, it's a fraction of a bit cheaper with GPT 5.5, then in the long run, it's going to get more and more cheap.
So, I won't dive into all of these other graphs because I think we've already kind of understood the differences here. But GPT 5 is green. Opus 4.7 is blue. And for the majority of the time, GPT is in the position that we would rather be in. Meaning, it's less and it's less and it's less and it is less. I could have just said all four that GPT was less. But anyways, that is going to do it for today. I hope you guys enjoyed this little experiment. I really encourage you to just hop into ChatGPT or, you know, plug in GPT 5.5 in your automations via API or maybe even hop on Codeex and play around with it a little bit. I am a firm believer in not one tool being the best, but figuring out which tool to use for your use case. I've been playing around with Codeex a lot. I just haven't started making YouTube videos too much on it yet because I don't really want to overwhelm you guys with too many things and I know I've been uploading a lot lately. So, maybe you are overwhelmed either way, and there's so many new features. But like I said, just try out new tools.
So, if you enjoyed the video or you learned something new, please give it a like. It helps me out a ton. And as always, I appreciate you guys making it to the end of the video and I will see you all in the next one. Thanks everyone.