Transcription
01 Pro is Chad GPT's newest model that costs $200 a month, and let me tell you, I tried it for coding, and it's probably one of the smartest models I've ever had a chance to play with. So, in this video, I'm going to go over the exact code that I asked it to write. You're going to compare it to Claude, Gemini, and even to regular 01, and by the end, I'll tell you whether you should get it, whether you should not get it, and whether or not it's worth it.
I just upgraded to Chad GPT 01 Pro. I was pretty hesitant; it's $200 a month. I saw people saying that it was really good, and within about 5 minutes, I could already tell that it is genuinely a step function above any other model I've used. I'm going to show you exactly how I know and how I got to that point.
I tried this code with the same prompt, same everything, with every model available: Gemini, Claude, 01. They all did not even remotely come close to the answer that I got 01 Pro to do. So, what were we trying to do? We had a code base that we needed to refactor, specifically our front end. What we needed to do was move towards a data grid, which is this sort of Glide app. For our product, it's basically a spreadsheet with a bunch of updates, and in the product itself, there are a lot of things happening.
Now, I did it in a really bad way, and it caused the whole app to be laggy and pretty much crashed. We had a bunch of real-time updates coming in, and we were using Electric SQL. The caveat is that the docs and the like—none of this is in the training set—so I have to explain to the models what it is and what's going on.
Here's the code base: there are a couple of files; it's actually all in this giant prompts MD folder because there are just so many intertwining files. I had to include it here; I pasted docs, I pasted a bunch of stuff, and it's around 14,000 tokens of files and stuff. We have this Ed sheet hook; it uses a shape hook from Electric SQL, which then sort of uses a React useEffect, and pretty much this is just crashing it.
That file is about 200 lines of code. We have this other one, which is about 850 lines of code, and then we have another one here—I forgot where it is; there are so many lines of code—it's about another 250. So, about four files that are all pretty intertwined that need to be refactored. There are a bunch of other things as well because we're using React Query.
Anyways, the point is it's a very long and messy code base right now. What I wanted to do was just basically say, "Please fix it for me; I don't want to do this; this is pretty annoying." I gave the same prompt to all of the models, and this is what they gave to me.
So, I'll go first to 01 Pro mode, and the devil is sort of in the details here, where you kind of have to know exactly why there are performance issues or what issues are happening to be able to say, "Oh wow, this is a good answer." So, here it thought for a minute and gave me a very, very detailed answer. It says, "Below is a set of guidelines and instructions," and it gives me the detailed answer. It says to remove the logic for the query said client, and it's because that's what's causing the renders.
The nice thing is I am getting rid of the useEffects, and then it says to go to this file, make these changes, and then it tells me what the things to change are, what the updates are, and so on. Then, I say, "Can you implement it?" It takes another 3 minutes and 24 seconds to do it.
Okay, this all looks pretty good; I don't have to go in and implement it. But even before I implemented, let me show you what Claude did and what Chad GPT did. So, this is 01; this is not GPT-4; this is just 01. Let me go here. It gives me a pretty good answer. That was the code that I pasted. It says, "Thanks for only 14 seconds," and it already starts to say, "Okay, pass it some refs." It doesn't mention that I should remove the useEffect hooks, which is actually what's causing the rendering, which is causing it to be so laggy.
So, overall, I can already tell at a quick glance that this doesn't seem to be the right answer. It tells me a half-right answer, but now, okay, so we had 01 Pro, we had 01, now let's go to Claude. And this is Claude's answer. I don't really know what to say; this is the output you could come up with, which I'm going to tell you right now is worthless.
Now, let's go to Gemini. So, Gemini actually—this is the new 12.06 model that came out today—and I can already tell from a quick glance that this model's quite good. It says, you know, I gave it a prompt, really basic system instruction, so I tried to keep it simple, and I gave it the exact same prompt, right? I said, "Okay." It tells me what I need to do, but then it says here, for some reason, it tells me to create another shape stream, which I don't think I need.
Still, you know, we have this, so it's not exactly what I'm looking for. I can already tell at a glance that we have a useEffect; this is pretty much what is causing our app to be laggy. So, already this answer is discontinued. Then, I also had it in Cursor, and I can actually just go in. I tried it here, so I can say here, I go to Composer, and here it gave me a solution, but it just—yeah, honestly, I'm reading this, and this is worthless.
So, for my initial test of this one task, and I haven't actually implemented yet, but the nuance and detail that 01 Pro mode provides is genuinely a step function above anything I've used yet. This was a really, really complicated sort of refactor—not in the sense that there are like 500 files, but because, as you saw, there are a lot of optimizations to be made when to use useEffect.
I also remember I pasted these docs; I actually pasted in the prompt a code example of how you're supposed to do it, and to me, this just says, "Okay, this model is actually thinking about what it is answering," because I'm reading through it, and a lot of this makes sense. It actually just gives me the whole written code as well, which took like 3 minutes.
So, overall, I am very, very impressed. I would say that it's very—like, I'm not sure when you would need the $200 a month. I think if you're doing Greenfield starter projects, maybe not. Honestly, I do think it's worth trying it out; I think it's worth paying the 200 bucks even for a month and seeing if it's useful. This feels like something else; to me, this is a pretty powerful model.
Hopefully, this kind of showed you the kind of task that I'm doing, which is not trivial by any means. You have to read doc referencing, you know, what are the best things to produce renders, and there are two different docs that, again, are not even in the training set, right? It doesn't know about these things, and it does a really good job.
So, yeah, that's my review. I will add that I think the new Gemini model, again, just like a quick reading over it, is pretty good. I think it was able to give a pretty okay answer, and I'm actually pretty impressed. Again, I'm really not impressed with Claude; like, what on Earth is this answer? Yeah, that's it.