Transcription
The month of August has been insane. We had GPT5. We had Gro Code Fast. We also had quite a few new agents. I'm just going to jump right into it. There's a lot to get into it. I ran an enormous amount of tests to actually bring this to you this this month.
Okay, first off, we have the new agent, Kira. Pricing of this looks kind of reasonable. They're actually charging per request, which I I kind of appreciate. It is just another VS Code clone, though, so worth checking out maybe, but I I'm not blown away by it yet.
We then have coder. Coder I'm a little bit torn on. They don't have a model selector. We don't know the pricing of it, but we know we think that it's powered by Quinn 3 coder and it does a a pretty good job overall.
Then we have the augment CLI, which when I first saw this looked a lot like Gemini CLI, so I thought it was probably cloned off of that. I don't know if that's the case. I don't think that's the case. But anyway, we have this as well that entered the market.
I also ended up playing around with claude code router. This was a request I got a ton. So I ran the models that I'm going to talk about here through cloud code router and it was a little tricky to get set up. In particular, there's this weird nuance where if you're logged in to cloud code in some way, you can't actually use it. So you actually need to be logged out of cloud code to actually kick this off. Took me way too long to figure that out.
All right. So, as you know, my tests are really about instruction following using unit tests to verify that it did what it was supposed to do along with linting check, static code analysis. I also have an LLM as a judge, which counts for a smaller portion of the overall score. My goal is to do much more complex projects. You know, 10, 12, 20, 30 files that need to be touched in some way.
One thing that I've learned though is that these agents and these models have gotten a lot better at navigating my tests. So, I've had to retire some over the over the period of time and create new ones. In fact, this month I actually tried to make two of my existing ones. I didn't have time to like add in brand new tests, but I made two of my existing ones actually more robust, like more features, more unit tests. So, the scores will vary a little bit. In reality, I don't think it did as it didn't add enough complexity. I don't think I think I need to kind of tune that in a little bit more. Uh, which you'll kind of see the scores just converging. I think they've just gotten so good, especially with the claw force on it models.
Just some notes here. I do think model knowledge is starting to be incredibly important. What I mean by that is I think a lot of the models that I'm testing here are are pretty good to great at tool calling and instruction following. Specifically, GBT 5. If you tell it what you want to do, it will do it incredibly well. But if it's lacking knowledge about a framework, a language or something like that, that's where a lot of the issues start coming up.
One of the other issues I ran into a lot of times is the model would have model and agent combination would have something correct or close to correct, it would get an error and then it would fix it in the wrong way and that kind of throws it off track. So I I think my tests do a really good job of testing that sort of iteration loop. Grock code fast unfortunately is actually really bad about that. If it hits any error in the if like it runs something in the terminal to check what it did, what'll end up happening a lot of times is that'll throw it down a path of just making kind of dumb changes.
To be honest with you, I am working on a v3 version of my eval suite. I really do want to try to open source a version with public test. I really want to help get the community to help me start writing some of these because it is very timeconuming. In fact, this is the most time that I've spent on any series of tests since I started this.
And then as always, I do think the model matters the most, but I do think the configuration of model is also important and the harness just helps bring out the most out of the model. So, here's an example of a retired test. So, you can kind of see the number of things that it's asking. This is a very simple Python one. So rather than just saying, "Hey, go fill out this function or do this small little thing," it actually is asking it to build a full-fledged application. That's the difference between what I'm testing and I don't really see a lot of other ones doing things like this. And that's what makes it so timeconuming on my end to actually run these. But I think it's worth it because it gives us some really good data points that we can then take and build on with additional data points that we get.
Now the models we we basically always test claw for sonnet. I don't know if it's worth testing it much anymore. Everything's kind of converged. They're all pretty good there. Clawed Opus 4.1. I've had a lot of people request it. This model is incredibly expensive. So, I did run it on a few. I think I ran it on a couple couple different ones. I didn't want to pay a ton of money for that one. It is usually, in my experience, slightly worse at the type of test that I am doing. I do actually, you know, subjectively prefer Opus 4.1 for planning, but from a daily coder, I actually find it slightly worse and you're actually a little bit slower. So, you'll see some of the scores from that and you can kind of make your own opinion on that.
I did test a little bit of 2.5 Pro just because we have the Gemini code assist thing and then we've got the uh Gemini CLI. So, I wanted to test those. That's really all I tested that in general. 2.5 Pro I don't think is relevant. Don't shoot me for this. In a lot of the other AI coding systems, we just have better things on the market now.
Quint3 coder still one of my favorite, if not my favorite open- source coding model. I really do hope we get a model that's in between the 30 billion parameter one and the 480 billion parameter. I would love to be able to run that here locally.
GD5 is a tough one because that model is great, but there's a a big range between minimal and high. So, what I typically did is I tested medium if it was selectable. I then some of them don't allow me to select it. So, I just have to assume it's default. Uh, which sometimes it felt like it was maybe high reasoning, sometimes it felt like it was medium reasoning. You don't really know in a lot of these AI coding agents. So, I did tag that specifically, but it should give you a little bit of a points of comparison to see how it's actually working in these AI coding assistants.
Grocode fast, it is incredibly fast, and I was really excited to see how this thing ranks. And to be frank, I think it did better than I anticipated. It does have cash reading, which I think is awesome. This model might be the value king in my opinion. Like if you're on a tight budget and you're just trying to like code a little bit, you don't want to pay a monthly subscription, this might be a model worth looking into. Like I was able to do some tests with it uh for pennies. So you could probably get this to under, you know, 50 cents a day if you're just doing some light coding and stuff like that.
Now, Claude 4, surprisingly, augment has actually made its first entry. Again, I do want to be like caveat this, but I think the Claw 4 scores kind of all converge so much that you're just going to have people jumping around back and forth slightly. Whether it's like runtime differences or whether it is something else like slight improvements here or there, it's just something to note because augment has traditionally always been on the lower end of Claw for Sonnet, but they boosted way up. So, whatever they've done over the last month period, they've made significant changes.
Cursor also is doing incredibly well here. And again, these scores are so close to each other, but this one didn't really surprise me. cursor usually does good with Sonnet and it bounces around in and out of the top three, but this next one actually surprised me a lot. Warp Warp has never been number one has never it's never been this good with Sonnet. And for it to just jump from where it traditionally was, which is I would say on the lazier side where it wouldn't actually complete the entire project that I actually gave it to now being the highest score. They've definitely tuned this in some. I hope they keep whatever they've been doing with it to actually because this is insanely better than where it's been in the past. So, warp is a big surprise to me.
Now, overall, you kind of look at all the scores here. Just to kind of touch on things, you can kind of see here pretty much all of them are 25,000 and up. We have IDER and Trey actually kind of sitting below Claude Code with Sonic 4. This is the thing that kind of blows my mind. And I don't really know how to actually think about this because Cloud Code used to be one of the better ones and it has actually come down to being 24,934. Now it's it's behind like Windsorf and Kira and Crush and and a part of me wonders if that just has to come with the nature of them trying to conserve tokens like is it are they trying to take shortcuts or there things that are actually happening differently in call code or is it just the the other agents have kind of caught up with that? I don't know. Regardless, call code is still incredibly good. is still an incredibly good value, but it's surprising to me how much it's fallen. It's fallen from being like number one to kind of like in the middle of the pack to kind of down at the bottom now. So, I do want to call that out.
GPT5. This is one I was very excited about. So, note that I've actually put default here at the top because in cursor I just picked the default version of GPT5. Felt very slow, honestly. But a really respectable 25,570 score for uh number three. And then warp did great with GPD5 and they did allow you to select reasoning. So I selected medium reasoning here and the speed felt okay. Like it wasn't it's still slow. A lot of these agents have it so slow compared to codeex which is painful cuz you'll put this in like tray or one of those and it just runs forever. And when you're testing hundreds of these things like it it's really painful for me. Maybe on a day you know if you're just using it as a your coder maybe it's not that big a deal.
Number one, codec IDE. I say IDE, it's not actually an ID, it's an IDE extension. It just did great. Uh, I actually put high here. So, I did kind of combine the few high tests I did in here. You'll see the codeex ID extension medium in a minute here, but the high actually did score, you know, reasonably higher than the others. And I actually really kind of starting to like the codeex ecosystem.
So, overall here, just a few things to call out. Open code uh unfortunately fell to the bottom here. Now I was using the open router endpoint into open code what would happen a lot of times and I wonder if this is like an environment issue and that's another thing as I've kind of zoned in on my test I've been starting to notice some of the issues these agents hit are environmentbased to where you know they'll try to do things like run a PowerShell command or try to do and they fail at that. So I do think there's some variance there. Now, this open code one I actually ran in WSL. And what would happen a lot of times here is it would just stop mid mid track. So, I had to throw away a lot of tests for the open code version because it would run and then just give up on it. Maybe that's just a blip. I don't know.
But augment actually had the same problem. The augment um extension using GPT5, it would just give up. So, it would have like weird tool call failures and things like that and it would just give up.
Now, let's take a look at the rest of them. So, we already talked about the top three. Codeci, GPT5, Medium. I've actually really been liking the CodeCli lately. It is still very bare bones. I did hear that they've actually released some of the new features that were missing there. So, it is getting better. Um, Zed surprisingly did incredibly good. Trey did great, but it's so freaking slow. Like, so slow. That one drove me nuts. Like, it took me forever to get that one. IDER did actually worse than I thought. I thought IDER because of the way that GBT5 was actually going to be set up and the way IDER works, I thought it was going to actually score a little bit better. Crush also did did pretty good. I would like to figure out what's going on with Open Code there. Uh, because I think this score is not conducive to what I would normally get. Uh, in general though, I would say most agents handle it fairly well. There's a few that I'd say just are a little bit lazier. they end up with environment issues etc. uh all the normal stuff.
Now quint coder I am actually calling this out because by the normal way that I would want to test this is just select quint3 coder 480b on open router and let it rip but quint coder and open code works phenomenally if you select fireworks and it does not work that great if you just do a default and you get whatever provider which is you know flip of the coin there.
claw code. Uh, this is using the claw code router. 25,914. This one surprised me a ton. I did not expect it to do this well, but it's worth kind of like bringing up. I appreciate everyone kind of forcing me to test that. But that one blew my mind a little bit how well Cloud Code actually work with Quint 3 coder. And then, of course, Quincode is still number one. Whatever they've done there to tune their prompts, it just works pretty dang good with uh Claw or Quinn 3 coder.
Now, let's take a little bit of a look at Quint3 coder. Overall, uh I've accidentally included some GBT5 ones in here. We can ignore that. So, really number four here is root code. Number five here is kilo code. We've then got clin is the number six cursor crush open code. Open code without the fireworks setting. So, just default whatever it is. So, who knows what provider that actually was at the time. IDER does not work that great honestly with Quint Coder. And I would say co-pilot does not work that great at all with quint 3 coder and zed did not work. This is again I can't really select provider and zed. So it could be zed works fine with subproviders but zed literally would do like two things and then quit. Two things and then quit. It just I that one ended up scoring a zero.
Now another interesting one. Grocode fast. Root code number three. I do know R code actually put some special attention into getting grocode fast working. They've actually I think they changed um some of their tool calling to particularly work well with that model because they were having some issues with other ones, but it it shows 23,170, which if you think about it's actually a really solid score for a new model that is as fast and as cheap as it is.
Cursor did incredible with this model. And it again in these like they do have reasoning levels that are available to be set when available. I would select the medium reasoning level in cursive for example you just select whatever it is and then GitHub copilot this one blew my mind too because GitHub copilot I didn't think it was actually going to work with it at all but it did I don't think that GitHub copilot has done any work with it I added it through open router and it is a thousand points higher in second place I was very surprised by that by the quality difference there and here we go here's the overall score you can see zed at the bottom did not do very good wind surf Cloud code grof fast. So cloud code using the cloud code router did not do that great. Typically if you get over 20,000 or so from open code up I would say those are all pretty solid implementations of it.
Now I only spot checked a few with claw opus 4.1. So we don't have a lot of data here primarily because of the cost of this thing. I've already spending a fortune on running these tests and opus if you run like I do some of my tests where I like do a webOS like test just to see what people do. That cost me $50 to do with Opus. It's ridiculous. Now, running that eval just becomes crazy. So, Ader ended up with number three. I kind of expected this one to be higher to be honest, but you know, it is what it is at this point. Root Code works with Opus. Well, now I do kind of understand why a lot of the Root Code guys actually say they just like running Opus and let it go. I still am a big believer that Opus is better at planning, debugging, stuff like that. not so much as a daily coder, but Warp, oh my god, Warp did have an option for this and it it did incredible. And so when I saw these, I'm like, "Oh yeah, that's about what I expected." I did not expect the 27,000 point score in Warp to be totally honest with you. So that is might even be the highest score I've gotten so far. In fact, Warp with Opus 4.1 ended up one of my one of my tests is like very hard and no one's ever got above uh like 87% on it and Warp was the first one to do that with Opus 4.1. It's incredible.
Now, here's just the overall view of the four. So, we can see Cloud Code with Opus 4.1 is last. How is this possible that Claude Code is scoring worse in all of these agents now? And it's consistent across SA and Opus. It's very odd to me. There has to be some optimization that's going there. It wasn't that long ago that Cloud Code was just number one in all of these charts. So, it does have me a little bit worried that, you know, maybe the days of like having that really value Cloud Code subscription are starting to diminish. I don't think it has yet because still 24,000's very good. Uh, I still feel like I get a good value out of it. But I will say there's something to be said about these scores kind of dropping.
Now I do want to just tuck touch really quick on the newcomers. So we have the augment CLI which actually topped the newcomer chart. The Kira Sonet 4 surprisingly did 25,54. That one actually I didn't you know honestly when I saw that come out I'm like ah it's just whatever. But it's solid. It's a solid implementation. To be frank though you're just cloning um VS Code. you're adding on agent capabilities that you probably already have like from open source software. It's probably not that hard to get a Sonic version of this up to 25,000. Now, I don't want to belittle like what they've done, but I also want to say like I don't think it's an extraordinary accomplishment to just clone VS Code and do something like that as it was, you know, a year ago.
Now, augment CLI with GPT5 ended up at 22,880. So pretty clear that GBT5 doesn't work quite as good in augment as uh sonet 4 and coder ended up with a 20,274. We don't know what models it using. Again, we think it's quint3 coder, but it could be re routing between different models. It is not that great of a score, but I think you could use it if it was cheap enough and you just wanted something that was going to be, you know, that would do code for you, but wouldn't do like exceptional quality for you.
Now, I know this is a lot to look at and I apologize, but I wanted to give you the full breadth of everything that we were testing here. Uh, the orange is OBS 4.1, the blue is set 4, purple is Quinn3 coder, red is GPT5, and the dark green is Grocode Fast. And you can see like a light kind of gray when those are the Gemini 2.5 pros. It's pretty easily to see here that we've kind of got three contenders. We've got Quint 3 coder, Sonnet 4, we've got Opus and GPT5. I guess kind of four if you count Opus in there. But we've got kind of those contenders as being the top models to code with now. There's very little difference between them. It's really starting to come down to cost, uh, speed, and then model knowledge.
Grocode Fast actually made a really good kind of topline score with Copilot. Very surprisingly to me, but typically it's on the bottom half here. Like it is definitely a more smaller model. I am excited about whatever their next version of the model is. I think that could be extraordinarily good.
So talk a little bit about some of the points here. Uh, Grock code fast, I guess this heading here is incorrect. You kind of exclude that, but Grock code fast is lots of promise, but there's some problems with tool calling that I ran into. And this one also significantly had problems with some kind of environments. So it would do things that uh would try to make up commands and have to work its way through that which sometimes the other models do but not as much as what I saw with brow code fast.
I talked about this already but warp seemed like it came out of nowhere. It'd been off my radar honestly because of how like far down the list it had been in my in my scoring before and it's doing great now. Now, the version of Warp that I'm actually using to kind of show you is this one, which is V0.2025.08.27.08.11. So, you can kind of see when I updated that. And I've actually come to really like the way it worked. I love the way you can just kind of click the the auto and let it run. I like it because running evals is so painful when they make you approve everyone. I just click that button and I can forget about that, come back and check on it in a while.
I do expect GPT5 to climb up the charts in the next few months because I have found in my particular coding with it and in implementing it in my production application as part of my chat agent that you can get good good performance out of GPD5, but your prompts do have to be kind of different. You really do have to instruct it differently.
what I'll be using in September. I do have the $200 a month Codeex Open AAI plan, the pro plan or whatever. And I do think I'm going to take advantage of that. I I was debating actually downgrading that to the $20 a month plan. I don't know if I will now. I'm kind of liking the codec to ecosystem, which is again I did a video on that already, which we'll see how it goes in the month of September. I'm worried about getting rugpulled uh with what they put out there because right now the limits are incredible, but I don't imagine that's going to stay that way for very long.
I will then be using R code. It's just one of my go-tos. I love using this. Um an honorable mention here is Klein. They've got this new condensed prompt for local models. So, I'll be testing that as well. But root code is just so easy to configure temperature. I can pick providers a lot easier. It gives me all the configuration I need. I kind of wish others would take note there. And one of the issues that a lot of the ones that are out there that do have configuration, they make it so complex. Root code makes that so much easier in my personal opinion.
And then cloud code, of course. This one I'm getting a little bit more nervous about though. So I may find myself, you know, moving more into codecs over cloud code. I it feels weird saying that because before before it wasn't that long ago claw code was number one and I do find myself even in the last week leaning more and more onto codeex especially when I'm doing any sort of like complex bug fixing. I feel like codeex just does a really good job navigating and helping me come up with ideations around that.
I don't think I'll be using the gra code fast model that much. I'll probably do it periodically just to see how models are how that model's evolving because they do say they're going to do constant updates on it. But main models are going to be GPT5, uh, Claude Sonnet, and Quinn3 coder. And I'm also playing around with trying to get some big local models running just to experiment more on the local model standpoint.
Honorable mentions, as always, I am such a big fan of Crush. Their agent starting to come along well. They fixed a lot of the the weirdness that I had when I first actually used Crush. It is It is honestly just a beautiful It's a beautiful uh tool. Augment's great. Their context engine's great. I love what they're doing. I do think um them adding the model picker is actually great. I think they just need to work a little bit on their GPD5 harnesses. Really excited they have a CLI. I do think the CLI is slightly worse than the ID extension which makes sense because the CLI has been developed only for you know a couple months now.
And then open code. Open code is phenomenal. It is so easy to run any provider. It it's it's basically like my CLI version of root code. So it when I when I'm testing models locally or when I'm testing models that I want to see if certain providers open code is phenomenal at that. I can't say enough good things about what they're doing there. Uh, I would actually probably put that in here in my number two. So, I would probably have root code and open code kind of tied in that position because I I like having one of each.
All right, that's going to wrap up this video. I have gone on long enough. This took an incredible amount of time. I would love to know your thoughts in the comments below. Anything surprise you? And if you would want to jump into my Discord and chat about it, let me know. I'd love to just kind of get feedback on the way I'm testing. if you've got ideas or things. I am not ever going to claim that what I'm doing is the best all end all be all. That's just not what I'm doing. My version three, I would like to get some feedback on how I'm actually building my next uh suite of evals and how I can open source parts but keep some private, but really focusing on larger, more complex tests and how we can automate that.
Anyway, uh I will update the website probably in the next couple days, but I've been running like non-stop for the last two days trying to get all this ready for September 1st, which we're like midday September 1st, so I think I'll still make it. All right, until next time, everyone. Have a wonderful, wonderful day. Peace out.