Transcription
This is just some of the AI coding assistants that I tested. I'm running out of room for all of them, but on top of that, I've also tested five models across as many of these as I can. Sonnet 4 being the most widely available one. So, let's jump into this, and then along the way, we're going to have a conversation about some of the weird things that happened. And I know there's going to be a lot of questions and controversy about some of the scores as we get further on.
Of course, this is the thing that I really want to make sure people understand when I am scoring this. What I am really testing is really long-running spec following, instruction following. Are they good at calling tools? Are they able to go and complete something if you give it very specifically what it needs to do? And the way I do that is through a series of static code analysis, unit tests. I also use LLM as a judge. LLM as a judge is kind of a variable, right? Because I have tuned those to be as close and as similar as possible, but it is a non-deterministic nature. It does have a smaller weight in the overall final score. And I do run that multiple times to kind of average the result, but what you will see is a lower score from an LLM as a judge is very consistent, but it might change by, you know, 3 to 5%. I try to keep that as low as possible in all my prompts. But it is important to understand, you may actually see something on here that has a high score, but you don't enjoy working with it for whatever reason. It could be slow. It might not have vision capabilities. Vision capability is actually kind of important to me.
A couple notes. Uh, prompting matters a ton. What I found is Gemini 2.5 Pro is kind of lazy, as is 03. And a lot of it is the harness because there are some AI coding assistants that will do well. Like, you give it the same prompt as I give Sonnet, and it'll go all the way through and complete the task completely. But I've actually tuned it some to actually make Gemini 2.5 Pro be a little less lazy and 03 to be a little less lazy. Um, and GPT4.1 for that fact. The harness also matters a lot. You will see that because you will see scores of zero where they just literally it fails, whether it's an infinite loop like I get in some places, or it's maybe that it just errors out, like it'll actually say, "Hey, I just don't know what to do," or just cuts off. Number of things I have tons of images saved away. I won't cover those because this video is going to already be long enough.
But what I found more than anything is the model matters most. Sonnet 4, everybody has tuned to that, and the scores are starting to converge because everyone is catching up with one another. So, it's making it harder to find who's actually winning or better in coding, especially for Sonnet 4. But the other models I think are a bit more interesting that I'll be running. Again, the way that I do this, this is a test that I retired. I just want to make sure it's very clear. I create an entire spec and I tell it to create the thing for me. I do this across different languages. I also do this with existing code bases where it needs to implement something. I do this by myself building out a solution and then making sure that the unit tests pass 100%. Then I build a prompt and I keep tuning that prompt until I get it to a point where it's stable enough where I can get consistent scores over and over again. And I really make sure like what's in the prompt is what's being tested with static code analysis, unit testing. I even do linting checks, things like that. Uh, the I have a quick grader process that actually loads the application to make sure it actually loads without any errors, and that counts as part of the scoring as well. And then we use LLM as a judge on top of it.
So, jumping into it, the models that we're using called for Sonnet. I've been asked a ton about 03. Honestly, I was very skeptical about running 03, but I'm glad I did because I there's some very interesting results I'm excited to talk about. Same with 2.5 Pro, Quinn 3 coder, and Kimmy K2. Now, to be very transparent, I do not think I'm going to be running Kimmy K2 again. The providers are just too unstable, and it becomes very hard, and a lot of these AI coding systems don't even have that available. And when I do try to bring it in, it just doesn't work at all.
So, let's jump into Claw 4. In third place, it is Root Code with a score of 26,014. You're going to see how converged these scores are. 26,214 for Klein. Like, basically margin of error difference. Like, this is basically a tie. If you really think about that, if I were to run them another 10 times, the scores could be reversed. It's like very, very minor percentage difference there. But then you have GitHub Copilot, who wins, which is nuts to me. Not by much, but by enough, by a measurable amount. Would you actually notice it on a daily basis? I don't think so. I doubt it. But it's still surprising to me. GitHub Copilot, and I said this last month too, they have really made the thing solid now. I'm impressed with the turnaround there because for a while, the thing was garbage.
So, overall, I've actually uh tried to color this so it doesn't flashbang you. So hopefully this is easy enough to read, but you can see how all the scores have kind of converged around this 25 to 26,000 point. You have Ader at the bottom and Zed at the bottom, which kind of surprised me because Zed just burns tokens like crazy, but it still scores lower. It's It's kind of bizarre to me. Augment Code is also on the bottom side of this. And a new version, a new one that I actually ran with Factory AI, which is an interesting one because it like syncs to your local desktop, and you actually put the job in the web, and you have it kind of work on your local desktop from you from the web. I'm very new to it. I may have not set that up correctly, but it scored, you know, what I would expect with a decent AI coding assistant. And we had Trey, who did very well, and AMP Code. You know, I've heard a lot of people talk about AMP Code being exceptional. And to be fair, you know, you probably wouldn't see much of a difference between any of these. Like, I think I did the math on this. It's like a 7% difference from the top of these. It's like less than two or 3% difference between some of these here. Like, they really have converged on each other. For example, the Klein family, Root Code, Kilo, and Klein, you know, they're all basically identical. What's surprising to me though is Claude Code is not number one. It has been surpassed, which blows my mind a little bit, but I do think some stuff has changed there. And I don't know, I don't know exactly what, but it it has actually dropped at least in ranking. So, you know, still up there pretty high. It's right in the Klein family of things. Still very good.
Now, 03. 03 is an interesting model. Cursor, of all things, lands number three at 21,000. Now, remember the baseline here. 26,574 is like the top of the Sonnet 4 version. Then we've got 03 at 21,26 for number three. Ader. Now, when I saw this, I started realizing cuz Ader is not so much of like an agentic harness like the other ones. It kind of I don't I haven't dug into the code of it, but it seems to like prep all the code that needs to be done, and then writes it in individual files. So, I feel like this is probably like purpose-built for 03. And it the cost is so low. I really would like to figure out a way to get the cost in part of this, but it's so time-consuming to get these things ran and to get that cost accurate when I'm running it over and over and over and over again. Um, so I am working on that, but Ader's cost is significantly lower than most of the other ones that you're paying for API cost on.
Now, who do you think's number one? This probably will surprise you. Windsurf. Whatever they've done, they have tuned it incredibly well with 03. And I was surprised because Windsurf did awful with Gemini 2.5 Pro. So that just kind of shows me that the the harnessing and the way that they're actually controlling these models matters a lot. But I did not expect Ader to be number two. I did not expect Windfur and Cursor to be one and three. But it does make me realize that when I talk to people and they're like, "Oh, I love 03 of the coding model." I wonder if they're using it in Windsor and Cursor because if you look at the overall score, warp.dev did a decent job. Actually, it actually completed the eval. Root code failed, client failed, kilo failed, open code failed, zed failed. Zed was the oddest. It just would do some thought process and just quit. Uh, open code would do kind of the same things. It would end up just going for a little bit and then dying. Kilo, Klein, and Root Code. I can get it running a little bit, and then it would fail. Very similar failure across both of them. A little bit unfortunate. So this model is probably very good at coding, but not really good at driving agents. And I don't know if I will test this again in the future unless there's a lot of interest for it, or if another AI coding tool comes out that makes it seem like it can control 03 even better.
Quinn3 coder though, this one I was very excited about, and I've been using it in Root Code a lot, and it scored 25,898 in Root Code. That is up there with Claw 4 Sonnet with the new test and everything that I have in there. It is insanely good. Now, I am using Alibaba's API directly. That is the best one. If you use a different provider or use a different endpoint like an FP4, an FP8 quantized version, you will probably get a worse experience. But this thing works so good. And this is one of the reasons why I think model selectors are so important. I think we need to be able to have things like this where we can load them into Root Code or load them into Tray. Freaking Tray, 26,214. I mean, basically it's similar here, but Trey's an amazing agent and it scored very well here. And number one, this also still blows my mind. And it may blow your mind too, cuz I when I actually cuz I put all the scores in a spreadsheet, I don't really pay attention to the final score until I get to the end and I sort it. And when I saw this, I was like, I guess it makes sense, but it's Quinn Code and not by a small margin. I it's kind of a small margin, but by several hundred points here. Uh, Quin Code works really good. Now, there's a little bit of a caveat here because with Quin Code, I did authenticate directly to Alibaba, and I'm pretty sure that it uses the 1 million context limit Quoder, whereas in the R Code one and the Trey one, I was connected through Open Router, which I think is using the the the one that's like what is it, 262 or whatever, uh, context. Quinn Code did a great job uh at the eval with Claude Forset and all the other ones.
Now, this is all of the scores here. We've got Quinn Code at the top, and if you look down here at the bottom, Open Code did complete some of the evals well enough to actually get a score. Co-pilot I could not get working. Ader did decent enough. It got 20,000 points. Kilo, uh, Klein, and Root Code all very, very similar. Kilo fell behind more than I thought because it had actually been, you know, right in the mix with everyone, but it's down enough to where it's like beyond margin of error difference. So, I thought that was kind of interesting. It could be just the update of Kilo, you know, trying to keep up with Root Code and Klein being slightly, maybe they've done something with that model cuz like Root Code and Klein is basically identical. Honestly, either one of those could have been number three. And Windsurf did a great job with this model as well. Absolutely crushed the Evals. All of these are 24,000 more. It It feels like a clawed for Sonnet model. It's insane.
Now, Kimmy K2 is a tough one because the providers for this thing are all over the board. I attempted to use Grock when it was available, and then I used the Moonshot AI one uh if Grock wasn't available. And what I mean by that is Grock kept going down or would rate limit me for whatever reason. But I did stabilize it as much as I could to actually be able to test this. This model's tough. We really do need like a very stable provider that I can use, and we need all of the AI coding assistance if we actually want to use this model to actually implement it. So it's easy to use. But Trey did a great job. 24,000, that is up in the 24,000 mark. So it's incredibly good. With number two, Klein did awesome at 25,610. And then Root Code, uh, basically identical. I mean, these scores are pretty similar here. Couple hundred points difference. And again, if you were to run this because of some of the variance with the providers, these scores could flop back and forth, you know, in in any number of way. Kilo scored significantly less here. And this also kind of confuses me a bit. Um, because it seems like the Kilo did worse in Quint 3 coder and in Kimmy K2 than the reg the other two Klein based ones. And again, this might just be an update type thing. Maybe it's like fallen behind a little bit on some of the prompt stuff. I'm not totally sure. It could be uh the to-do list stuff that's kind of rolled out into Root Code helping it. Who knows? IDER kind of worked, but very poorly. I would not use it in haidider in any way. Um, and then that one I did actually use the Gro one. Copilot, I couldn't get it to work at all. Open code, I couldn't get it to work at all. Wind surf, I couldn't get it to work at all. And so in this particular case, I think your best bet is going to be using like a Root Code, Klein, Kilo, Trey type thing because all four of those I think are pretty solid with Kimmy K2 as long as you hit a good provider, which to me is Grock or the Moonshot um AI one directly, which the Moonshot AI one is very, very slow unfortunately.
Now, Gemini 2.5 Pro. This one is tough because there was a while when this was my favorite coding model, and it wasn't even close. But this model also needs a lot of specialized prompting. The scores are just going to be lower than Claw 4 Sonnet unfortunately. So, we have Root Code at 19,516 in third place. Very happy with that. GitHub Copilot does an incredible job with it. you know, significantly better than Rue Code at 20,380, and then Ader number one. Now, when I first started testing like Ader, I didn't think it was going to actually make the charts, and it it makes me kind of realize that these big reasoning models that aren't always the greatest at tool calls work better in in Ader, and significantly better. Like, it's not even just a couple hundred points. It's like 1,300 points better. So when I hear people say things like 03 is the best coding model or 2.5 Pro is the best coding model, you know, ADER is probably a good representation of that because it's not driving an agent as much. It does a little bit, but it's just a different workflow that I think works well with these.
Now, let's take a look at all of these here. You can see here uh Kilo and Root Code and Gemini CLI and Klein are all basically identical. They're very, very much very close, and honestly, you could have put Kilo at number three. It's like what, 20 points different from real code. So basically identical. But Gemini CLI, now that has come a long way. I would say before it was kind of unusable. Now, it is in the same ballpark as using Gemini 2.5 Pro in Root Code, and you probably wouldn't tell a difference. Klein fell down a little bit, but honestly, it's, you know, that could be runtime variance there. We've got Cursor and Trey both doing reasonably well. Uh, and then we have Open Code actually did complete everything with Gemini 2.5 Pro, but I will say this is with their latest update. There was a while um I would say a couple weeks ago it didn't work at all. So they've made massive strides on getting open code to actually be able to make a ranking here, and warp.dev actually was able to complete. Wind surf surprises me. But I would just keep getting infinite loops where it would just keep saying the same like phrases over and over again. I could get like an eval to finish, and then I'd run my next one, and it would fail. I'd run that, try it again, and it would fail. So, I gave that one a zero because it had too many failures. Zed and Void had very similar problems. The harnesses that whatever they're doing with Gemini 2.5 Pro, it just doesn't work good. Whether it's like temperature config that they've got in there, or if or if it's just the prompting that they've got, I'm pretty sure like Open Code is actually building custom prompting for Gemini 2.5 Pro and custom config for it. So, I think that's why they're actually starting to do well there.
Now, the next one you get a little bit of a flashbang because I wanted to color it very clearly, but I've tried to dim it some, but it is going to be bright. So, you can kind of see here the colors. Orange is Sonnet 4. Purple is Quinn 3 coder. Green is Kimmy K2. I actually have an entry for red, which is the GLM45 model. Black is 03, and the blue is Gemini 2.5 Pro. You can still see Sonnet's the king. They've got the most in the top up there, but Quinn3 coder has put a dent in that. Like, it's actually trickled in throughout all of that. And Quinn Code is honestly the scoring makes me actually want to go and actually try this out like for a couple days or something. See if it's just that enjoyable to work with in Quinn Code as it's been running through the evals. But again, this is only one dimension of testing. Sonnet 4 has vision capabilities. It has really good prompt caching, and I can keep the cost lower if I'm using the API. Quinn3 coder when I use it in my codebase, I don't feel like the prompt caching is actually working in Open Router. So I have to keep the context limit small, otherwise the price gets racked up really quick, and I do not like that. I really hope that that gets fixed, and I'm using the Alibaba provider there. So I would assume that Open should be able to hook up caching there. It would make a big difference. Now maybe in Quint3 code that would actually use caching because Alibaba endpoint does support prompt caching, but I know in Open Router and especially in Root Code, if it gets to a certain point, I can rack up too much price. It's just important to keep that in mind. Kimmy K2 for example has a limited context window. It's smaller than what Claude offers in Quint 3 coder. Sometimes context window matters. That's just important to know. And I will say that Quint Coder actually has a slightly bigger context window than Sonnet 4. There's just a lot of dimensions to all this stuff. I will also say I have found that Quint3 coder can will actually be a little bit tool call happy at times, and you kind of have to prompt it just a little bit differently uh if you're using it in your normal codebase. And I found that Sonnet 4 is starting to take more shortcuts. I feel like it I have to prompt it specifically to not take shortcuts. Just each of these models have their own little quirks to them, and you may like or not like that based on the behavior that they have. But at the end of the day, I'll say I'm very happy I actually ran through this test.
Cloud Code not number one this month. I think it actually was number eight. It's still very good, at least from an Eval perspective. I'm still using it a ton personally. I've just found that I have to talk to it differently. There's definitely something that has changed with its behavior, and I think it's probably some model update that has happened because I see it even in Root Code and others. Zed you have to be careful with cuz I can let that run, and it will run until it it basically runs out of the number of tool calls that it allows you to do. you hit continue, it'll do it again. Zed will just go on forever, and it's just something to be aware of if you're using using Zed. I do like Zed a lot, but I found that the eval literally would take two to three times longer in Zed to run, just simply because it just kept doing so much. There were lots of surprises this month in my mind. Quint3 code was a big surprise. 03, Gemini 2.5, and Ader was a big surprise. And again, this is measuring like one dimension, so I'll be very curious to see what everyone's feedback is on this.
So, let's talk about what I'll be using in August. For number three, Open Code. I really like the what the team at Open Code's doing. It's new. There are times that it's kind of wonky, but they're making fast improvements. And I'm re I really like how easy it is for me to load up new models, drop in that open code config, and be able to put in some provider that I want. I really like just the simplicity of that.
Now, Rue Code has done some massive, massive updates, and it's come back on my board now. Root Code has just been something that I've just been a fan of because it's it's really like an advanced mode of client, and I definitely am a tinkerer. They've got custom slash commands now, which are just freaking awesome. They've got background editing, which is also freaking awesome. It's so easy to configure different modes. They've got the marketplace for modes now. So many cool things. The to-do list has actually been great, and I think that really does help guide the models a lot. I I really like root code, and it I am just gonna continue using that probably forever because I've just got such a cool config built in there, and it works great with quint3 coder and cloud code. I'm paying the $100 a month currently. It does not make sense for me to go and pay a ton of other API costs to use Sonnet somewhere else. Cloud code is great. They're making constant updates as well. They have the new sub agent thing which I've been playing around with. I actually really like that as well. and they're just constantly rolling out new features. You can actually now set different models on each sub agent. So, very, very cool.
But I do want to call some honorable mentions out. If you watch my recent video on Charm Crush, that thing is freaking awesome. I need to get if they get a few of the little like user things kind of ironed out, that thing is going to be great. I'm excited to follow that. I'll see how it kind of evolves in the month of August. Augment is so good. This context of the codebases seem to decay some because I used to be able to actually ask it questions, and I'm getting what I would say is not so good answers, and I don't know if that is Clawed 4 and some of the changes with it, or if it's something with Augment, but it's still really good. It's a great agent. I'm actually a big fan of it, and I'm so excited to try their CLI, but I will actually I have moved it to just an honorable mention because I I'm a big fan of Augment and what the Augment team is doing. But from a coding perspective, I've actually found that other agents are just slightly better.
And I also want to mention AMP. There's so much hype around AMP on X right now. And it didn't it scored well in the eval. It wasn't top of the charts in any way, but it scored with all the other clawed force on it ones, but they are taking a different approach, which is almost like the Apple of AI coding assistant. Super simple, very little options, no model selector, but they also do have the CLI, which I really appreciate. And I don't know, I don't know. I actually have debated like giving an entire week to AMP just to see how it actually works. I have probably burnt some amount of money, probably 20 or $30 actually coding in my codebase with AMP at this point, and I thought it was okay. Like, it I used it as a second pair of eyes on something I was having trouble with, and it didn't find it, but it, you know, it was good enough. It navigated things well, had good ideas, but it wasn't like mind-blowing to me. But I really do want to understand why there's so much hype around AMP because there's a lot of people talk about the magic there. So, I'm I'm debating giving that a try.
Now, that is going to wrap it up. This video has gone on a long time. Let me know in the comments below which of these things are you just surprised about. And if you wouldn't mind considering liking and subscribing, that would mean the world to me. Till next time, everyone. Peace out.