Transcription
Everyone is calling these new AI coding agents cursor killers, but they are mostly just based on hype and vibes. So I decided to spend time to build my own standardized set of benchmarks to test Cursor against Claude, code on the same repo, same code, same rules, same prompt, same everything on an actual live production app to tell you the difference between those two coding agents. What I found might just change how you pick your next AI coding tool. Let's dive in.
So, the test will cover five different areas, with some covering more than one. So, there will be code exploration, test creation, feature implementation, bug finding, and tool use. To make sure these tests are realistic, I'll be basing this on a live production app I've recently built. It's called Boss Factor. It allows you to do easy analysis of open-source GitHub repos to find out the livelihood of their contributor activity.
It's currently built using Next.js, React, using Superbase for O and database, using Playwright for the end-to-end test, and using Cloudflare for deployments. And it's just over 4,000 lines of code. So, it's fairly representative of a common web app, and probably even verging on the smaller side. So if the coding agents can't even deal with this well, then it will have way more problems dealing with larger code bases.
And for Cursor, I'll be using the agent mode with Gemini 2.5 Pro, not the max mode. And so just on the standard $20 subscription tier, and for Claude, I've set it to use the four set. And again, also on the same $20 tier subscription here to do all of these tests.
And just to give you an idea what the app looks like, you can see that it's a simple app that shows a list of the recently analyzed repositories. And if we go into a specific one, you can see that it's analyzed to have a bus factor of 11. And you can look at the active contributors over the last three months, total number of commits, and how many issues have been opened and closed. The contributor activity distribution. And so you generally want to see something that's widely distributed across multiple contributors and also see the history of open and closed issues to check that things are constantly being closed, and the fact that open issues also means that people are actively raising issues for an actively well-used project. The core premise of the app is really simple. So we'll be using the test to constantly add new features to this.
So the first test we'll be diving into: I've asked both agents to review the codebase and suggest critical end-to-end tests that it can add to the repo, and I've asked it to specifically watch out for the GitHub API usage and suggest how I can better test those areas. So if we just look at the Claude code run over here, and it looked through it and triggered its thinking process because in Claude code, if you have the keyword "think" inside of a query, it will always trigger the thinking mode. It went through pretty much exploring most of the files and using quite a bit of tokens during this process and spending quite some time going through the different files, and then you can see that in the thinking trace it's found the different types of tests: the authentication flow, the overall architecture of the app, and then very easily suggested the OAuth flows that I haven't tested, the app flows, the data analysis flows that I currently don't have in my test because in my current test I only contained one single one for testing the happy path of the analysis of a new given repo, specifically for the GitHub usage. It's additionally suggested the different types of repos I should test and also how I should test different private repo access, even though I haven't planned to do this, and additionally for caching the data because this is dependent on a lot of GitHub data analysis, so we don't want to make calls to the GitHub API constantly, suggesting ones for those error handling, data accuracy. So all in all, a great set of suggestions.
And similarly for the GitHub API specific testing, again it's suggested quite a few things and that I should be adding. If we compare this to the Cursor run here, again with the exact same prompt on the exact same commit in the same repo, it went through the code but interestingly only looked at the test file itself. It didn't even try to look at the feature implementation, the actual components that implement the front end. Obviously, without having context of what the app really does, it ends up just suggesting tests and kind of guessing the additional work that's actually required. So I'm not sure why it's not doing the extra exploration. Maybe this is due to cost optimization on the Cursor side due to the heavy token usage for some of the larger requests. But what is suggested for the GitHub API testing was still very much correct. It specifically mentioned that it should also create different GitHub sites, GitHub repositories to test this analysis result. So that's great to see. But in general, many of the information it actually got directly from the rules file rather than using the tools to explore the codebase itself. In this task, this is a very much clear win for Claude.
And if we go back to Claude, what I really like over there is also the formatting. And somehow I have found that Claude generally generates way better formatted output. Given that it purely does this in the terminal, it's quite amazing. In this task, it's a clear win for Claude.
The next test: we ask both agents to do a simple end-to-end test creation. And so this is on an existing feature that's already been implemented. I've asked it to test the refresh button for the analysis data for each one of the repos and even given it the example codebase to use for this analysis test. And going through the Claude run, you can see that it went through the existing spec for the end-to-end test, checked the layout, read through the configs, and then started generating the tests, uh, very much correctly and also tried to actually run the test at the end. But while the test run was correct and complete and it did go through a few iterations before the test actually passed at the end, but what's interesting though was the test was only testing the animation state of the refresh button without really trying to test the data has been updated on the page itself. So while you can say the button kind of worked and it did react, but it didn't really check for the real data being updated. So really this is a bit of a half-assed test. I had to specifically tell it that it's not properly testing the data has been updated, and so that should write it again, and then it was very happy to go into the test update and run it again. So this iteration feature worked very well across the board. But the problem is you still needed human intervention to get it to write good tests before it started succeeding.
Now if we compare this to the run inside of Cursor using the exact same prompt again with the same reset clean repo state. It did the very same thing of reading through the existing test files and then going through the spec, updating the file, and then you can see that it ran the test, tried to look at the results. Well, actually, like passed on the first run, but strangely decided to somehow delete the test after it successfully ran it, and I had to prompt it specifically to recreate the test afterwards. And um, so just again was a very strange behavior. What's really odd about the test that Cursor created: it fell into the same trap that Claude did, which is just testing the animation state of the refresh button, which is not a realistic test. And again I had to prompt it specifically that it needs to check for the updated state. Of course, it was very happy to go and update the test itself and iterated on it until it succeeded.
So, you can also see this is a point I generally stress across all of my videos, which is tests are so important for the current agentic workflow. By creating tests, you give agents a great way to iterate within the agentic loop itself without having to have human intervention. Otherwise, the flow that you would have had would be first to get the agent to create something, and they ask you to verify in the UI. You try something, things don't work, feed it back, and it goes back into the iteration mode. But once you have great tests like this, you give agents the capability to self-iterate for a long time before it can return to you with the exact fully satisfied implementation for your code. So tests are super worth the time. But comparing the results across the two, given that they fell into the same trap, and while both tests were working at the end, both required one extra prompt to get things working. In this case, I'd say both are at a tie here.
Now, we get into the meatier part of the test, doing a full-on feature implementation. The reason this is more interesting because it requires changes in the database schema and leveraging command-line tools like the Superbase CLI itself and also changing the UI to satisfy the new requirement as well as writing new tests to make sure it can satisfy those requirements correctly. I've also given it very specific requirements on how it should be surfacing the data inside the UI. So really I would expect both agents to be able to complete this in one run. And interestingly, both did, even though they failed the simpler end-to-end test in the previous run.
And if we look at the Cursor run here, it was able to look at the database and then decide it does need to make a schema change and then executed the command using Superbase correctly to generate the migration, created the migration, and then it proceeded on to applying the change. Actually hit a few hiccups due to its SQL formatting and it went through debugging the issue. While the thing it generated at the end was functional, it definitely wasn't the cleanest table I've seen. But we'll give it a pass mark for now because at least the end work was functional. And after you have successfully applied the database changes, generated the types using the command, and then proceeded to implementing the rest of the feature. So I won't go through the details of the whole process. You can see it's an extremely long process, and this thing ran in the background for just over like 13 minutes. But at the end it implemented the feature, implemented the test, made sure everything ran correctly and then returned to me with a fully functioning feature in just one go. I think this is one of the strongest points about having tests or even just simply asking AI to write tests because you make sure that it always checks itself whether the thing is actually functioning correctly without the human having to interject. So here definitely a huge pass for Gemini inside of Cursor.
So how did Claude do? Let's move over inside of Claude. Again, same prompt here. The one thing I really love about Claude's output is it always clearly outlines the plan and its current progress inside this super simple to-do list inside the repo. And if you scroll through the change, you can see it step by step progressing through the different steps that's required. I won't bore you with the details. So the SQL here was generated better and it was able to apply it successfully in one run and it moved on to implementing the rest of the feature correctly. Albeit I think some of the file naming and code wasn't the cleanest I've seen, but again it was functional at the end, and also I would have definitely preferred if it, both agents, created a test first, but it was fine, and so that it created tests afterwards and still ran it successfully. Going through this entire run, it went through quite a few different checks and fixed some linting issues along the way and then iterated on the test as it passed and failed a couple of times in the middle while it also experienced issues with past broken tests, but at the end it got to a working function and so given that both passed it should be equal, but I would say Cursor with Gemini actually has a slight edge here given the code quality is definitely slightly higher by Gemini.
I think this is also the issue I've generally seen where Claude is generally great by generating very comprehensive and functional results, but the code is not necessarily the cleanest and all the most elegant, but Gemini's generated code tends to be of better quality and more readable. I do wonder whether this is because Google also trained more of Gemini on some of their internal code quality and styling before releasing, and I know how vigorous Google's internal code review process is to get code committed. So that might be the benefit that Google has for code quality side. So here I would give the slight edge to Cursor with Gemini in this test.
So we move on to the next test: bug finding. In this case, I have given both agents a very broad task to find critical bugs in this codebase and list out the must-fix ones. So this is testing the agents' capability to explore the whole codebase and try to fit in quite a lot of things into their context and then prioritize tasks that need to be done. And if we look at the Claude code's output, it actually worked really, really well. And again, it creates the to-dos for the exploration. And this thing actually ran on for quite some time. As you can see, the time really adds up for its codebase exploration stage. And what's interesting though is at the end it outputted a beautifully formatted list with priorities, the issues, and what the fixes required, and the risk level for each. So even though I've only asked for the fixed bugs, they also listed out some of the medium and lower priority ones that should also be fixed. Some of the issues that it has listed here, it's actually quite surprising to me that it even caught the issues. While I did not notice them while coding this up, albeit I did vibe code the majority of this codebase, but I was still reviewing most of the code along the way. Just really shows how important bug finding is in your codebase and how much AI coding agents can help you along the way. And so at the end you can see that it found a total of 10 critical issues. It does say all of the issues are critical. I would agree the higher the medium priority bugs are the ones to fix, but yeah, the lower ones definitely are not max fix right now, but it was great to see it creating such a comprehensive list. Again, goes along with the Claude vibes to always be a bit more comprehensive, albeit could be some bit verbose from time to time.
And if we compare this run to the Cursor's one, Cursor interestingly only went through a very few files during this process and it only looked at some of the core files across the codebase. It didn't even try to go through all of the different components for part of it. It failed to look at the authentication part of the repository and at the end only found one, interestingly only one bug with the role-level security that I was applying, and I would say this is also somewhat of a lower priority one, but perhaps this is maybe limited due to the limited number of files it was able to go through, and perhaps this is an artifact of Cursor trying to optimize that cost and Claude being quite generous with their quota limits. But again, for the same price, it looks like you're getting a bit more bang for the buck inside the Claude code side. I tried to make sure this wasn't just an artifact of a single run. I tried to run this command again without any changes. It generated two other different issues, but still very limiting compared to Claude. So, it's a clear win for Claude code in this test.
Next, I want to test both agents' capability to follow guides and the ability to browse the internet. And I've given it a simple tutorial it should follow to migrate my application to be deployed onto Cloudflare Workers using Open Next because I've done the manual migration following this guide myself. So I know the guide fully works with a few caveats that I also want to test if the agents were able to catch, and then so just given a simple prompt with this deployment guide and then started it off to do the full migration. So Cursor did browse the page correctly and started off the work first installing the relevant dependencies and then setting up the package.json. But already here it's already going wrong because the guide gives a very specific set of commands it should be adding to the package.json, and these ones are definitely incorrect. Then straight after it got distracted with a middleware issue. It tried to fix in the file and then went through quite some thinking time to update it while it was completely irrelevant and does not change this migration. And then later during the process it's tried to set up the Open Next projects through Cloudflare because the command line requires inputs from the users. It tried to run it a few times and then just gave up at the end. And so we can see that even though these things could be very easy to follow for humans, but when tools require a bit more interactivity from the agents, they currently still fail. This is definitely an update that we definitely should see in future agent updates so that they can provide inputs to command-line tools so that they can better use it. I mean they can already use browsers. Great. So this should be an easy addition here. It's a fail for Cursor.
But how did Claude code do here? Given again the same exact prompt. It created the great to-dos, and it was able to read through the page and I saw the fetch actually happening in the background. They didn't actually print it out here. But while it went through updating the code, it created a config just by imagining something itself, while the migration guide already gives it a very clear example configuration file, it could have directly used. Very strange to see it not following the guide at all. And then so going through the rest of the process, much of the configuration is actually invented correctly, though it was trying to follow the right task to get it migrated to deploy on Cloudflare Workers using Open Next, but it's purely based on its own knowledge and it didn't even say that it failed to read the page because I know it fetched the page correctly. Very strange to see how it failed to follow most of the rules, and again the things it added to the package.json, the script scripts added to package.json was completely incorrect, and going through the rest of the change. Again, Claude code also fell over in the place where it required human input. But it did go one step further by trying to echo Y into the input. But obviously, when you do that way too early on, it's not respected as an input specifically for the app and therefore doesn't actually progress the command line. So again, looks like Claude code is also a fail here. But while it tried better to interact with the command line, it still failed the task overall. And I did try running this a couple of times to make sure that this is not a single run artifact. And then found that there were one attempt once for Claude code and once for Cursor where they failed executing the task but still claimed to me that, hey, I've done the migration. I wonder why it tried to lie though. But very strange to see the agents still doing these kinds of behavior. Just reminds us that we should always, always check the output of the coding agents before committing or before proceeding to your next task.
So the next and final task we move into: tool use. I've set up both Cursor and Claude code with the same Puppeteer MCP to allow them access to interact directly with the browser. And specifically, I've asked them to navigate through the dev site. So, this is something I've already started based on the current repo and explore its functionalities. And I've already logged in the user so that it can check out the feature as if they were a real user there. And inside of Claude, it's interesting to see that how it's navigated to the different pages and it did quite a few screenshots to check the content. Originally, I didn't expect this to actually work because when I first tested Claude Code just a month or two back, it didn't have the image capabilities. So, great to see that added in. It's able to analyze the repo, navigate through the page, and even did some scrolling, interacted with a couple of UI elements to ensure interactivity, and there's places where it failed, but also corrected its own actions. Again, great to see how it's able to iterate through so much of this actual tool use. And at the end, what I really liked about the Claude code's output was the fact that it went through the entire flow of the app, tried out all of the different...
Pages, clicked on pretty much all of the buttons that existed in the app, and then even tried to log out the user and then check the different kind of authentication behavior for a not logged in user. So the intuition here was really great.
I wonder whether Claude 4 got most of this directly from Anthropic's previously developed computer use model and they shoved most of their learning in here. The results were really great, and the great summary here even gave me feedback on what they thought of the user experience. Probably being a little bit too generous in terms of the praises, but it worked really well.
And if we compare this to the cursor run, of course, cursor was also able to use the tools really well. Went through the same process of navigating, screenshotting, filling, interacting, and you know, sometimes even not able to input things properly using the page to interact with the page elements. And then there were places where it was waiting for things to happen, and it knew to wait and time out directly using the tool itself on the page for analysis to actually happen in the background. Really cool to see, and again, hit a few hiccups, was able to correct its own action along the way, and then at the end it produced also a decent summary of the actions it's tried. But in comparison, cursor's action was way more limiting in terms of the actual UI elements it was actually tested, because I watched both agents trying executing task across the board, and Claude went through probably twice the features that cursor did. And then cursor had quite a bit more limited interaction when dealing with those different UI elements and, say, for authentication, it doesn't try to test out the logged out users' behavior.
And obviously here we might say that if we created better rules or had a more specific prompt, cursor would have done better. I think that would probably been the case, but very interesting to see that Claude code had great intuition to do a better task than cursor did. So in this case, it's also a win for Claude code.
Now if we tally up the score, it's four for Claude and then two for cursor. So it should be easy conclusion to say Claude code was the best. But it's actually not that simple. And going through these code changes and going through the intake flow, I had to realize that while building these tests, trying to get consistent behaviors out of these coding agents, the thing that matters the most wasn't the actual agent you were using. As long as you were using one of the bleeding edge ones, the quality actually came very, very close. What made the difference was how much effort you put in to the rules files.
And then during the creation of some of the test to get the consistent behaviors out of both agents, I had to do significant additions inside of my rules files to get the great behavior out of the two. But once the rules file has been improved, both agents actually behaved really well. Take the full implementing feature test. Both were able to do really well at dealing with 700 line changes, working the first time, creating database schema changes, writing tests, and creating the feature all in single prompt. Right? And that was more due to the benefit of great specifications that was set out in the rules rather than just the capability of the agent themselves.
So really here, when I choose between the two, I would prefer Claude for doing things like exploration and more intricate issue fixing and for things even like bug finding. But I've definitely prefer cursor with Gemini and specifically for feature implementation because of the higher and different code quality that I get together specifically with Gemini. Obviously now with Gemini CLI also coming out, I definitely need to test that very, very soon.
So there you have it. Given the importance of the rules files, you should check out my video just up there to learn how to better write those things to improve the quality of your coding agents' output. Let me know in the comments below whether you'd like me to use these tests to put other AI coding agents head-to-head or if there are other scenarios that you'd love to see me covered. I'll definitely be looking to improve this test suite in the weeks to come to better tell myself when a new tool comes out whether I should be switching over to the new yet again cursor killer or maybe in the next time is the Claude code killer. And until then, happy shipping, and I'll see you in the next one.