Transcription
We created a pull request on GitHub that looked clean. However, we hit subtle bugs. We also purposely sent some of the pushed some of the credentials, email credentials, which could lead to security issues. And then we asked four AI code review tools, top AI code review tools, which you'll figure out in the video, and asked them to do a head-to-head comparison.
One of them created a beautiful architecture diagram. One of them actually reviewed it like a senior software architect or a senior engineer. And one of them just said, "Looks good to me" or "Let's get it merged." And one of them pointed out just obvious.
In this video, we're going to be comparing them one by one, top four AI code review tools in the list. Let's break them down one by one. All right.
So, the first one in the list is Kodo. So, what I've done is I've created a code change which is doing quite a bit of things: comprehensive user engagement, performance monitoring, creating engagement validation, enhancing the user interaction with clicks and all of that, error handling as well. So, I've already pushed the changes. If you look at over here, it is asking me to raise a PR. So, I'm going to raise a PR. Okay. And going to ask it to merge with the create pull request. Okay. As soon as you do it, let's see what is the first thing which Kodo does.
So, we've raised the pull request and if you look at over here, let's go back to the pull request. This is our latest pull request. Click on it and uh, first thing it does is it, using the bot, it tells us the changes. It tells us the PR type and then it gives us the description of what all it is going to change. It gives us a walkthrough as well, uh, with a diagram, pictorial view. So, it is saying that it is adding a CTA component with analytics, ranking, event tracking, and all of that, which is really good. I, I like it. Uh, first of all, I like it that it is giving me a pictorial view. I don't have to go through all the code changes. Okay. And these are my commits. So, I don't want to go through the commits altogether because there's just too many lines of changes.
What I'm going to ask is, I'm going to go back to the comment. I'm going to say, slash slash ask, what are the potential risks in this code? I am from a security background and I need to check whether it, it, it kind of lives by all the, um, security standards which we have set. That, okay, uh, it has reacted and then given us the suggestion, uh, that there are potential three issues. Like, the first one is the SSR hydration mismatch, and it is, um, asking us to change as well. You can apply it right over here and then suggest, suggesting it is important. The second one is, it is telling us that there are SSR, um, access issues as well. And the third one is, uh, throttle general. So, these are some of the issues which it is already.
So, first thing I wanted to check is that security compliance issues, and it is giving me that this is the performance risk, these are the security risks, reliability. So, pretty point-to-pointed, not bloating a lot, not spamming me a lot, and giving me this recommendation as well. Uh, another thing I wanted is, I wanted to add a bunch of docs. It has too many changes. So, I wanted to ask, "Can you add docs?" So, as soon as I do it, it should start generating results as well, that these are the docs which are important for this particular PR.
While it is generating the docs, as you can see, work in progress. So, what I'm going to do is, I'm going to ask it to review as. So, if you, if you don't want to say anything, if you don't want to put any custom prompt, what you can do is, you can do a slash review as well, and then you can probably say, "Update change log." So, a lot of times you generate PRs, developers generate PRs, but they don't generate change log docs. You need a separate team for that. With this feature, you can ask Kodo to create the change logs as well.
As you can see that it has started to review as well. And now it has started to create the change logs as well. All right. Within a few seconds, within a few microseconds, it is, um, created the change logs as well, which is the enhanced CTA, performance monitoring, session management. These are the changes it is doing. And that's it. And then you can check for the compliance as well. If you don't know how the compliance, it has already got a compliance flag which you can just type and it is going to tell you that, okay, these are the compliance issues which you have. You can also use flags like /describe, /review, /mps. However, you can start re-adding labels and do the compliance review as well. If you do the compliance review, where it's telling you that these are the potential compliance issues, these are the ticket compliance, and then custom compliance as well. All right, that's pretty much all from Kodo.
All right, the next one in the list is Code Rabbit. So, I've done similar stuff. I have just created a new branch, uh, making changes in the same, uh, CTA enhancement. So, it is doing the same stuff, adding user engagement, implementing performance, blah, blah, blah. So, I've already created the PR. Uh, you can see over here, it pushed the changes. Now, I'm going to raise the PR and as soon as I raise the PR, let's see what Code Rabbit does. So, I'm going to hit a refresh button and, uh, um, start probably with a small little suggestion. "Code Rabbit, can you help me refactor the repeated logic in user validation?" So, you need to start with @code rabbit AI, while you had to do slash and then the flag name for Kodo. So, while I've done that, uh, also the bot has already kicked in. It has seen what are the commits. This is all happening automatically. Um, for on my comment, it has already reacted. So, hopefully, it should take a couple of seconds.
So, while it is doing it, what I'm going to do is, I'm going to ask Code Rabbit to remember, um, uh, Kodo also generated a beautiful, um, sequence diagram, flow diagram, what all the changes. So, we're going to ask Code Rabbit to do the same. So, it's going to be @code rabbit AI and then generate, generate diagram, and then hit on comment and let's see, let's compare one by one, what is the diagram comparison between Kodo Rabbit and Kodo. All right, looks like it has created the diagram. If you see, there's, um, it has done a couple of things over here, regenerated the diagram and, uh, refactored the repeated logic as well. So, let's look at the changes. So, it's saying that, um, "I can see what you exactly mean. There's significant duplication across your validation." And if you see, um, it has given us the file and, and this is how you would refactor, how to refactor one of your components using the validation. You can just copy here, or what you can also do is, it, you can ask Code Rabbit to make the changes as well. So, these were the refactoring changes. Okay.
The next one is a sequence diagram. If you look at over here, compared to the previous one, it has, it has created a very, uh, detailed sequence diagram. So, it's saying that, okay, making the changes in the CTA file, and then you've got user engagement. These are the components it is trying to add. These are the repeated patterns, and if you see over here, it's, it has done a schematic diagram as well, and then it has hasn't stopped here. It has created, uh, a diagram over here as well, like component validation units and validation rules as well, uh, with repeated patterns and everything. So, these are the current issues. These are the recommended settings and potential issues as well. So, uh, pretty comprehensive. Sometimes it might look like spammy, but it, it looks at least, uh, very comprehensive at the moment.
So, I'm going to just end with a couple of more. So, I want, uh, to generate doc strings as well, like the docs as well, which we generated for, uh, with Kodo as well. So, I'm going to do the same and I'm going to ask it to suggest a couple of code changes as well. Probably suggest a bunch of tests as well. So, let's see how it reacts. All right, it has already created, uh, the doc changes. If you look at over here, it is trying to add the changes over here. And these are the doc changes. Not very comprehensive compared to Kodo, but that's the changes it has done. And it is created, it is mentioned in a pull request as well. All right, that's it.
Compared, kind of both of them head-to-head. Uh, few things I liked about Code Rabbit, few like sequence diagram and the detailing. However, when it comes to very code quality and, and keeping things very, uh, much to the point, Kodo kind of did the job over there. All right, I'll see you in the next one.
All right. So, unlike Kodo and Code Rabbit, where you can just command in your comment in the PR, Kodaci is another platform which gives you a similar experience. However, it gives everything in a managed SAS platform where you can go to app.kodaci.com and just configure your repository. You don't get the same level of experience, but you get everything at the same, uh, space. If you're a big fan of using SAS tools, um, nice UI, simplistic UI, you should, you should definitely give it a try.
So, if you go to the repository, you'll see I've got all my repositories configured over here. I'll go to the front end wherein I was making all the changes. If you see, it has given us metrics around issues, complexity, duplication of the code as well. These many issues, these are security ones, and these are the changes which is happening over here. If you go to the commits, if you'll, you'll see be able to see all the commits and then file changes as well. So, these are the file changes. What are the issues? If you click on the issues, it'll tell you that, uh, you can kind of filter based on the severity, like I want to see only the critical ones, maybe at the moment. So, it'll list all the critical issues like one of the application dynamically construct file path information, uh, is probably has got some access issues. User flows has got some issues. So, it, it not only suggests, it also tells us the pattern and the, uh, recommendation around documentation. And from here, you can straight away create the Jira tickets as well. You can complete the coverage as well, and then, um, you can, you can look at all the open pull requests and then you can select the coding patterns as well based on your team structure. If you can customize as well, you can add, uh, like, like if you want to, if you want to, uh, enable linting, or if you're working on the infrastructure, you want to enable check off or Terraform configuration. If you're working on, uh, JSON, then probably have some of the basic linting over here as well. So, you, you got a wide range of patterns enablement over here so that you don't have to write anything on your own. Uh, that's fairly it from the Kodaci. If you're a big fan of SAS and want everything at the same place, go ahead and give it a try.
All right, last one in the list is Gemini, and there is a reason of keeping this at the last one. Um, I've already, uh, raised the changes, have pushed the changes. I'm going to compare the pull request, and it's, it is pretty much, pretty much got the same changes. Um, one of the reason keeping it at the last is the maturity level is still not high, uh, as compared to the rest of the other tools, especially Kodo and Code Rabbit. Um, so what I'm going to do is, I'm going to say, Gemini, I'm going to say, "Summary." Um, if you look at over here, as soon as the pull request is raised, um, there hasn't been any auto suggestions given, unlike other tools. So, that's something, um, which what I noticed. I've created a summary. It says, "These are the summary of the changes until you've asked it." So, these are the highlights. These are the assistants, um, uh, given auto assistance given by Gemini. Um, and then, uh, it has already mentioned that you can use, uh, these commands as well. So, I'm going to use the review. There's not much you can do, uh, because you've got, um, uh, only four of these two, uh, flags only. So, I'm going to say, "Review." And after that, I'm probably going to say, "@code assist," and, um, I'm going to, I'm going to ask if it can catch a compliance issue. So, I'm going to say, "Can you see if there are any security issues in the code base?" So, I'm going to do that and, uh, give it a couple of seconds to review those changes.
All right, it's been a couple of minutes. I have again given Gemini review and it could not detect any changes. Neither it could, um, react on my customized prompt, which is I wanted to check whether it could flag any security, uh, issues. Beyond these issues, there are a bunch of other things as well. It doesn't support all of the languages at the moment. So, um, that's not something which is preferable if you want an AI, uh, code review tool. That's it for it. I'll see you in the next one. Thank you.
So, what we've done is, we have picked up these four AI code review tools. Treated them like a team member, like a senior engineer. Asked them to review the code changes one by one in five different parameters: code quality and compliance, custom prompts, which one does a better dog generation, change logs, dev tool extension, and security and compliance. Few of them kind of did the job. Couple of them could not, as they could not figure out the tool changes. However, in this era where code generation is simpler than ever, making sure to review them with the right parameters is even more important.
If you care about code quality, go check out Kodo's recent blog wherein they've listed all of these tools, head-to-head comparison between all of these five to seven tools. Which one should you pick up? What are the parameters? What are the skill sets? What are the support levels? What are the commercials which you can get started really quick? I hope this was informative. Don't forget to subscribe if you're liking such videos, and I'll see you in the next one. Thank you.