Local AI Coding is Finally Good Enough

So, I've been wanting to make this video for a long time now, but I never could because frankly local AI was just not good at coding. You'd spend more time debugging the nonsense code that it generated than just writing it yourself from scratch.

But, things have changed recently. Local AI has gotten better, and believe it or not, is finally good enough at coding. And this video is to show you what they can actually do by coding in a real TypeScript code base and a real Rust code base. And I chose these specific code bases because they're actual production software used by thousands or sometimes millions of people. Not the hundredth app I've I've coded last week because that's frankly a completely different use case.

And I know your first thought here, so I wanted to address it. Local models don't make any sense when you have companies like OpenAI and Anthropic subsidizing an insane amount of inference on frontier models, meaning that it is both cheaper and better to go that route. And you're not wrong. If you can go that route, do it. Why not?

But, there's a glaring issue here, and that is a lot of devs can't use these models on their code base because these models are hosted in the cloud, somebody else's server. I actually have a lot of friends who work on defense contracts with ITAR-controlled code, in health care dealing with HIPAA regulations. They have IP-sensitive work, and some in finance hedge funds who they have a policy that says no code or data can leave the building.

Now, there are ways to get around this, like BAAs or FedRAMP or GovCloud approved vendors and a handful of other ways. And some companies may go that route. That is also fine, but that still doesn't solve the problem for everyone because even if a cloud provider provides a compliant path, your company still has to approve that provider, approve that model, approve that region, approve that data flow, and approve sending your code to the third-party processor at all. And that's if your company even goes that route. Because again, sometimes you can't use any third-party processors, period.

So, long story long, my friends are stuck either choosing to handwrite every line of code like some sort of caveman or running local AI models on their own hardware. This goes for the devs that I know as well as the business owners that I know. Because like I said, that GovCloud path and the other paths are not that straightforward. So, if that's you, or you just simply want to retain control over all of your code and all of your data and all of your IP, well, this video is for you and I think it'll really help.

Now, there are a lot of local AI models out there. Some are better at coding than others, and the two that I'm using in this video are Qwen-3-72B, which is the bigger model here. It's an 80-billion parameter MoE model with about 3-billion active parameters per token. And I'm running this quant version of it. And the other is Qwen-3.6-27B, which is the smaller dense model. It's a 27B model and I'm running this quant version. And for the Frontier baseline, I'm using Opus 4.7. I know Opus 4.8 is out. It literally just was released a couple hours ago, but I already dialed the coding. Frankly, the point of Opus is to give us a reference for what one of these Frontier models, how they would perform on these tasks. It is not meant to be a comparison between models. If that was the case, I'd be doing it on the same hardware. Opus runs on the best hardware that Anthropic's billions of dollars can buy. And the local models are running on the best hardware we can buy, which I've partnered up with AMD on this video to obtain and showcase what these models can really do.

So, for this local AI coding test, I'm using an AMD Ryzen Threadripper 9980X system with an AMD Radeon AI Pro R9 700 GPU. And the R9 700's 32 GB of VRAM gives me the headroom to run the 27B Quinn coding model fully on the GPU in this setup. While ROCm provides GPU acceleration and the Threadripper CPU gives me room for larger model workflows and CPU offload, which I'll be needing for the 80B Quinn model. And to top it all off, we have a [snorts] rather comfortable 128 GB of DDR5 RAM.

So, with the 32 GB of VRAM, I'm obviously running the quantized GGUF models. And Quinn 3.6 27B fits fully on the GPU. Quinn 3 coder next, however, does not. It is much larger. So, for that one, I'm using llama.cpp's MoE CPU offload with this, which that's what I'm using for running these models, llama.cpp. You can use oobabooga, you can use LM Studio, that's fine. But, I found llama.cpp to provide the best performance for these models. And for the operating system, I'm running Ubuntu 26.04. And as you can see when looking at all of these specs, this is a machine specifically built for running local AI models. I mean, look at the description for the GPU itself. And if you want to learn more about what AMD has to offer from CPU to GPU to ROCm to everything else that it provides, really leaning into local AI models, I'll leave a link in the top of the description. I highly recommend you check it out.

So, that's the hardware, that's how I'm running these models. Now, let's get to the actual code. And if you're wondering why I keep pointing over that way, that's where the PC is. Here you go. And yes, I'm aware that I have way too many computers.

So, for each codebase, I gave each model two tasks. One easier task where the model mostly needs to follow existing patterns, and one harder task where it has to understand more of the system and touch more of the architecture. First up is the TypeScript codebase Excalidraw, which I'm I'm you've all heard of Excalidraw. I had the model start with the easier task. This was the prompt and basically it was to add a highlighter mode to the free draw tool. So ideally the way it works is you would click the free draw tool and over in the settings for it there is a highlighter mode that you can click and it automatically turns it into a highlighter. Is this similar to just changing the opacity? I suppose. But we're not making practical features, we're just seeing if AI can make features. And actually I I pulled all of these feature requests from issues from the actual repositories for all of these codebases. So this is something that people have actually actively wanted enough to make an issue about it or just put in a feature request for something like this.

Anyway, for this one I put Opus up against Quinn 3.6 and both of them passed the TypeScript check. So it's looking promising. And spoiler alert, both of them actually properly implemented the lighter feature. But what's more interesting is how they implemented it. So let's take a look at Opus first. Opus modeled highlighter as a real property on free draw elements. Then you can see here that the rendering code checks the semantic flag, which based on how other things are done this is the clean proper architecture for implementing this. The element actually knows it is a highlighter stroke. So after creation or save, reload, export or future editing that intent is still represented in the data model. But Quinn 3.6 took a more direct approach, I may say. When highlighter mode is enabled, it creates the free draw element with modified normal properties. So what that does is it gives you the visible behavior, but after the stroke exists it's not really a highlighter anymore. It's just a regular free draw element with a large stroke width and low opacity. So, it still works. When we look at both of the examples side by side, they look the same. But with Quinn 3.6, when we're looking at the actual code, it's not quite as good. Because with Opus, where the highlighter remains a part of the element's data model, it doesn't work that way with Quinn 3.6. So, they both work, but the code quality is not quite there on the local model.

Now the harder Excalidraw task was to create a five-pointed star shape. And this was the prompt that was used. And this may not sound like a difficult task. But adding a shape in Excalidraw is not just drawing a polygon. Because when implementing a feature like this, you have to touch the toolbar, the element types, the rendering, the hit testing, the collision, the restore logic, and anywhere the app assumes it knows every possible shape type. Because right now it has what? Squares and circles and triangles. Oh, no, it doesn't have a triangle. It has a diamond. So, it seems easy. But I mean, the architecture is there for a model to be able to look at and say this is how they do the diamond, this is how they do the circle. Let's just do the star that way. So, let's see how they did.

And again, both Opus and the local model, which this time was Quinn 3 Coder Next, got it to work. And for clarification, I do swap every now and then between Quinn 3.6 or Quinn 3 Coder Next. I don't take the best from either one. I just kind of chose them at random, to be honest with you. Let's take a look at the Opus one. What's interesting is that you'll notice in the toolbar that it overtook the five keybind. So, now there are two things that say if you click five, it'll use that tool. However, it overrides what was previously five, and now that other five no longer works, obviously. So, that's not ideal. But again, it still works, and it works exactly as expected. And if we look at the code, it looks very deliberate. I mean, we can look at how it creates the actual star with this star-specific geometry that it's using. And it keeps the star-specific distance and collision handling separate instead of accidentally routing diamonds through star math or something like that. And you can see just by the UI that it added the star type, the toolbar entry, the icon, the restore support, and rough JS rendering, and it passes [clears throat] the TypeScript check. In the other examples, I'm like, "Yes, it compiles. No, it compiles." So, this is my version for TypeScript, it's like, "Yeah, it passes the TypeScript checks." In case you're wondering why I'm so focused on it.

Now, let's take a look at the Quinn 3 Coder Next version. You will see that it also is a part of the toolbar, but it didn't override any keybind. It didn't give it a keybind, but I'd rather it not give it a keybind and wait for me to prompt what keybind I want than override an existing keybind. So, with that, Quinn 3 Coder Next, in my opinion, did a better job. However, when you look at the code, not so much. You know how I said it was good that the Opus version didn't accidentally route diamonds through star math or anything like that. Well, that's kind of what Quinn 3 Coder Next did. It tries to generalize diamond and star collision handling, as you can see here, which that part sounds reasonable. But, when you look inside that helper, you'll see that it always uses star points. So, parts of the diamond collision path can now run through star geometry. That is a bug, a bug that the type checker will not catch because the code works, it passes all the checks, the star draws, it's on the toolbar, everything on the UI looks perfect. If you're a vibe coder, job well done. However, now you have a a slightly broken behavior for another shape. These are the minute details that add up over time and these little bugs that appear that if you're not actually checking the code will compound and not be so pretty after a week or a month or longer. But again, still works, something you can catch. I mean, if I were to try to give this to models a year ago, it wouldn't have even come close. Local models that is.

Now for the Rust code base, Warp, which if you didn't hear, maybe a month or two ago, they open sourced their entire code base. So I figured this is a good one to test the models on. The easier task was adding a clear history {slash} command where the idea was to clear the local agent's conversation history for the current pane. There's already existing {slash} command patterns in there, so follow those. That should make it easier for these models. And both implementations, Opus 4.7 and Quinn 3.6 on this one, pass cargo check, which is good. We actually have both of them working. The difference here though is what each model prioritized. Let's start with Opus. It didn't quite accomplish the task that I wanted it to. What did I say that it was supposed to clear the local agent's conversation history for the current pane. But as you see, when I type in {slash} clear history, it asks to delete that entire conversation, which is I guess one way to clear the history, but it's not exactly what I was going for. So in other words, it failed. The code however wasn't bad. I mean, it doesn't do what I want it to do, but the code is not bad. Let's take a look. So Opus wired the command through the existing workspace action and confirmation dialogue system, which fits the app architecture pretty well. It reuses an existing confirmation dialogue flow and extends it with a new confirmation kind. But after confirmation, Opus calls this self.delete_conversation_with_toast. And as you can see, when we manually test it, it does exactly what it says it's going to do. It just doesn't happen to be what we asked it to do. It deletes the entire conversation. So, while the UX was clean, you saw that when you typed in /clearhistory, it recommended it just like any other command, and when you hit enter, it asked for confirmation to delete that conversation. And architecturally, it's polished in kind of how it's supposed to be, it's not what we wanted.

Warp 3.6, however, did actually do what we wanted. So, you can see that it performs similarly in the UX. You type in /clearhistory, and it shows as a command, and you hit enter, but it says, you know, it's like, are you sure? So, you have to hit enter twice, and it doesn't clear immediately until you move your mouse, then it disappears. And you can actually see that code right here, where it makes clear history require a second run to confirm. And then what it actually does is it truncates the conversation from the first exchange. So, weird UX. I think the double tap of the enter is odd, but it did clear the history, even if it is just truncating it. But, the conversation was still there. Actually, correction, it looks like it only cleared the history from that view, because now that I've closed Warp and opened it back up again, we can see that that conversation history is still there. So, in this next clip, when I say it does what we asked it to do, that's not entirely correct, but it doesn't look like we're too far off. So, this is a weird one. Warp 3.6 did what we actually wanted to do, not super well. Uh UX was not as clean as Opus, architecturally not as clean as Opus, but again, did what we wanted to do.

Now for the harder Rust task, which was command bookmarks. The idea was that when you ran a command and you wanted to save it, you could right-click on it and bookmark it. And when you wanted to view those bookmarks, you could open up a side panel to view all of them just like you would your conversations. And to implement something like this, the AI is going to have to go into terminal history and context menus and the left panel UI persistence SQL light schema and command execution. Because when you click on the bookmarked command, it'll run it.

And if we look at the code changes that Opus 4.7 made, we will see that it added a real command bookmarks module, persistence changes, SQL light schema changes, terminal context menu handling, and a left panel view. Everything that we asked for. You can see the code for the sidebar panel for the saved comments. You can see where it gates the whole thing behind a new feature flag as well, which I did end up going in and disabling because even though that is following Warp's best practices, I don't need that. So, after I did that, cargo check and it all passes. And then when I come over here in a Warp terminal and run a command and then I right click that command block, you will see right here where I can bookmark it. And then when I right click it again, you can see that I can unbookmark it, confirming that that command is in fact bookmarked. Which I have to tell you it that way because I don't see anywhere how to open the bookmark side panel. So, either this is really bad UI UX or it didn't add it. And looking at the code, we see that it did create the sidebar panel and it added an icon that was supposed to go up here for me to click and display that bookmarks panel. So, it should work. However, Opus decided to create its own new panel type instead of just integrating into the existing left panel state model. And so, that panel type is not fully integrated, therefore we don't see the icon that's supposed to be there. And therefore, we can't view the bookmarks, nor can we test clicking the command that is bookmarked to see if it works. But we can look at the code, which is this code. And I want to show the prompt next to it as well, because in the prompt you'll see that when clicking a bookmark, it should rerun the command. However, in the code, you will see that it inserts the command, but it does not execute it. So, it only does half the job. Not the biggest deal in the world, but not exactly what we wanted either.

And then we have Quin 3 Coder next, which did worse. It did touch the right general areas, though. Persistence, terminal view, action wiring, left panel UI, and schema changes, but it didn't compile due to 47 previous errors. It's not just some, you know, one-off missing import that made it not compile. There were issues like missing UI variables and wrong Warp UI APIs and persistent type mismatches and missing enum variants and non-exhaustive matches and moved values. And it tried so hard to try to fix these errors time and time again and so many times failing those times that it just gave up. It said, "Given the complexity, let me stop here. The bookmarked commands feature is mostly implemented, but has compilation errors that need to be fixed by someone familiar with the Warp code base's UI API." So, in other words, this is a clear ceiling for these local AIs, or at least this model. It didn't do it very well. This was not a good test for it.

So, there we have it. Now, I'm going to let you come to your own conclusion based on those results, but I'm also going to share mine. And that is frontier models are better than local AI models on your own hardware. Who would have thought? >> [laughter] >> But again, that's not what this video was about. Opus 4.7 was just a baseline, and I know Opus 4.8 was just released. That will be in line with 4.7, GPT 5.5 typically is a little bit better, but it they're all frontier models.

But focusing on the local models, the Qwen 3.6 27B and Qwen 3 Coder next, they actually got some work done. Was it the cleanest architecturally based on the prompts we gave it? No. But I found with a better spec sheet and more rigorous requirements and explanations of what you want and how you want it, it performs better than what it did, not better than the frontier models. And particularly on the easier tasks or more split up tasks. So what what I'm trying to say here is that with the local AI models, you have to treat them as if they were models that you're coding with maybe a year or two ago. You have to be very specific. You have to give it more information, which this is good prompting practice anyway, but you also really need to break those tasks into much smaller tasks and give it one by one by one, maybe with context of what's going to do next as well. And that's where you're going to get the best results. So I stand by my claim that local AI models are finally good enough when you look at it from a standpoint of being able to help you in your work as a software developer, not comparing it to a frontier model.

And there's one more thing that is very, very important here that I have yet to mention. The local AI model on my beautiful AMD PC took at least five times longer to complete the task at the level that you saw than Opus did. So this is something where you're not going to wait around for the model to finish, which you really shouldn't anyway, but really not with the local AI models. It's something where you can have two tasks, you work on one, it works on another, or you have it work on one, and then once it has performed what you needed to do, you review the code, and then give it another task. Basically, you have two tasks running simultaneously. You're working on one, it's working on another. And I mean, let's face it, all of us have been assigned tasks that are less favorable or interesting to us than other tasks. So, now you can have AI help work on the mundane tasks, while you can tackle the interesting tasks. And I think that's a beautiful use case for AI when it comes to coding.

So, if your code can't leave the building, and you need to use a local AI model for your development work, I think you're in luck, and I think it can really help you in one way or another. But you let me know. What's been your experience with local LLMs and coding with them specifically, and maybe some other ways that you can use them to benefit your workflow as a developer or otherwise. I'm curious because I really want to use this PC a lot more. I want to use these local AI models a lot more than I have been. So, any idea that you can give me that I can try out, well, I'm all ears, and I'm excited to try it. So, I'll see you in the next one.