Transcription
I know you've seen it in the news. AI is finding bugs in software. And it's not just finding a couple. It's finding hundreds of bugs in software that we've been using for a very long time.
I want to use this video to address some of the questions in my comment section about why is AI finding bugs? Like why is it so good at finding bugs? Why aren't humans this good? And what what makes it different than like a normal static analysis tool, right?
Before we answer these questions, I want to make a very strong caveat. I don't think that AI is like this like mythical creature that like we're creating that we're like giving consciousness to and it's eventually going to be its own person. No, I think AI is literally just a big bucket of vectors, right? It's just a bunch of parameters that when you shoot your vector, right, your tokenized words into other words come out. It is a very very good text predictor.
Now, to make these AIs profitable, they have to make it good at certain things depending on what they want their users to be paying to use it for. And recently there's been obviously a big push to make it very very good at code generation. Like at basically sonnet 46 there was kind of this big exponential increase in like the quality and the the overall shape of the code that it outputs. Now an accidental side effect of it being able to produce good code is its ability to read code and as a result find vulnerabilities.
The reason for this breaks down to three primary things that AI is really really good at based on just the nature of what an AI is right. static analysis, the ability to reason about a codebase. Vulnerability research, the ability to find bad patterns in a codebase, and then just general automation. A computer can do this on repeat forever.
So, if you're new to this channel, maybe you don't know what's going on here. What is static analysis? Right? The act of finding a bug in a codebase first depends on your ability to know what the code is, right? You can't find the use after free in the codebase. You're not really sure how it how it works. Static analysis is literally just reading code and internalizing context about what the code is doing and what it is supposed to be doing and what it does incorrectly. Right?
As someone who's done a lot of static analysis, also called reverse engineering in some cases, right? When the codebase is not white box, it's closed or you don't have the actual source code, you have a binary. The hard part in doing this process is literally just the limiting factor in doing this operation of of reading code and understanding how it works is it's limited by how much I can read. It's limited by how fast I can look at text on a screen and process it as a human being. I don't know the actual hard number on this, but I've read that there is like an upper limit to the amount of code that a human can produce or read in a day. And it's on the order, I think, of like thousands, maybe 10,000 lines of code. Now, not on production, but on like ingestion and like understanding.
AI does not have this limit. You can point your model at like a 400,000 line codebase and it will just boop. Yep, I get it. In a matter of minutes or maybe like a couple minutes, it can completely take on a codebase. This is a tension problem that humans have that AI naturally does not have. And again, not because it's like this big mythical creature. It's not because it's sentient. It's literally because it's a computer. It just has a lot of memory, right? the VRAMm of whatever GPU you're running your model on can just hold that information and the human brain cannot. So again, step one, figure out what the codebase does. AI is very good at very quickly understanding what the codebase does.
Part two, vulnerability research. So, so what is vulnerability research, right? When humans author code, when people author code, they are writing code with the intent of performing some kind of action. But there may be edge cases in the functionality that gets you there that create bugs and those bugs create these things called primitives, right? primitive is like a bad way the code behaves that can turn into an exploit, right? It gives you an outofbounds read. It reads memory that it shouldn't read, an outof-bounds write. It writes memory that it shouldn't write. It there's a use after free where you free memory back to an allocator and then you use it somewhere else on accident, right? You overflow a buffer because a length value was trusted when it shouldn't have been. All of these things that turn into vulnerabilities that hackers use to create exploits are patterns.
The example that I use all the time for like a pattern that is common in finding overflows for example on like crappy IoT devices is a mem copy with a user controlled n right take for example you have a TLV structure a type length value structure and the L value you have the type of the data the length of the value then the following value and all that gets read off the network right if that L value is trusted by the device and not validated against some upper bounds which unfortunately in 2026 does happen that is an overflow condition right and that is a very easy pattern pattern to look for like literally all I have to do is find the receive function trace the data find the mem copy and look for guards in the middle that ability to do the source to sync analysis for a human like a single function or a couple functions is very easy but if that happens across a very large codebase with maybe multiple steps to get the structure populated with data it's a little more complicated but because the AI from the static analysis already has the context of the entire program in its like computer brain or whatever in the VRAM it's very easy for it to do this kind of pattern recognition against the codebase. The only really hard one is like use after freeze cuz those are more temporal than spatial. But that's a whole other different topic for a different video.
Another reason, by the way, that AI is so good at vulnerability research is the people that do vulnerability research, right? Have to like get very smart on the process of reading code, reading assembly, reading using tools like Yedra and IDA. And at the end of the day, they have to kind of choose a specialty, right? Like choose your class of vulnerability research. Maybe you're an IoT person, maybe you're a hypervisor person, maybe you're a person that finds sandbox escapes out of browsers, right? The reason that people choose these specialties is because to know I said before the first part of static analysis, right? Knowing how the codebase works so you can actually find bugs. It takes a lot of time and a lot of knowledge to know like how do the internals of V8 work? How is an IoT device set up and how do you get the firmware out of it and what actually matters for an a vulner vulnerability in V8 versus IoT. These these silos, right? These columns of research are kind of the specialties that people pick and it's very hard to be a really really good expert at like two of these fields because it's hard to know a lot about all of these things.
Do you know what's like really really good at researching topics very quickly and pulling weird chipset spec sheets off the internet and learning about them quickly? AI. AI's ability to take a researcher's intuition, a researcher's kind of code smell, knowing historically what could go wrong and having it research that topic and learn about it very quickly over a codebase is what makes it really really good at VR. So the pattern matching and also its ability to source information very quickly.
And then finally automation, right? The ultimate like the real winner behind all of this is not so much again like the ability for it to reason about the codebase is good. The ability for it to do this pattern matching is good. But the more crazy part is you can literally I'm I'm going to say some words that if you've been on Twitter for the last couple weeks, you may get triggered by this. You can set up a loop. I know, don't don't leave where basically you can just let this process of reasoning and automating the process of finding bugs in software. You can let it run forever. You can just let it go and do its thing. The AI doesn't need to take a nap, right? The AI doesn't need to take a break so it can actually think about the codebase. It can just continue to do this. Obviously, you have some limits there where you have like the context window, right? The AI can only reason about so much code at one time. So, you can't necessarily give it a whole codebase. You have to give it kind of like snippets of a codebase at a time so it can like make good deductions of that codebase. That being said, it can just run on this forever in the loop.
By the way, the problem with all of this, at the end of the day, the AI is going to make stuff up. It is going to hallucinate, find bugs that aren't real, right? Sean Healing wrote a blog post about this back in like the chat GBT like 03 days where he was trying to get AI to find a bug in the Linux kernel's SMB handler that he knew already existed. In the process he did find that bug and he also found another bug in doing that just using all AI which is really really cool. The problem that Sean highlighted at that time, and this is still the case today, is that of the two bugs that he found, the AI reported, I think it was like a 100 or 200 bugs, like a signal to noise ratio of like 1 to 100, right? Very, very bad. If you as a human have to now go through and triage all those reports, uh, take it from my experience where I've previously worked, um, it's not it's not a good time reading those reports. Those reports suck because the AI is so insistent that it has found a real thing and it's on you to deduce what is real.
The easy way to easy in some cases depending on on the technology to end that loop is to use either asan or ku asan right basically a harness or like a framework that wraps around the codebase that can detect out-of- bounds reads writes on these on the software you take the stochastic nature of the model right in your harness and then you tell it okay write me a fuzzer or write me the asan harness for this codebase and prove to me that the bug is real to try to like reduce the amount of bug reports that you have that are fake.
So what's like the take away from this, right? What do we do as like an industry with this? Right? There's a lot of fear that especially I think like the security VR world that like AI is going to take the job of vulnerability researchers. That may happen in the short term, but I think companies with brains will realize that like oh using these AI tools, we can take our already like cracked really really smart VR people and we can turn them into a 100 copies of themselves, right? We can have them like task their agents using their intuition, using their knowledge of already what likely will be wrong in a codebase and point them at code bases and be able to do this automation very quickly um and scale up the work that they already do. It's becoming more and more difficult to ignore the fact that these tools are getting very good at automating a process that normally took a person with a lot of skill a very long amount of time. It still takes a skilled human to drive these things and that will never change in my opinion. with the right person, with the right tool and the right harness.
That is why these tools are finding bugs so quickly. They're able to reason over code bases very very quickly, do pattern matching, and then output code that can be used to prove that the bug does or does not exist.
If you want to try this right now, go pull firmware for some crappy old IoT device like a router, like a camera, and then use Gedra MCP. or if you want to use an open source codebase, go pull like an old version of engine X, right, that you know a bug already exists and just go start talking to your AI about the codebase. You will find very quickly its ability to understand more about the codebase than you. I literally took a a device that I had worked on previously that I stopped working on because it was using some like very obscure chipset and all I had was literally the firmware pull. It was a binary file. I spent a long time trying to diagnose like what the actual instruction set was, what chip it was using, etc. I pointed Opus 47 at it, I think, and it was like, "Oh, dude, it's this chipset. Here's the entry point. Here's the uh interrupt vector table, and uh yeah, here's the function. Here's like the arts startup." Like, like almost instantly. It's becoming more and more difficult to ignore the speed at which this thing moves.
So, anyway, I hope it answers your questions. If it does, hit a subscribe button and a like and a comment, and uh we'll see you in the next one. Okay, goodbye.