Transcription
There was a time last year where most of the engineers that I knew were starting to use AI coding assistants. But I refuse to be one of those people. And I know that might come as a surprise to you, especially if you've seen any of my recent content. But I was one of those people that thought that large language models were great for helping you plan your development. But why in the world would you trust them for implementing and validating?
Now, obviously, my understanding and my AI coding systems have evolved since then, and I use AI coding assistants to help me write all of my code. But validation is also a very important part of what I like to call the PIV loop. This is my system for using AI coding assistance that I've covered a bit on YouTube and also I'm covering a lot in the Dynamis Agentic coding course.
So, validation is all about giving your coding assistant the ability to check its own work after it has finished an implementation. And coding agents checking their own work is super common. You'll see it all the time. But I still think that validation is the most undervalued aspect of any AI coding workflow because most of the time when you see this kind of thing, it's just have the coding agent write a couple of unit tests, do some linting, maybe run the application, but there's nothing that's really end-to-end validation.
And so I took it upon myself as a challenge recently to see how far I can push AI validation. And I have gotten some insane results. I basically created a process for your coding agent to create a living and breathing system for validating your entire application end to end. Basically replacing all vigorous manual testing you might have to do yourself and going through all the different user flows. And I've applied this to some pretty complex projects with a lot of success. And it all boils down to a single prompt that I'm sharing with you in this video.
So what we have here is a meta command. You can run this on any codebase with any AI coding assistant. And so it'll do deep research on your project, figure out how it can validate things completely end to end, and then it's called a meta command because it creates another command called validate.md. This is what you can run going forward to kick off this full end-to-end validation cycle. So I have a link to this in the description. You can go and try this out right now on any project. And yeah, you might already have a system for validation, but you'll probably be blown away by what you can do with this. It might seem overengineered, but I've been so impressed with this.
And let me be clear, running this yourself is as easy as bringing the command into your codebase and then for cloud code, you can just do ultimate validate command. Boom. No arguments or anything. You just send this off and it's going to analyze your codebase and create that validate.md. And then if you are using a coding assistant that doesn't support slash commands like cursor for example, you can just say go and read and then give it the path to this command like right-click copy relative path and then execute these instructions now. Right? So commands are just prompts. It's okay if you don't have slash commands in your AI coding assistant. You send this off, you get your validate.md, and then you can just run slashv validate once it's created. Probably iterate on it and like validate it yourself. You want to, you know, validate the validate command. But once you're really confident with what you got here, you can use this going forward to like go through that entire process and save you potentially dozens of minutes every single time you build a new feature. So it just makes your development workflow, all your cycles that much faster.
And this entire thing started when I was working on this application. This is my remote coding agent. And I've been pouring hours and hours into building this. You can think of it like cloud code for the web, codeex or factory, like all those remote coding agents, but it's entirely custom system that I built and I built it in a way where it's super extendable. You can integrate with a ton of different platforms like Telegram, GitHub. I'm adding in Slack. You can pick your coding assistant like Cloud Code or Codeex. I'm adding in Open Code. It's pretty cool. And I'm giving this all away to you, by the way, but only in a live stream that I'm doing on November 29th. So, check the description in the pin comment. Go enable notifications for that stream. It's going to be a blast and you're going to get all of this.
And I'll show you really quick like going into GitHub, I can mention the agent and through the GitHub web hook, it'll kick off my coding assistant to handle this entire issue for me and even open up a pull request. And I can inject my own custom commands and process into this. And I'll show you something kind of similar in Telegram as well, just having it summarize a readme on the current GitHub repository that I have it operating on. And this is entirely remote. Like I can even do this from my phone. It's super cool.
But you can see as I'm demoing this here that it's pretty complicated. Like every time I implement a new feature, I have to test Telegram. I have to test GitHub. I have to try the different coding assistants. And so as I was going through this, it was a bit overwhelming all of the validation that I had to do. And so I started thinking to myself like how could I create a process for the coding assistant to go through all these different user flows and test these different integrations so I don't have to myself. And a skip and a hop later, I ended up with this ultimate meta command that I showed you earlier. And I'm just kidding. It actually took a while to get to this point. But I just started with that problem. I have too much to validate and I need the coding assistant to help me. But I still need to have confidence. That's why I needed to go through all of the user flows. Basically testing the app just like I would do if I was testing it rigorously.
And so even using tools like the GitHub CLI to actually create issues and pull requests and tag the agent, go and read the pull request comment after to make sure that everything is aligned with the request. Doing all of that completely end to end. And man, it has been incredible seeing it work. Even go for like 45 minutes with the validate command that I have for this that I'll show you in a little bit. But it did take a while to get to this point, especially this more generic version that you can now use on any codebase.
But it started really simple. Like I'll actually show you an example of this right now. I'll create a new command called pleasehelp.md and I'll show you kind of what it looked like at first for me to just ask the coding assistant to help me with this. So I just started with: "I need help validating my project. There are just so many different edge cases and user flows to test. So I want you to analyze the codebase deeply to understand all the different flows, edge cases, how the application is structured, how we can really validate things end to end." Now this isn't exactly what I typed at first, but it's pretty close. So I'm giving you kind of a live recreation here.
And after you understand the codebase and user flows, I want you to think about how you can validate this app end to end and what tools you would use specifically. So, for example, this is a Docker application. We're using GitHub. I want you to think about how you could use the GitHub CLI. Think about all the tools you can use to validate things as a user would. And I think this is like the key that really unlocked this next level of validation for me. It's just like telling it to act as a user and use tools like the GitHub CLI to do a lot more than just simple unit testing with justest or piest, for example. And so like if I were to do things end to end myself, how can you mimic that? And when I say end to end, I mean end to end. Do not hold back. Like, yeah, usually coding assistants overengineer and I try to tell it to keep it simple, but here I wanted it to generate a massive validate document so that we have this command that goes for a really long time going through all these edge cases.
And then finally, I want you to help me create a command that does all the validation end to end. I want it to be extremely comprehensive and also include any of the other testing we're already doing here, like, you know, the unit testing, the linting, the type checking, like all of that. I still want to include that, but most of the prompt is going to be the end-to-end testing.
And so I threw this into the coding assistant and what it came back with was pretty much what I accepted right away. I did some iteration on the tooling and things like that, but I'll show you the validate command that I've been using on this project and I've run this probably a dozen times now. It is super, super accurate. Like it's actually so consistent every single time. It goes through all of these tests. So take a look at this.
So first of all, it does all of the higher-level testing like the type checking and linting and unit testing, all that. But now it gets into this test repository setup. It will literally create a new GitHub repository for me so that I can test my remote agents on top of it. So it cleans things up. It creates the GitHub repo and the web hook so I can receive those mentions in GitHub. And then it will mimic some of the testing for Telegram. And it even created its own code, like it injected API endpoints to mimic Telegram functionality. It's so cool. And so it does all that after it does its Docker testing. So that's the test adapter. This is kind of like the Telegram simulation.
Then after it does that, it even does database validation. So it checks to make sure that my conversations are being stored in my database. Very, very cool. Then it does the GitHub integration testing. So it'll create an issue, mention the agent. Like man, it cannot get more comprehensive than this. It makes sure that the pull request is created and it validates all that. It tests to make sure that it can kick off different coding agents in parallel. It says it's optional, but I have seen it do this every single time and it works really, really well. Um, and yeah, it also tests all these custom commands that I have it create for like priming and planning and executing, kind of like the PIV loop that I was talking about earlier.
So, yeah, I don't need to like keep going through all this right now, but yeah, you can see how long this is, and it's still super consistent because it's just a super structured step-by-step process, even with a really nice summary report at the end. And I guess the AI coding assistant is just trying to maximize my dopamine here cuz I get all of these green check marks at the end. I'm actually showing you a previous execution that I did of the full validation workflow. And it's not that long overall because it's mostly just a ton of calls to Docker and the GitHub CLI. So I can scroll all the way back up to the top where it runs that higher-level initial testing like linting and unit testing. Then it creates the GitHub repository and validates all of that. Um, and then it does everything in GitHub, even validating the database.
And sometimes it makes these mistakes, but it always corrects itself. And so, yeah, the validation is non-deterministic because we're leaving it up to a coding assistant, but like I said, it's caught real bugs and it's just fascinating to watch it rip through all of my user flows and edge cases. It just gives me so much confidence. Like the bugs that it's found are things that I don't even know if I would have found myself doing manual testing. And so in some ways, this is actually better than me validating. And I think that there's still room for me to test things myself. And I wouldn't ship this to production only trusting its validation. But me plus the AI coding assistant is just way better than only me, especially when the testing is this comprehensive.
But yeah, looking at my private repos, you can see that I've been testing the heck out of this. And I've been getting really consistent results. These are just some of the repos. I've been cleaning up a lot of them. But yeah, let me show you how comprehensive this really gets. Like if I click into one of these issues, which by the way, all of these issues and pull requests, the validate process has created itself. I've made none of these. So if I click into this one, like "add a footer section to the readme," just a really simple issue because I want to test things on a high level. I load in the different commands that I have. So I'm bringing in my own system. I get the response, which it's me every single time, but every other is actually the remote coding agent. And so then I invoke my prime command so that it understands the codebase and it reads the readme. So there we go, it gets the project analysis. And then I ask it to, or rather the automation for the validation asks it to plan the feature. So it's invoking commands. It's going through this entire PIV loop of priming, then planning, then executing this entire process that now I don't have to test myself. And it even goes back and checks these comments, comments to make sure that everything is aligned. And then it invokes this plan in this feature branch that it also created as a part of the automation. Like you have no idea how much is truly going on behind the scenes. And then it creates the pull request in the end here. So there we go. PR number 13. I can open this up and I can see the final result of my test here, which it's also going to go and validate this. Like this would take like a solid 10, 20 minutes to go through this whole thing myself. But now I can kick it off, go and eat my lunch and come back and make sure everything's looking good.
And so the final thing I want to say is please go and try this command right now. Even if you already have a solid system for validation for your coding assistant, you might find something new here. I mean, personally, I've never had something as comprehensive as this. And it blew my mind what it was able to do and do consistently with the process that came out from this command. And like I said, I've made it general now so that you can use this on any codebase. So try it out. Let me know in the comments how it goes for you.
And so with that, if you appreciated this video and you're looking forward to more things on AI coding, more crazy fun stuff like this, I would really appreciate a like and a subscribe. And with that, I will see you in the next.