Transcription
So, someone joined my Discord and was asking me for help. Basically, he is trying to vibe code an application and he's noticing that as the application gets larger, every time he prompts, there's a higher chance that it breaks existing functionality and he pushes that code to production and his users are starting to complain saying that stuff that used to work is starting to break now.
And I think this is becoming less and less as the models get better, but there are some strategies you can do when it comes to agent coding and vibe coding to make sure that when you add a new code, the existing code doesn't break. Now, the main way I would recommend achieving this is by using unit testing or end-to-end testing or integration testing.
Now, if you're a vibe coder, you have no idea what that stuff means, I'm going to try to walk you through how to get it set up. Um I'm going to be using Cursor and we're going to be trying to add one end-to-end test using Playwright over this Video Crafter application that I've been kind of playing around with when I get some free time. So, for this application, the most critical thing that I need to test is a user should be able to log in, they should be able to go to their dashboard, they should be able to create a new video, and then they should be able to basically view that video after it's done. Okay?
Now, there is like a progress screen that kind of shows you stuff is doing stuff behind the scenes, but the happy path, when I say happy path, I mean like what is the everything works fine, there's no issues hitting Fal AI to generate the videos, there's no issues hitting 11 Labs for the audio, there's no issues with credits. The happy path is everything's perfect. Can you write a test that clicks end-to-end verify a user can at least generate a video because that's probably going to be like 95% or 98% of all interactions with my app assuming everything is set up properly and there's no like downstream services that are broken.
So, I'm going to load up Cursor over here and we have this application selected. So, the first thing I'm going to do is I'm going to switch it to a more powerful model. I think Opus 4.7 could probably do this really well. Also, GPT 5.4 could probably do this well. I would probably recommend using one of these over something like Composer 2 because this is a more complex thing. Writing end-to-end tests is a little nuanced, right?
So, let's go ahead and prompt this out. There's a couple of things that I want to mention, so I'm going to be going back and forth to really refine this prompt. I want you to write an end-to-end test using Playwright. This test should log in with a user. Assume that the user already has a bunch of credits, let's say 5,000 credits in their account, and they should be able to go and click on a new video button, which would navigate them to the video generation page. Then they should type in a prompt that says an educational video of how Play-Doh is made. They should keep all the defaults of all the other options and they should just click on the generate button. And then we should verify that a progress page shows up. And then when that goes away, we should verify that we have a video details route where they can actually view a video and they can also see links to download the video assets. Okay?
Now, this is a little bit more complicated because this application does a lot of stuff. We we integrate with 11 Labs, which is doing the voice. We integrate with Fal AI, which is doing the text to video. We integrate with OpenAI for generating the scripts and the descriptions for every single video. So, there's a lot of third-party integrations. And unfortunately, in order to write test, you don't want your test to burn through $4 of Grok credits every time it generates a video, right? So, you do need to learn a little bit about mocking or having some type of context that you can switch out when you're doing tests just to try to emulate everything behind the scenes um so that you're at least saving some money.
Now, there are other ways you can do this. Like I could probably say, "Okay, let's just use a cheaper model like Wend 2.2, which I think will cost me like, I don't know, 40 cents or 30 cents for a video." But that still adds up because eventually, you want to automate all this stuff. So, here are some additional things I'm going to add in. This might be a little bit more complex, but you may need this as well. I want to make sure that when these tests run and when you load up the service, I want you to turn it into a mock mode so that every third-party service such as 11 Labs, OpenAI, and Fal.AI is all mocked out, meaning that we can use a preset set of fixtures, which I do believe is in my fixtures folder, to just pull stuff over um to emulate like it's being finished by these services instead of actually invoking them. Find an elegant way to decouple my code from the actual running working implementation for contacting these third-party services that we can easily switch with an environment variable so then it then points to using a stub or a mock, which just returns some uh fake data. There should be enough example assets in my output directory where you can pull in the voice to text file or the .ass file or whatever to get this uh working. So, feel free to copy those assets from one directory to another if it helps this test be more reliable and rerun-able in the future.
Will you be able to prompt like this if you're vibe coding? Absolutely not. I don't think you're going to be able to prompt as detailed as I am. Okay? But the idea the the the key parts is you wrote this happy path test that kind of describes what it needs to do. You specified I want end-to-end with Playwright. And then I also specified mocking. You could probably reduce this into a simpler prompt and just trust that Opus can kind of get it done for you. Um but we're actually going to turn this into a plan mode because there could be something that I forgot. Even with this large prompt, it seems like it's very specific and detailed, there's probably one or two things that I forgot and using plan mode is just very useful for like catching any edge cases that you totally forgot about.
Now, I will say I already have an AI video mock fixtures flag in my code base. I already kind of put in some groundwork to make stuff mockable so that I can actually run this locally without being charged a single dime or a single penny for anything. Um but we'll see if I can kind of extend that because I haven't written an end-to-end test yet. So, I need to kind of let this run and it might take a while. This is going to be a very long prompt and Opus 4.7 seems like it's a little bit slower than the other ones because the high model uses more tokens and it thinks a little bit harder. Um so, I'm just going to cut the video and we'll come back when it's done.
All right. So, like I mentioned, I thought it was going to ask me some questions and here we have two questions that we kind of need to go through and answer. So, the first one is, "How should storage be handled in the end-to-end test? The app currently only has Cloudflare R2 adapter in the video details page downloads and plays via R2 pre-signed URLs." Yeah. So, in order to get this mocking stuff working, I do need to figure out a way to have all this stuff be able to run locally without using R2. R2 is yet another third-party service I'm using to store the assets. Some options are to use a local file storage adapter and then we can enable this automatically with AI video mock fixtures true. So, this would basically store the files in some type of directory locally, right?
Now, the other option is to actually keep R2 and it really depends on how thorough you want your end-to-end test. Technically, an end-to-end test should be a fully running application, everything is integrated. So, I'm kind of doing something a little bit beneath that and the tradeoffs are my tests will run faster and cheaper and more reliably, but it's not fully testing the system, right? So, this is more of like an integration test in a sense cuz I'm not fully integrating with everything, but I'm still going to call it an end-to-end test because typically, it's clicking through the UI and it's integrating with the back end. Anyway, so I think we can just go ahead and just use the file storage. I think it's fine.
And then we're going to say, "How should the test user's 5,000 credits be provisioned? Uh Playwright global setup call better off sign up for a dedicated end-to-end user. If missing, then update the user credits to 5,000." Now, what I've seen is typically, the first response is usually like the recommended one. Usually the best, but it's not always the best. So, you do want to read through these and make sure that you understand all of them. But if you're not a coder, just A. Just click A. Whatever. Hopefully, it works. Um but you could also probably prompt the AI and tell them that you have no idea how to code. And I think the outputs that it gives you might be a little bit more digestible for your skill level. So, keep that in mind that you can actually prompt the LM to give you more um you know, explain to me like I'm a 5-year-old cuz I don't know how to code, right?
Okay, this is strange. I've never seen a plan come back with literally just a sentence about context and five to-dos. So, I'm actually going to reprompt this. The plan looks very uncompleted. Can you double-check the plan has been fully thought out? I don't know what Opus did, uh but it it this plan is I've never seen this before for a plan.
Now, even this approach of like just asking it to like double-check the plan, you can do this with different models, too. Like sometimes I'll even switch this to uh GPT 5.4 and say, "Hey, verify the plan to make sure I didn't miss anything." Because this is a larger thing, uh it's a larger addition to the code base. It's a little bit more complex and you want to make sure that the plan is thorough from the get-go, otherwise you're just going to be coming back and like it's going to keep on trying to run your tests. It's going to fail. It's going to try to fix the test. It's going to rethink what it's doing. It just takes a lot more time, a lot more tokens. So, like having a good plan up front, I think is very valuable in the long run of like doing agent coding.
All right. Now, you can actually see it like adding real detail to this. So, like hopefully, this will be good enough. All right. So, this is what a plan should look like. I mean, it should be filled out. You should have diagrams. It should be very detailed of like what it's doing and what it's planning to do. So, just keep an eye on what's happening in AI because sometimes it does it stops early and then you have to go and you have to make sure that it actually works.
So, I'm going to run this. When it's done, I think it's Is it still running? Now, I would recommend reading through the very last paragraph. Every time you try to generate a plan or run something, usually the very last paragraph, it's important. It has some key information. It's going to ask you, "Hey, I skipped X, Y, and Z. Do you want me to also do those?" And if you don't read those, you think everything's done, but really there's more that you could have had it do for you, right?
So, this one's saying, "Do you want me to adjust anything like uh additional assertions like credit decrement verification?" I didn't think about that. We do have a credit system when they try to generate a video, it needs to decrement their credits. So, after they generate the video, I want to make sure that the credits at least went down. You don't have to be super thorough. If you wanted to, you could be like, "I need the, you know, the exact amount of credits to be decremented." Um but at the very least, I do want to verify that it did go down. So, I'll say, "Yes, add additional assertions like decrement uh verification."
Now, another thing I'll point out is maybe you don't have the technical skill to prompt like I did and to improve the prompt and to double check everything. I think there are skills out there that are like there's probably a playwright skill that you can install. So, go learn about Claude skills. They can be used for a cursor, they can be used for Claude code and various other agents. You should probably bring in some skills that help you write best practices when it comes to testing. And also a skill that just reminds your your agent to add end-to-end tests. Um but I will say that some of the stuff like I added like the mocking I would assume that the you know Claude code or Opus might not have automatically thought about mocking stuff out when you run it locally. And these are things you have to kind of think through like okay, if I were to run this test every single day, I'm going to be burning through $4 every time I run it, right? Right?
So, I don't know. I think I must execute there is a little risk they noted credit math is coupled to both the full duration and mock audio length. All right, at this point let's just build. And then honestly, this may take you know 20 30 minutes to actually get through all this because it's going to try to run the test at the end. It's probably going to have some bugs. It's going to have to fix those. And look it just added more to do. So, like it's expanding the scope of what it's trying to add in as it's doing it. So, I'll come back in a bit and we will see how this works.
All right, so it looks like the plan is done. I would highly recommend at least trying this. It says that it verified this stuff. So, I mean you could try NPM test to verify that it does run the unit test, okay. But then also we should probably run the end-to-end tests and it uses end-to-end port 3100. I wonder if I need to say end-to-end port 3100 and then run this. Okay, and that does pass. So, that's good.
I also want to add in a way I'm going to I'm going to switch this to composer two so that I don't have to wait around. Add in a way for me to run the end-to-end test with the playwright um visual runner. I believe it's just like a hyphen hyphen argument like an argument called like open or something, but I just want to be able to visualize and show you in this video what exactly is happening because just watching something print out in the terminal is not enough verification from a professional standpoint of like is this test actually doing what we think it does. And I I do think you should get out of this mindset of like letting the AI do everything because it's not perfect and if you don't visually manually check things, not necessarily checking the code, but like testing your UI, testing that your tests are good, you're not going to have um as good as outputs as you think you would.
So, I'm going to run this command that they gave us now. So, we have two new commands we can run. And this is the playwright tester. So, over here you have a list of all your test files. You can click on them and you can click play and that's going to run through all of the test scenarios and kind of drive a browser over here to verify that. Can your page load? Can it click buttons? Does it go to the right uh routes after things happen? And when it's done running, you can actually click into this. So, you should be able to um Okay, I can't right now. So, there is a little bug where after I ran it the second time, the credits got decremented, but then the assert statements were like no longer valid. So, we will have to go and fix this. This is something that you're always going to see like there's edge cases that you didn't prompt, Claude didn't figure out and you ran into them.
So, I'm going to say I noticed when I run the end-to-end tests for a second time, the credits actually started at a 4,990. I think you need to make the test more reliable so that the credits always start at a baseline. So, you should take the user and like force their credits to be 5,000 at the start of every test. Okay, so I want to make sure that it's a repeatable test and it's going to make sure that we have enough credits at the very start. So, you could just use composer two for this one. I think it can figure it out. Um it's probably just going to like do a database call to force credits on that user. But then we should be able to run this over and over again and not run into issues like that.
And the way this is typically achieved with code is that there's a test up before each and it runs a command. In this case it's resetting all the end-to-end accounts to be the baseline. Um this is very important when it comes to testing like you want to have a repeatable baseline that everything starts at. Otherwise, you'll have flaky tests, tests that run and fail on the third or second run like we just saw. Um so, yeah, definitely having some type of scenario set up like that.
So, after running this, let's go through and just look. If you could just hover your mouse over this, you'll see that it's kind of logging in with the playwright account. I know it's very small to see. Let me see if I can zoom in a little bit you guys can see it. But um it logs in with the playwright account that it creates and then it has 5,000 credits and then it clicks on new video at the top right and then it enters in a prompt, an educational video of how to make Play-Doh. It clicks generate. It note it verifies that the credits have been decremented. Um so, it went down 4940. And then we expect the video detail page to show up. So, kind of just waits here and then expects this to show up. And then at some point, I'm going to keep scrolling. I noticed that the history no longer shows that video. So, I'm going to add into that test. I want to make sure that if we were to hard refresh this UI, that Play-Doh, an educational video about how to how Play-Doh was made should show up. So, I'm actually going to go over here and copy this. And we're going to go back. I noticed that when the test is done, the very last couple of steps the history doesn't seem to show that video we just generated. Can you make sure in the test assert that when the page is refreshed the history of recent runs still shows up and we should see that Play-Doh title show up on the left navigation list. Okay, so that added in a little assert statement at the very end to make sure that hey, you can refresh and go back to the dashboard and it should show up on the dashboard.
So, let's run this. I want to make sure this actually works. Okay, another issue we're running into looks like it's expecting this to say complete, but it says uh pending. So, we're going to go ahead and just add that back in. It looks like it tried to run the test and it's failing and I did verify that. So, I went over here and I wrote ran the test. It's failing on this edge case. So, I think it's trying to go back through and fix the test to make it more reliable. All right, it says the test passed now. It did a lot of different stuff to try to like make this work. Um I would probably do one more pass through with another model to verify it's like best practices. It looks like that worked. Now that the video shows up in the side navigation, it looks like we're properly waiting for it.
Now, I will say that it didn't show after generating, we should expect like a status page. So, that's something that's kind of missing as well. Um I'm going to try to prompt that as well. There's one thing that seems like we're not checking. After we click generate video, we should see a status page that shows like the progress and accordions and stuff. Can you verify that we didn't mock that stuff out? It should still navigate the user to the status page and if you need to mock out a three or five second type of timer for one of these mocked services so that we can at least verify that the status page shows during these tests so we can assert that.
Now, I will probably stop using composer two at this point. I don't know why I'm using composer two. I think this test is going to get I'm going to stop this. I'm going to switch it back to Opus. I think composer two is great for smaller stuff, but this context window is huge now. There's a lot of stuff we added in. This test is getting complicated. It has to really understand the entire code base in the happy path. So, again, that's when I switch off the the weaker, dumber models to a more intelligent model that's going to take longer and it's going to think a little harder.
All right, that's done. Let's test this out again. I'm going to run this. I want to actually verify that it shows that status page for at least a little bit. There it is. There's the status page right there generating video segments and it's done. And then um yeah, everything's good. So, at this point there's more. There's more. We're going to we're going to do a little bit more.
So, I'm going to go ahead and make a new context window. And I want to say, can you make a GitHub action which runs these end-to-end tests on a pull request? Okay. Simple uh simple as that. So, let's just run this. So, basically when your project gets larger if you don't want to like manually run these tests all the time because A, you're limited to typically your local computer can only run like maybe one or two of these at a time. Like you're going to run into port collisions. They're all going to try to share the same database. And so, setting up the infrastructure locally to be able to run your tests on different work trees, like if you're doing different work tree development it's just complicated. So, it's better to have these tests run in a pull request so that A, it's not running on your local machine using up your resources. It's running somewhere else. You can come back later and check it to see if it passed or not. Okay.
So, GitHub action is a way to uh allow you to kind of expand your work and make it a little a little bit more organized so that you can work on various different stories on different work tree at the same time. You can make a pull request, separate branches. You can have those branches and that code be reviewed by more AI agents in GitHub. Like there's GitHub Copilot you can use. There's uh bug finder which runs in cursor uh on your pull request. There's Claude plugins or Gemini plugins. Basically, every time you make a pull request, there's more code that gets reviewed by AI which is great. But the important part is that your end-to-end test will run when you commit your code.
So, I'm going to go ahead and say check out a new branch commit, push and make a pull request. Okay, I think there's actually a way to go over here uh commit and create PR. So, I create branch, commit, and push. Yeah, I'm going to do commit and create a PR. I think at this point everything's good. We have a test that works locally when I run it. We have a GitHub action that should hopefully run. And now we can just test it when it makes the pull request to my my repository.
All right, so let's go and open up the pull request over here and we should see okay, we have our repo AI Video Crafter. And we have a pull request. In that pull request, you'll notice at the bottom it has some checks. These checks will run automatically when you create a pull request. And if you click into it, you'll see that it's running our end-to-end test or it will. It's right now it's grabbing a Docker container. It has to set that up. It has to run NPM install. It does take a little bit of time. But uh hopefully this all runs successfully, it'll get to the playwright section, it'll run the test, and if they all pass then we can have multiple agents creating these branches, making pull requests, and we can have, you know, these things being checked automatically. And this gives you certainty that you're not breaking your code in the long run, right? This is the original issue I made this video for. Someone's doing vibe coding, a bunch of stuff keeps breaking all the time when he tries to add features. Having tests to validate your code does exactly what it's supposed to do, so that when you add changes, these things will will turn red saying, "Hey, you broke something, right?"
So, now at this point, it's failing. And we have to figure out why it's failing. So, I think it's probably related to, I don't know, missing a a database or something. You don't have to worry about debugging it if you're vibe coding. You can just grab the pull request URL and then go back and say, "The PR is failing to run the tests. Debug why." And then paste that in. And it's going to go and just use the GitHub CLI if you have it installed, it's going to grab all of the the failing tasks the failing GitHub actions. It's going to look at the logs of those. It's going to start debugging it. It probably needs to modify, like, I don't know, the the GitHub action to have more, like, spin up a database or something. Um it looks like it's failing to do a CI because of a conflict with OpenAI and Zod. So, we'll see if that'll kind of fix it. And it's automatically going to push it, right? It's It's pushing it. And we should be able to go back and notice that it did push a little fix there.
Now, you get the gist. At eventually, this will pass. It's just a matter of, like, okay, keep doing that, get it to rerun and figure out what's going on. Eventually, when this all turns green, you can just merge your pull request and that'll merge it to main. But, then you have to remember all of your code editors, you'll have to pull that. I will use AI to just pull the latest changes, and if there's merge conflicts, you can just have the AI fix the conflicts as well. Um but, that's the overall process. The the key is tests and end-to-end tests. Try to get the application in a state that is as close as possible to production, but also with the tradeoffs that I mentioned, so it runs faster and cheaper. And you have these things run automatically.
All right. If you guys like this overview, I know it's very long video, but this is actually like legit software engineering what we're doing, uh you know, day-to-day when we're doing real engineering work. Hope you guys found it useful and leave a comment if you have another video idea I should make. Have a good day and happy prompting.