Transcription
So, I still see a lot of people out there not using large language models like Gemini 2.5 Pro here, uh, not optimally. So, today I just wanted to go to a workflow that is working superb for me. Uh, it might be a bit different than you're used to, but I suggest giving this a shot. So, the app we can make today is an AI video-to-video app. Basically, we're just going to keep it quite simple. So, we're going to upload a video. Uh, I say like max 10 seconds, 8 seconds. And from that video, we're going to let's grab a last frame using ffmpeg. We're going to enter a prompt, and we're going to include that last frame as an image reference. From that, we're going to generate an AI video using cling AI. I really like that model, but you can pick like Stable Diffusion Google V2, or something, too. Uh, then we're going to merge the uploaded video with the AI video to make it like a smooth transition. And then we're going to return the new video that is kind of combining our uploaded video with our AI video in one video, right? Uh, we might also add some music on top of that. We will see. But that is basically my idea here.
Now, initially, uh, we're going to run this on Gemini 2.5 Pro. Best model ever made if you ask me. So, I'm just going to go through kind of the workflow I use for this because, uh, I think a lot of people can actually learn something from this. So, the first thing I always do is I like to spend like 5-10 minutes preparing the project. So, what I always do is, uh, let's say our app, we know we need uh some information about the cling AI. We might need some text generation. So, we can use Gemini AI for that. Maybe we need ffmpeg, but I think Gemini 2.5 knows ffmpeg. If we're going to use some video, we might need some documentation around that API. So, I always start by gathering documentation. And, uh, I just fire up Cursor. I create a file docs. Uh, here I have a cling AI.md. So, we're going to fill out some documentation about this. So, I'm just going to head over to Replicate where they host this cling AI model. Here we can generate uh up to 10 seconds with an input image. That is perfect. So, what I'm going to do now is I'm just going to go down here and I'm going to find uh they have some documentation here. I'm going to find that. I'm just going to click on the API, right? And here we have documentation. I'm going to do Python. So, I'm just going to start by grabbing this from "Get started" like this. And I'm just going to put that unstructured into here. It doesn't matter too much because these LLMs are very good at unstructured data. Uh, we can grab uh "Learn more" here. So, basically, we try to collect as much information as possible right into here. We have some parameters, we need schemas, input schemas, and yeah, I think that's basically enough for now. So, if we need any more, we can go look that up. Um, for music, I might want to use Sonato. So, I'm just going to go to the API and I'm going to find the documentation. Right, here's our documentation. So, I just grabbed this, put it into a music.gen. I also went to Gemini, into their documentation and over to text generation. I just grabbed this and put everything into an MD file here. So, now we have some, at least some starting documentation. We can feed our model here.
So, what I like to do now is I just reveal this in my finder. I just go to Gemini and I put these three files into my context window. And that will actually—it might seem a bit wasteful to spend time on preparing the project. But I think you will have a return on investment by doing this, by uh minimizing errors and debugging you have to do if you gather enough documentation to actually fulfill this. Uh, but of course, there are some documentation you don't have. So, the ffmpeg here is that we're going to rely on the Gemini 2.5 model to fill out kind of what we don't know how our app will work. Right. The next step now is, of course, going to do our prompt. Uh, so I also like to spend a bit of time on setting up at least the initial prompt. So, let's just go write that, and we can go through that together and then send this. So, here's the prompt I wrote for our app: "I want to create a simple video-to-video app today. Here are the feature requirements. The user is going to upload an MP4, 5-max 10 seconds, in `video/mp4` video. We're going to store it there. We use ffmpeg to grab the last image frame from the video because we need that in the input. Store this temporarily in `images`. Use the—the user prompts the cling AI model from the Replicate API and adds the last image frame to generate a new AI video with an 8-second duration. That's fine. We could do 10 seconds too here. Uh, we can do yeah, 10, that's fine. The generated AI video is stored in `videos`. Then we're going to use the Sonato API to generate some music tracks with the parameters `instrumental` set to `true`. And we have some tags: `uplifting`, `meditative`, `chill`. The music track is stored in directory `music`. Finally, then we're going to merge the uploaded video and the generated video and the music into a new video and store it in the directory `videos`. Write the code to create this app." So, let's just crop this now. And we can go back to Gemini. Now, we're just going to feed in this. Uh, I think we also want to add here, uh, "My API keys are stored in `env.py`." Okay, so I think that's fine. Now, we have the context, we have our prompt. So, let's just run this, and hopefully we'll get some Python code here, and we can build out our front end for this uh afterwards.
Okay, so we have a response from Gemini 2.5 here. We have a setup. So, we're going to create a folder. That's fine. Uh, but let's do this. So, we need some subdirectories, `env.py`. I think we're just going to do this. The other thing we can do ourselves. So, I'm going to just use Cursor for this. Just select the agent Claude, and let's do send. So, hopefully this creates some subdirectories for us here. Uh, `images`, `music`, and `videos`. Perfect. We got that. And we're going to create an `env.py` and an `app.py`. Was it that `app.py`? `.py` like uh this. Okay. And then we're going to install some dependencies for our setup. So, just bring up our terminal. Okay. So, I put my dependencies in `requirements.txt`. We can just install this. I pretty much had this anyway. Uh, let's continue down here. Uh, we're going to populate our `env.py` file. Let's do that. So, let's just populate this by this. Okay. So, we need a Replicate token. So, `token`. We can fix that afterwards. Let's continue down here. So, here's our code for Python, `app.py`. So, I'm just going to grab this `app.py`. Okay. So, I'm just going to go grab the API keys we need now, and then we're going to see how this works, and we're going to look at how to run this. So, we're going to place a short video in the `videos` directory. Uh, let's just run through this. We need to prompt it. So, it's going to be interesting to see if this works on the oneshot or if we have to do any adjustments here. Okay. Okay. So, let's do `python app.py`. Uh, enter the video name file. Yeah, that's going to be `cliff.mp4`, right? Okay. So, we got the last frame here. Perfect. Yeah, that's correct. Enter the prompt. "Guy jumps off the cliff into the sea." Let's try that. Okay. So, now we are running the video generation with the prompt: "Guy jumps off cliff" using the last frame. Perfect. This looks good. So, what I'm looking forward to seeing now is that after we generate the video, we need to create the music, and then we need to merge everything together. So, that's going to be interesting to see if that works. Now, we can just create our front end, and our app is pretty much ready to go without any big issues here.
Okay. So, you can see now our video is ready. Now we are actually sending some task to Sonato to create our music. We are generating that. So, when that is complete, finally, hopefully we can just merge everything together. Okay. And let's see if it works. Okay. So, you can see we got some merging failure here. Uh, we need to fix that. So, I'm just going to head back to Gemini. Let's paste in our error. Let's uh fix this right because uh we are quite dependent on the—we got the video, we can look at that, that's fine, but we really want the music to be merged, but let's watch the video now without this. So, this is me now, of course, and now here the AI video comes. Okay, that was a bit unrealistic, but pretty cool though. Uh, so you can kind of see it was a very nice transition here between my video and the uploaded video. So, that is the idea behind this. Uh, so let's check Gemini here now. Okay. So, the error log is helpful here. The problem. So, let's find the solution. Okay. Here's the solution. So, let's just uh use Cursor to fix this. Uh, we can just paste in this. And let's do "Update my code with uh this," something like this. So, I like to—sometimes it's faster just use Cursor to update the—the additional debugging from Gemini because, of course, you can go in yourself and fix it, but uh I think this is very helpful to just do it like this. So, hopefully now with this fix we can actually merge this together. Okay, perfect. So, now we have completed the merging. Great. We have a video here called `final`. So, this is probably going to be the music and the video. Uh, so let me just set uh some audio on [Applause] [Music] this. Okay. That was not very good, was it? Uh, something was a bit off there. No, I just didn't see it. I thought the clip started before that we end in the air here. Uh, but yeah, it's working right. So, the app is working now. We have every feature we wanted. Uh, so now I think we need to move into the—the front end part because I want a simple front end where we can upload a video, do the prompt, and then uh send everything and get uh maybe the URL or displaying the video that is generated back. So, again, I'm going to head back to Gemini and do some prompting. So, again, I'm just going to drag my `app.py` file here just as context so we have everything up to date. And let's do a simple front-end prompt here to get uh a simple front end. So, I'm just going to keep it simple. "My app is now working. Now I need a front end where the user can upload the video, enter the prompt, get the merged final output uh to be playable in the browser. I just put in `localhost:3000`." Generated front end for my app. So, now we're just going to run this, and hopefully we'll get some simple instructions how to create a simple front end. And this is something you can expand upon later if you wanted to do this very well. But for now, we're going to keep it simple and see how this turns out.
Okay. So, now we got like a new structure for our project. So, I just went ahead. I created exactly this structure here. Good. Now, we're just going to keep following the instructions. We need to install Flask, right? That is very easy. Let's do this. Okay. And then we need to update my dependencies. Yeah, I'm going to add Flask. Okay. Further down, we need to refactor uh our logic. So, we're creating `video_processing.py`. So, I'm just going to grab this and I'm going to fill out every single thing we need now. And then we're going to try this to run this as a front-end app instead of just running this uh in our terminal. Okay. So, how to run this now? We're just going to run the Flask app: `python flask_app.py`. We're going to go to `localhost:3000`. Okay. So, let's go here. We have a simple thing here. So, we can upload our MP4 file. Let's try that. Okay. So, I find my `clip.mp4` file here. Perfect. That looks good. Let's do the same prompt: "Guy jumps from cliff into sea below." Okay. Generate video. And now this is running. We can look at the back end. Now we have started. Perfect. So, now we're just going to let this run and hopefully when we get back we can actually play the video in this format here. So, I'm just going to give this some time, and we come back when we hopefully have a result here. Okay, processing complete. Here is our video. Uh, let's try to play this now. Perfect. Okay, so that's our app. Right now, we generated a front end for this. We can go back, generate another video; we can start over again.
So, you can see by following our workflow today in Gemini Pro, right? This turned out pretty good, and I think uh we didn't get a lot uh into a lot of issues because we provided all the context we needed, and we kind of set up our workflow to yeah, just uh feed all information we had that we knew about what we wanted to create. We did not tell uh anything about the ffmpeg part, and that is where we ran into a few issues, but where we provided relevant context, documentation, we had no issues at all. So, I would say the only thing we struggled a bit with was some of the merging of the videos. Other than that, this went super smooth. I didn't spend a lot of time, and now I kind of have the option to work more on the front end. Maybe I could add some music prompts here. We can display the video, maybe the extracted images, but for simplicity, we're just going to keep it like this because I just wanted to show you kind of how this workflow can reduce your error rate by providing, like I said, relevant context. And I think this is just a nice way to get used to working with LLMs, uh, if you have a project that requires this. So, yeah, hope you enjoy this. Uh, hopefully you learned something. I'm probably going to do other videos on these types of workflows going forward, so look out for those.