Transcription
Hey guys, Matt here for another WAN 2.1 deep dive, but this time with vase. And here are some examples I'm playing. If it looks a little jittery, it's because I'm recording and I got comfy loaded at the same time. So, it is actually pretty smooth. Most of these are 20 to 22 frames per second.
Here's one you might remember from my previous video trying to improve that with a reference. And I kept this one in because I want to remind myself that there's an error here in this guy's face. It should be a robot face. So, I want to show you how to fix that because we can't just use the complete universal settings. Even though I think I've done a pretty good job making something universal that you can kind of plug and play, but there are some tweaks we need to go through.
Thought that was pretty cool. There's me testing it out. That's my first real test. And again, I love trying to get movement. You see the rain or the fire? In this case, the girl's long hair. trying to match that to the reference image. And then we're using a combination, of course, of the control and the reference and and trying to get exactly what we want each time. So, let's go through it.
Here's a really cool challenge, of course, underwater. Here's the entire comfy workflow. I know it always seems intimidating when you open these things up and you think, "Oh god, where do I start?" I'm going to walk you through it. Let's just go into the model loading first because typically that's the first thing you have to do. I usually do a control enter right away and then in red it'll pop up which models I need to load and you'll have them probably all downloaded by now except for maybe the vase.
If you're running Windows 10 like I am and I have a 4090 by the way that's what people ask me. The Triton needs to be installed and there's separate videos about that. It can be a pain in the ass to get working. But if you have it installed and it's working, get the SAG attention. If it you don't have it installed you'll know right away you'll get an error. If you have Linux installed, then you know to attach your torch compile. If you want to check out my deep dive video on WAN, I explain what all the BF and FP8s and all that stuff mean.
In this case, you have to be sure to match the 14B with the 14B vase model. And then we're using the block swap here. And of course, we see a new one, the vase block swap. And that is going to also unload models from your VRAMm which does the actual computing and it's going to dump it some of it onto the CPU RAM and that'll allow you to produce higher resolution and more frames with the limited amount of VRAM that we all have on our consumer grade GPUs. The rest of the stuff you can leave. We can change this to FP8 if you want. Remember FP and BF-16 are larger in file size and therefore take up more VRAM than your FP8 which are the quantiz models. And in the previous video, if you remember, they didn't have an FP8 when I made it. And we had to use a quantization right here and select it to FP8. But because this is already an FP8 model, we don't need to add a quantization to it. It works just like as if you downgraded it to an FP8.
Okay, so that's what this all means. The Florence model is just something that I added. I'll show you that later in a bit. And I think the rest is pretty explanatory except you see that I've dragged the Laura way over here on purpose. And I'm leaving it here because that makes a big difference in the output, not the speed. This cause vid Laura has more to do with the reference image and how it's used than it does to speed up the rendering. You can't just add it in and get the results. So, this fine-tuning this number is part of my big experiment that went through and really going back and forth. And I'll just tell you since we're here and I don't want to forget the values that I recommend that you use for your Laura are 045 to 75.
Okay, I'm going to make a note. Probably will flash up when I do the editing. You use that and the higher the number, the more of the reference image it uses and the less it confines to the actual video dance. I'm saying dance because the last video I did here of Michael Jackson is a dancing video, but it more of the less of the movement. So the higher the Laura, the more of the reference image and the less of the control net movement. We want to balance all that out because ideally we put in the image that we want and the reference image and we get a perfect blend of both. But it doesn't work that way. There is slight adjustments but I'm going to go through all the ranges so that you run whatever you have right as this default. You get your result and then you make the adjustments that are required.
Let's talk about the two real main elements of this whole operation. That is your input video that we're going to use as the control net and we're going to get our movement from. And then of course you have reference image. And when I loaded the first example from Jai wasn't clear what direction are you adding video to the image or are you adding the image to the video and what I found the best results are that you are adding an image to your movement. So if you think about it that way this that'll help understand this sort of flow that I've laid out here. And what we're going to do is first just choose this image. And now it can be anything. You can see that I it doesn't even matter the aspect ratio. That's how this is set up. So, you don't even need to touch any of the settings here. The width and height, even though they're set high, you don't have to touch them. This will automatically be referenced without having to crop, remove the background, anything. Whatever's in this image, it's like IP adapter is going to be used with the video. And then if you also don't have any image and you just want to rely on your prompts, then simply connect empty image right here like this.
Okay, that's all you have to do. You'll see this one noodle is still left and that's going up to this whole Florence captioning idea that I have. So, we're going to ignore that for a minute, but I want to use the image. So, I'm going to connect the noodle there. And then this will just show you the final preview size just so you can confirm that this has been scaled properly. The next thing that's important is the source video. And that is going to really make a difference whether or not you're going to crash out on your memory.
Okay, that's always the thing that people seem to be really confused at is uh how do how do I get the longest video typically, right? And it's going to be a combination of resolution and frame count. Okay, that's the really the only thing that matters once you have all the ma the all the models configured and you're all using the quantiz and everything happens and you sort of set this up that you know will work. How do you squeeze it all out? Right? So then it comes down to yeah, you've got your block swaps all the way to the top. This goes to 15. For some reason in my head, I like to leave a little window and then have some of it absolutely 100 balance out onto the onto the VRAM. Whatever. Don't ask me why. So, there we have our input video. And the first thing we need to do is have it scaled to something that we know is going to work. So, whatever you put in, because I'm doing this is these are um some of these are 4K videos and uh 1080p immediately will crash everything. So around 1K is what I like to work for because that's got the right amount of definition. It's not quite 1280p, but it's definitely not potato resolution.
I'm going to hit generate, and you're going to see that the first thing is going to load the image. It's going to use one of these various control nets. Okay, we have the pose and the depth. Those work the best by far, and you want to use one or the other. However, what I found, you want the best of both worlds. And the reason for that is because if you use 100% depth, which you can, especially if you're wanting to do and replicate somebody's mouth and they're talking, but if you have something like a helmet or some slightly different facial feature, you don't want all of that depth because then your reference image is going to have to conform completely to that person's face. But if you use the pose estimator, then you are going to have just the eyes, the face, and the mouth. However, the mouth doesn't quite line up with exact pronunciation. It also doesn't really track a sword. So, when I was doing these sword images, you didn't have that sword because, of course, the pose isn't there. So, I didn't I didn't want the full structure and the limitations of depth, but I also didn't want the flex complete utter flexibility of pose. So, what I've did was blend these two. So, you can change this amount. And that's where this comes in where it does actually use both of these. It's really cool. It picks up unfortunately the people in the background, which I don't want, but whatever. the if you if you take away and isolate the image with a mask then the depth doesn't work very well. So whatever you can try that I actually have that not not to get sidetracked but I do have an opt optional isolate human. You plug that in before you do your depth calculation. Okay. So you can use that. It's way off to the side. And this will work pretty well with this. See how you like her? There's a length of her hair that I want. This actually this combination works well for most cases.
Now, with the control net, I like to take a little bit off off because we don't need 100% coherence to our control net. You can take off all the way up to 60% depending on your subject. And we do that especially with the architecture that I work on. And big part of my uh day job is working on these Google imagery uh luras that I've made and trained and they convert Google images into realistic photos. So that has knowing these settings is something I spent a lot of time with. So it kind of works here as well. And that basically with combined with your Loris, your LoRa, the Cosvid Laura, these two values work really well together.
So something might start glitching out because I'm recording. And I I should mention just now when I saw it stutter, hopefully I didn't lose some of that recording, but this is what I check on. Okay, you don't want that to go past 23 GB. You you need to save a gigabyte of your VRAM, whatever it may be.
All right, I had to stop the video. We're kind of stuck with Michael Jackson, Darth Vader thing. You'll notice here with the sampler that there's only four steps, and that's the power of that Laura Cosvid. Five doesn't really make much of a difference. And you can do tests as low as two, but four is definitely the sweet spot. If you want to do a test, then I recommend you simply come over here to the frame load cap and just put that down at a lower frame. This forced rate makes a huge difference. Now, if we zero this out by hitting that, you'll see that the total frame count here will be 144. If I change the frame rate to 20, that's going to go up. Now, if you have a talking mouth and a lot of detail, you want to go as high as 24. And that's going to obviously bring up the frame count to 210. Now you may have to lower the resolution but generally once you have this resolution as a known value that you can work with then the rest really comes down to actually does nothing to do with the resolution of your image.
There's the resizing here but that's nothing more than there's no reason to be working with anything larger than our control net videos. So and our final output so I've resized it. This really is the whole thing is that you adjust this to something that you know because once you have these sizes, I was producing all these tests and simply just not going beyond 170 for safety and as low as 30 to do it a sample. And then you can also bring this out to, you know, the middle if you want to see where a certain part of the dancing is. You just do the first skip frames and, you know, do 30 for a test. And then also you're going to free up VRAM. So if you wanted to, you can lower that. But for the most part, when I'm doing a test, the steps are so fast, two to four steps. Really, I was generating something within less than a minute in some of these samples. And you do that because one of the problems will be the balance between your control net and your reference image. And now your reference image makes a big difference to how it fits within your uh your this this video.
The really interesting thing when I was trying to get this dancing, I thought, well, wouldn't it be great to have another character, a male character, and so I was experimenting with uh the underwater scene. So if I use this image, you think, okay, well, it'll they'll separate the two. It doesn't really work like that. She just ends up starting to dance and he's just moving around like this. And the sort of context and the the other environment doesn't quite fit in with the other objects. The the the way vase model seems to just try to shoehorn the other objects that are in the scene that don't have movement applied to them with the control net and it sort of just figures it out. And that's why fire was a big challenge. You see some of the background fire in those examples. They don't flicker way out in the back as much as you would hope. They become almost like a a static backdrop. And in this case, you don't want this woman with the the guy in it. You just got to take your roll of the dice how the other characters are going to look. And that happened with some of these other videos. Um, sometimes if it was the person was too far away the and and this was a close-up then a duplic this becomes a static shot and then the person's face sort of in the photo becomes a kind of a jump up right at the beginning and that's what you end up seeing and then it it fades away. So there's all this weird stuff. The idea would be that you copy your video and your static first shot as accurate as possible. So this guy is shooting a bow and here I have similar perspective and I made these with flux and you don't really have to match it that well but this is definitely the idea that you would have very comparable aspect ratios and reference images like the bow and the arrow or the appearance of the sword. Those things will play into the final generation.
I must get out of here. I must get free. All right, let's talk about prompting. We're going to come up here. You notice that I dragged up the reference image. I'm not overly interested in capturing the appearance of my reference video. I really want to just I want to capture and have this conveyed as, you know, in this case, you know, a Darth Vader samurai. The only thing that's added to this is dancing and that there's movement and so Wayne can understand that. So I do have this combination where I thought okay well we could do both. We could combine this why this is uh blanked out is that we could combine both the video and the image and then mix them case which I really like is capturing the reference image you know somebody dancing and this shows you what the image is because it doesn't pop up here. So that's why this box is here. This is just a base prompt and then I link it together like that. So this is how you combine these two where you do one and then if you want complete manual you just extract that and then you write in. So you don't have to use this complex system that I have here but you know it's there if you want to use it.
Let's go back to the WAN sampler. You notice I already mentioned four steps is the ideal. Uh two three four five doesn't seem to make a difference. Now the CFG and the shift with vase and the cos vid I thought okay for sure if you have all this huge movement that's when you increase the shift but with the control net you really you get less results and the only difference was that it added a finger in the archer when I crank this shift up past uh two so one to two and two is just where I'm landing and then the cfg does matter But because we're relying so much on the covid prompt and that whole Laura feature, then you don't want cfg is recommended to be at one. However, I do want some slight importance placed on my prompt because I'm using a pretty solid system here. Then after that, everything is the same. The schedulers, you can try them. I I just uni unique PC seems to produce the best results. You should always match, especially if you're using music or sound, have to match your frame rate to your frame rate in your video. So, that was 18. I'm going to choose 18 again. And then we have this full optional. Just delete that if you're not interested in showing anybody your comparisons. But that's just the whole comparison thing right there.
Oh, a quick note on the SLG. A little bit experimenting with this, trying it out. it. I don't whatever it's doing, it's very subtle and I found the experimental arguments uh were just slowing it down. I didn't notice anything. I must be missing it. Please tell me how to use this in the comments. I'd be happy to to figure it out more, but the results were the same and it was just taking longer. So, I've left it in there, but I'm bypassing it.
Okay, I think we're almost done. However, I kept that mistake in there at the beginning because when you have a robot face or the Darth Vader where it doesn't have any human features, you have to toggle off face. You have to put disable or it will fa it will force a face onto your masked robot. Let me show you. Yeah. So, it forces it forced my Darth Vader and it just took his mask off completely because it needed it needed the face. Um, that's ridiculous, but that happened all the time and it would somehow also like kind of do a weird gross hybrid. So, toggle off the mask, especially if you wanted something like this if you wanted to have the that one worked because again, you just toggle off the the face. So, I'm going to leave that enabled, but that was important.
Anyways, leave me a comment about what you want me to do next because I have no idea what the audience is out there for this. I mostly do architecture and so this is a lot of fun. Anyways guys, thanks for watching. Peace.