Transcription
Hello everyone and welcome to lecture 4 of CM296. So before we start, I just want to say that those first three lectures were really the most technical and mathematically challenging. So in that sense, I'm still glad you're here and I really hope that those three lectures gave you the intuition as to why we were going to use this generation paradigm because they're going to be used as building blocks for everything that we're going to see from now on.
In particular, during the first three lectures, we focused on unconditional generation. And now what we're going to see is how we can generate images with a condition which can be a user prompt like input text or an input image. And so in that sense the title of today's lecture is multimodel guided generation.
With that we're going to start and as usual I'm going to just recap what we did the previous times. So as I mentioned the first three lectures were mathematically challenging and they were about seeing how we could generate images. what were the paradigms that we could use. So the first lecture was focused on the diffusion paradigm with DDPM where we saw that we were taking clean images that we were noising in a discrete way. And so here we had this formula which was a combination of a clean image and gausian noise that we waited with uh those coefficients. And what we saw with a bunch of math was that we could derive an expression that we could optimize in order to reconstruct those clean images that we we've noised. And we ended up with a very nice expression, an L2 loss on the noise that was added. And in particular, if we wanted to d noiseise, we had the following formula that allowed you to go from a step t to a step t minus one using this uh model e theta which was the predicted noise that was added. And so in that sense it was parameterized with respect to the noise epsilon.
Then in lecture two what we did was that we derived the continuous version of what we derived in a discrete way in lecture one and we also saw a second paradigm a second way of seeing things which was score matching and we saw that when we looked at these two way two ways of doing things. If we were to take the increment of time from t to t + one and tend it to zero, we had a general expression of the increment in x which would be as a function of dt the increment in time and dw which is the continuous version of adding noise. And in particular you had these two terms. So on the left side the um the drift term and then on the right side was the diffusion term and we saw that in order to reverse the noise that was added. So aka reverse process. We could use a result from theories on stoastic differential differential equations where we saw that we could derive the reverse process if we knew the score which is why this formulation was parameterized as a function of the score. So here is noted s of theta.
And in lecture three we saw that actually we could think of this whole thing as a completely in a completely different way. And we can think of it as transporting a probability mass from an initial distribution up to a target distribution. And in that sense, we could express a noisy image as a combination of noise and sample from the target distribution. And all of what we had to do was to just predict the vector field aka the velocity which was noted u of theta. But one thing that we kept silent was how we could actually represent all these quantities in particular XT the representation of the image when it's noised at a given level the noise itself. All of these we just assumed that there was some way of representing these in some n or dimensional space.
So one thing that we will do today is to see what space could actually be more appropriate for this kind of work. And then the second thing is as you can see the models that we need to uh estimate so epsilon theta s of theta u of theta they're only a function of the noised image and the time but they're not a function of any condition any user input. So this is also something that we're going to see aka how can we incorporate a condition that could guide the generation towards what we want.
So this is the menu for today. So to recap we're going to see three things. The first one is how can we represent those noisy images. The second thing that we're going to see is how to represent the conditions that we want as inputs. So for instance, the input text here, a teddy bear reading a book that we want to generate an image for or an input image that you want to to condition your generation for. And then the third thing is how we can guide the generation process using that as an input.
So, we're going to start with number one, how to represent these noisy images. And I'm going to start with the most natural approach that you could have when you see those images. We as humans, we're very accustomed to seeing things in the pixel space. So here if you take this image as an example this image has a certain height h a certain width w. So what you can do is to represent each pixel as a function of the intensity of the color that composes it. And in particular a very common representation is using R. So red, green and blue. So in other words you would have each pixel that would have three values which would make up the color and you would have these values cover all pixels of the image. So you would have a dimension n that would be equal to 3 * the size the number of pixels which is h * w. And so you could represent your image as a tensor along the axis uh the height the width and the number of channels here which are three which represent R and B. So this could be one way.
But the big problem with that is that it's very high dimension. So if we take the example of a quote unquote regular image, so 1024 * 10 2024 pixels, then we would end up with a number of dimensions which would be on the order of a million. So 10 to the^ of six. And if you think about it, this is just a lot of dimensions to consider in our trainer generation paradigm algorithms which are of course a function of the dimension in which you operate. So that's one very big uh drawback. The second drawback is if you look at your image, you actually see that you have a lot of redundancies. So if you see for instance uh in a given neighborhood of pixels you have a lot of very similar values. So in a sense you can consider this as a waste of dimensions. So it's not very compact. And then the third thing is if you remember our whole first three lectures were about taking the image and you know adding noise and then just moving it in the space with respect to added noise. Well, here if you take an image and you were to move it slightly in that space by adding some noise, you would actually obtain an image a noisy image that would actually not be very meaningful. Would not it would not mean anything. So in particular, so I'm not sure if you recall these nice uh graphs or charts where you would see the areas of high density of uh your target desired distribution. So here what we want is clusters of observations that are likely to occur. But what we get is actually the thing on the right which is a space with very spiky regions where each spike represents uh an actual valid image and that's not good.
So for all these reasons we're going to just come up with a wish list of a space that we would like to have. So I'm going to go through that step by step. So the first one is the high dimensionality observation that I made. So here what we want is a dimension that's more quote unquote tractable. So that's number one. Number two of our wish list is to have a representation that does not have a lot of redundancies that has some um some way of compacting things. So number two is compact representation. Number three, regarding the representation not being meaningful. So here we want a representation that is meaningful. And you can think of this when I say you know what meaningful means. It's when you look at the space and you're able to have clusters of regions where valid images happen. That's what I mean by meaningful space. So this is our wish list.
So before we move ahead, I want to take the time to introduce two terminologies that are widely used in those papers. So I prefer to define them now and clearly so that you know what they refer to. So you will see that there is a lot of talk about semantic similarity. So what does semantic similarity mean? So semantic similarity of an image is something that refers to its global geometry to its overall structure. And in particular what that means is if two images are semantically similar what that means is that they roughly represent the same thing. So here on the slide you have on the left a teddy bear that is reading a book. On the right you have also a teddy bear that is reading a book. They're not exactly aligned pixel-wise but they're semantically similar. So that's for semantic similarity.
Now you have another notation which is perceptual similarity. So this one refers to more of of the local details of an image, more to the lower level texture of an image. And when you say that two images are perceptually similar, what that means is that to the human eye, they look the same. So for instance, you have again two pictures of a teddy bear. So, I'm not sure if you see the one on the right has some little glasses, but it's perceptually very similar. Does that make sense that that distinction? Yeah. The reason why I say that is I'm going to use that terminology going forward. So, I just want to make sure we all know what these are.
So, what are we doing? What we want is to come up with a space that fulfills our wish wish list that we established before. So here is our first attempt. So, how about we take an image in the pixel space and what we try to do is find a representation that's meaningful using the following architecture. So, let's assume we take that image and we put it into some model. let's call it an encoder that encodes the image and produces an output that is of lower dimensionality as the input. So here we go from an image of size H, W and number of channels which as you know is equal to three in the pixel space because you're like in the RGB representation. And here we end up with a latent representation of size lowerase H lowerase W and little C. And then what we do is we put that representation back to the decoder. So if we were to do that, our objective is to recover the original image. So we don't want to lose any information by encoding our image into a lower dimensional space. So here such a setup is called an autoenccoder. So the only job of an autoenccoder is to find a lower dimension representation that's called the latent representation. And what it does is take an image in the pixel space as input, compress it, and then decode it. And here the task is making sure that what you get as output is the same as what you get as input. So you may think okay it's a very useless model uh which I agree unless you're interested in the latent space which we are this is what we're interested in. So does this setup make sense? Cool. And then uh terminology wise, the latent space uh like that part is called the bottleneck just because we're forcing the model to rely on fewer dimensions and we want the model to still preserve the richness of the details that are present in the image. So let's see how we do that. But before we go, I want to introduce another notation that you also see a lot in papers which is called the spatial compress compression ratio which is defined by the amount by which you go from big h to little h. So it's just a ratio between the two and that ratio is typically on the order of eight. So that's the order of magnitude that you should have in mind. Cool.
So let's see now based on what I told you how each component works. So in terms of what is input in the encoder as we mentioned we have our input image let's note it X that we put in our encoder that we note EI and we get a latent representation Z that is equal to EI of X. And so the way we would do that while would be to use a network that contains some downsampling operations. So what that means is it allows us to go from a bigger dimension to a lower dimension. We're not going to go into too much details into what that network is because we're going to talk more about this in the next lecture. But I just want to tell you roughly the two main operations that you can expect which you may have seen. So the first one is convolutions. So who has heard of convolutions before? Yeah, everyone. Okay. So I can go fast. Convolutions is just a way to scan your image using a filter. So the image is in red, the filter is in blue, and what you get is what we call an activation map or a feature map. that is in purple. And then the second operation that you see in these kinds of networks is pooling. So pooling is a down sampling operator that allows you to either take the maximum of a given square or the average or either max pooling or average pooling. And it allows you to introduce some spatial invariance just because you can aggregate the pixels to a higher granularity. We're not going to go into too much detail, but these are just for reference.
So when it comes to the decoder, what you would do is take the latent representation Z into the decoder d theta and you would get your reconstructed output Xhat. So here you would use upsampling operations for instance reverse convolutions and the goal of this model is as much as possible for your reconstructed output to be as close as possible to your input. So the loss function of such a model would be something along the lines of just an L2 regression between the input X and the reconstructed input Xhat. So I have a question for you. Do you think this model is a good model? So this is what we're going to see now. So let's go through our checklist. First one is okay. Is the dimension more tractable? Yes, more tractable. Is the representation more compact? Yes, it's more compact. Is the representation meaningful? Meaning, do you have a nice shape in your latent space in the way we represent it? >> Not necessarily. >> Yes, exactly. Not necessarily. Well, the reason for that is we have not constrained our latent space to be structured in any particular way. So the model is just only incentivized to reconstruct the input as best as it can and it really doesn't care what shape the latent space is. So you actually still end up with this kind of space which is not good. This is not good because your flow matching model, your diffusion model is just not able to learn very well in those spaces because first of all the representations are all over the place. And if you remember, one common thing with all of these generation paradigms is that you start from a normal distribution like a standard normal distribution. So you need a way for you to not get lost. So if everything is far apart, it just doesn't work. So that's not good.
So let's go back to the drawing board and let's see what we can change. So we take our image, we compress it, and then we decompress it. Everything's perfect. Well, let's see what we can do in the latent space. Our goal here would be to quote unquote enforce the latent space to have some structure. So this is our goal and we're going to see how we can do that. Okay, let's go back to where we were. We take our input image X, but instead of just letting it through a standard encoder, what we will do is to actually not try to predict or generate the latent representation directly. We will actually predict two things. the mean and the standard deviation of the distribution that it can be drawn from. So in other words, what we're going to see to say is instead of saying that one x corresponds to one latent representation, we're going to say that one x actually is mapped to some probability distribution, which sounds kind of weird, right? It's not very natural, but we're going to see why that mitigates the problem. So here as a reminder, so the output of the encoder will not be just a latent vector. It will be so the mean mui and the variance sigma squar. So these are the two outputs of the encoder. And what we will say is that the latent representation of this distribution, this latent representation is actually drawn from this distribution. That's what we're going to say. And we're going to say something else. So in a bit of an informal way, we're going to say that this distribution is going to be roughly forced to follow or to approximate another distribution. So it's not very mathematically rigorous. We we were going to see exactly how we can express that. But in your mind, I want you to know that this is a condition for us to structure the latent space in a way where we say the latent representation of an input X would be roughly in a given region of the space. We're going to force that. We're going to incentivize the model to learn that. So that's the idea. Cool. And yeah, this uh distribution is okay, you may have guessed it again, a standard normal distribution just because it's a it's a nice distribution with great properties. And by the way, this distribution we're going to call it the prior the prior distribution.
So let's see how the decoder works. So let's assume we have put our input image into the encoder. We have predicted the mean. We have predicted the standard deviation. We have drawn a latent representation from the distribution. So now the decoding stage would be around taking that representation and passing it through the [snorts] decoder which again will output two things. the mean, mu theta of this latent representation Z and standard deviation. And what we're going to say is again that the reconstructed image is going to be drawn from a probability distribution of these two parameters. Well, one thing that I want to say is actually we're going to simplify this part of the problem a little bit. We're actually going to say that the variance is not something we're going to predict. We're just going to fix it to be constant just because there is less impact in you making sure to introduce some variation in the output because in the pixel space even if you add a little bit of noise it's not going to change the image a whole lot. So that's why people just say okay we're going to take some variance that's constant fixed and we're not going to predict that we're only going to predict the mean. We can draw from this distribution but sometimes people also just take the mean as the output. So this part is more of a detail. Yeah.
So uh the question here is that uh whether the latency is drawn from the gausian or is drawn from the q the encoder here q5 z given x. So it's a great question. So this model which we're going to see what the name is is both a model that we can use to learn the latent space and so have a reconstruction task but it's also something that can be interpreted as a generative model. So if your goal is to reconstruct the input then your input here is going to be to the Z that you drew from that distribution. So it's going to be the latent representation of X. But if your goal is to generate just a new image, then you would actually draw that Z from the prior distribution. So the question is the encoder gives you a mean and a variance. It does not give you something deterministic exactly. It gives you a mean and a and a variance. And what you're doing is just drawing from that. So you have Yes. you draw from that. It's a great question. So the question is well if you have a generative model here why would you bother and just use diffusion use flow matching use all these things? It's a great question. You have a number of differences between the two which makes diffusion flow matching and score actually perform better than VAE. One of the differences is here you only go from latent to pixel in just one step whereas the other ones you have several steps that just allows you to put more compute in there. That's one reason but there are some others that I hope we can get to if we if we get some time but we're going to see the relationship between the two. But great question. Yeah. The question is does the latent space look like a normal distribution? So it's there is something that makes it makes it be regularized quote unquote to be a normal distribution. It depends on a term that you can tweak with respect to a coefficient that we're going to see. And depending on the value of that coefficient, it may look more or less than a normal distribution, but we're going to see that. Cool.
So, the first time I saw this model, I was confused. You may be confused now and I hope what we're going to see m will make you a bit less confused. So here one thing that I want to say is in the autoenccoder case the loss function was pretty clear. You would just try to estimate the reconstructed output in such a way that it's as close as possible to your input. But here you have a bunch of distributions everywhere. There's not like an obvious way of expressing that without doing some math. So what I propose is that we derive the loss function for this model which I'm not sure if I can tell it now but I'm going to tell it is called the variational autoenccoder and we're going to see what makes it variational. So if you remember in a lot of cases the way you derive a loss function is by doing maximum likelihood estimation. So what does that mean? That means that you have some model of parameters theta. And what you want is to find the parameters theta that maximize the probability of your model seeing the data. That's what MLE is and that's what what we're going to use to see how we can derive a loss function that can help us learn theta. So in particular we what we want is to maximize the log of p theta of x. So I'm going to write it down. So what we want is to express log p theta of x with respect to something that we can compute because this we cannot really compute something just by looking at it. So let's see how we can do that. So I'm going to just start with expressing P theta of X as a function of something. So this model can be seen as a generative model that starts from a latent variable that you draw from what we call a prior distribution. So one idea of expressing P theta of X if you remember lecture one we can consider the joint probability distribution of P theta of Z a latent and X and some across all latent Z that led to X we can do that right. So what we say is that P theta of X is equal to this joint probability distribution of X your image in pixel space and Z your latent and you're just going to integrate over all possible latent representations. So this quantity is nothing else than you drawing from a prior. So P of Z and then using your decoder to estimate X given Z. So this quantity is the integral of your prior times p theta of x given z dz p of zion >> you know yeah exactly so it's a gausian you you know p theta x given z well it's your model your model that you want to find the problem is that you need to compute this for all possible latency and Z is still a high dimensional a high dimensional vector. So this if you were to compute this quantity for all Z is actually not something that you can do because you would have to compute a lot of such quantities with given Z's in order to get a good estimate and it's just not something that you can do. So do you remember a similar setting we had in lecture one which is the reason why we're going to see how we can find a convenient bounds that could be maybe more tractable because so far I have not used the fact that we also have a way to encode the inputs in order to get latents that are more likely to the the ones that I want to consider here. So what I'm going to do now is to find so what I want to do first is to maximize this quantity. So I want to find theta that maximizes this quantity. So we saw just now that this quantity is not something that I can actually compute. So what we're going to do is to find a lower bound that could be something that you could compute. And what we want to do is to maximize that lower bound in the hope that it will also maximize this in order to find our parameters theta. So let's do it. So you would have uh the following. So I'm going to just come here. I would just say that P of X is equal to the integral of P of Z * P theta of X given Z which is just at that point I'm going to use the same trick that we did in lecture one which is to involve [snorts] the output of the encoder and here it is Q5 QFI of Z given X time Q5 of Z given X. So what does this represent? This. So this represents the probability distribution of the Z's that are likely to occur for this given X because here in the original form what you were doing was to just sum across all possible latent representations out there but there are many of them. So it's like, you know, you're in the ocean and someone tells you, okay, just look through the whole ocean. But the thing is only a small subset of the of the Z's are actually the ones that you want. They're actually only a subset of the Z's that are the ones that are responsible for your X to appear. which is the reason why we're letting this Q5 of Z given X pop up and this expression. So this is equal to this and this gets equal to just the expectation of P of Z * P theta of X given Z over Q5 of Z given X where your Z is drawn from Q5 of Z given X. So why am I doing this whole thing? Because what I want to do is to take the log of this whole thing. I want to take the log of this whole thing and do the same trick as we did for the DPM, which is the Jensen's inequality. And I'm going to say that it's the expectation of the log of this whole thing. So I'm taking the log of this quantity which I expressed as a function of the output given by the encoder for which I can find a lower bound. And now we're going to see that this lower bound is actually something that is tractable. You're going to see it now. So what I'm going to do is to just rewrite this expression. So we have so the expectation of the log of the whole thing which is equal to the expectation of the log of p theta of x given z plus the expectation of so I'm going to be a little bit part I would say minus the expectation of the log of Q5 of Z given X over your prior distribution. So here what I just did was I take the log of PZ * P theta of X given Z over Q5 of Z given X and what I did is I just I use the property of the log which is logarithm of the product is the the sum of the logarithm of the terms. So I had this one pop up and this one I just rearranged. I put the denominator in the nu numerator and I added a minus and given that it's an expectation for both of the both of these that are over z drawn from your encoder output which is Q5 of Z given X. This term is actually the K divergence of Q5 of Z given X and your prior. So what does this all mean? What this all means is in order to find the theta that maximize this quantity we derived a lower bound that is something that we'll see structable. So it's this expression which we have divided into two terms. So the first one is expectation of the logarithm of p theta of x given z and the other one is a k k divergence. So we're going to see that these two terms are actually quite interpretable. The first one is actually a reconstruction term because if you remember P theta of X given Z is a normal distribution with fixed variance. So we know the probability density analytically. So if you take the log of the probability density of a gausian, you actually end up with something super simple which is the norm of x. So the inputs and the reconstructed outputs and then this second term is actually a term. So back to your question, it is a term that penalizes the distribution coming from your your encoder being far from this fixed prior distribution. So loss functions usually have a coefficient in front of this term to penalize more or less the fact of being far or close from this prior distribution. And depending on the coefficient there, you're going to have either something that will look like a normal distribution or not. So that is the part of your lo that will allow you to do that. I'm looking at the time. I'm not super in advance, but I'm going to just pause for one minute if you have any questions. >> So the question is we're pushing the distribution of the encoder to be more of the normal distribution. So we want to do that to normalize the latent space but we don't want to do that too much otherwise so the nor the prior distribution is p of z doesn't care what x is and in other words if you push it too much towards something that is just static it will just not consider the input at all it will collapse. Yeah. So it's called posterial collapse. Um so just one one other question and then and then I'll move on. Yeah. So the question is uh so we had this image with the little um spikiness and that was the autoenccoder the standard autoenccoder part. This one is actually structuring the latent space in a way that makes it more smooth more well put together. And yes so I'm going to speed up a little bit. Let's go through our checklist. Tractable dimension. Yes. Compact representation. Yes. Meaningful representation. >> Yes. >> Yes. Oops. Truthful representation. No. Wait. Why is that? So these models they actually end up reconstructing the image in an imperfect way and what we actually obtain are more blurry images and so I'm hoping that in the next 9 minutes we're going to see why that is the case and how we can solve it. So first of all let's look at the loss function that we have derived which is composed of a reconstruction loss and a regularization loss. So the reconstruction loss what what is it? We have not derived it here but I am telling you that if you derive it, it is the pixelwise L2 between the input and the reconstructed output, the reconstructed input, sorry. And so what that means is if the generation is a little bit off by a few pixels, it will actually get very penalized because the distance is pixel wise. So what ends up happening is the model tends to produce values that are more averaged which produces this blurriness sensation that you're seeing. But then you will tell me well ashin the standard autoenccoder also had that loss. So why didn't it happen there? Well, the fact is it also happened there, but it's even more pronounced here because of the structure of the model that we have defined which is that the latent representation is not a onetoone mapping with the input X. It is not. Instead, what we have said is a given input X maps to a probability distribution like a set of probable points that could be actually latent representations of this X. So the model will introduce this uncertainty even more in the reconstruction that it will output. And here what we want to do is to combat that blurriness sensation because here if we were to perform diffusion in the latent space and if we were to decode the final output in the latent space, the last thing that we want is for our output to be blurry. We we do not want that. So now we're going to see a couple of strategies that will allow us to combat that blurriness artifact that we're seeing. And the first one is going to see to be about a kind of loss that's called the perceptual loss. I'm going to say exactly what that means. So let's imagine you have your image X The intuition here is we don't want the reconstructed output to be compared with the input pixel to pixel because it does not allow for you know little shifting here and there. So what we want instead is to be able to compare the input and the reconstruction more in terms of similarity that something that looks similar to you as a human perceptually similar. So how can you do that? Well, there are some models out there that that are convolutional based for instance that actually extract features from these images in their layers. And here we refer them as feature maps. And one good things, one good thing about these feature maps is that they represent things from this image that have more of a spatial invariance compared to the original image. So what does that mean? That means that if you have two images that are slightly shifted, slightly different, then their feature maps will actually not be that different. So the idea here is instead of comparing the pixelbased images one to the other, we're going to compare their representation, their feature maps one to another. And I'm not sure if you've heard that loss, but there is a loss that's called LPIP. So learn perceptual image patch similarity. So it is a loss or it is a metric I should say that is composed of the difference between the feature maps of two images and that are weighted by a coefficient which is here WL. There are coefficients that are tuned by the authors of this paper in order to match what humans consider as being perceptually similar. So long story short, so by the way, the formula it's scary, but it's not that big of a deal. Like we don't care about the whole thing. What we care about is that we're not comparing images in the pixel space. We're comparing images between their feature maps. And what we're saying here is that the feature maps of two images better reflect their structure, their details, and they're less sensitive to spatial differences. So that is one strategy. So one thing to note is that we would weight this loss by some coefficient. You could think okay this loss is answering all our needs. Let's have the the weight for this part of the lus be super high. Well if you do that actually there are some bad things that also happen. So we have what we call checkerboard artifacts which as you can tell is like a gridlike artifact in the image that is similar to this one. So you don't want to have too big of a a weight either. So that's one. And then a second strategy to combat blurriness is what we're going to see now. So the problem that we have is that the decoder of the VAE is outputting an image that is blurry. So one idea could be to ask another model to tell you whether the input image so in this case the blurry image is a real image or not. And based on that, what we would do is incentivize the this other network called the discriminator to distinguish outputs coming from the decoder compared to real images, which would then flag blurry images as being fake. and we would penalize the decoder for generating images that would be flagged as fake by the discriminator. So, have you heard of GANs before? So, we don't have time to talk about them in detail, but I think this slide is summing up what is happening. So, you have two models that are competing with one another. The decoder that wants to produce images that the discriminator can cannot distinguish from being fake. And the discriminator that wants to distinguish the output from the decoder compared to images from real life. And what this will do is incentivize the decoder to produce more realistic images. And in doing so it will help with removing that blurriness. So that is called adversarial loss. So what we saw was that the original loss of the VA which we derived here led to outputs that were that were too blurry. It was fulfilling all the wish list that we had but it was not leading to a truthful reconstruction which is the reason why we saw two different strategies to add to our loss to counterbalance the tendency for the model to produce blurry outputs. So these are the perceptual loss and the adversarial loss.
So before giving it to Shervin I could not let you without telling you how this fits together with the diffusion models. So in particular, the reason why we did this whole thing is for us to use our diffusion flow matching model not in the pixel space but in the latent space because computationally it's just cheaper and it's able to scale. So now we're going to see what our strategy for training and inference is. So for training what we do is we first train the VA exactly like we like we saw and then what we're going to do is to train our image generation model like using for instance flow matching in the latent space of the VA and we're going to see how we do that. So let's suppose we have an image from our target data distribution. We pass it through the encoder. Our encoder gives us two quantities. So the mean and the standard deviation. And what we do is we sample from that distribution to get our latent representation. And then what happens is in our latent space, so we hopefully have a smoother latent space, we have our Z1 which corresponds to the clean image. We sample from the noisy data distribution. And here I'm just illustrating the case of flow matching. We have ZT which is weighted combination of the two. And what we do is we optimize for let's say our uh velocity like the L2 loss. We do that. This is for training.
So how about inference? Well, let's assume that you have trained your model. Then now what you want to do is to generate new images. So what you're going to do is to first sample from an easy distribution in the latent space. So for instance gausian noise and you're going to use the for instance flow matching the velocity that you have learned to solve it numerically and arrive at let's say point Z1 which is the latent representation of your end trajectory and what you're going to do is to use the VA decoder to decode that output back to pixel space. So let's see how it looks like. So this is your latent space that you have learned. You sample from the noisy distribution in that latent space. You solve numerically the OD using the learned vector fields that we've trained right before. You obtain Z1. The problem is Z1 to us is doesn't mean anything because it's it's not in the pixel pixel space. So what you do is you take that Z1, you pass it through the decoder and you obtain the image in pixel space.
So to conclude, what we've seen is a way to find a latent space that is of lower dimension, that is smoother, and that is also able to preserve the details of your image. And so in particular, we say that the encoder acts as a lowass filter. because what it does is represent those images mostly based on their semantic similarity. So this is actually something we have not seen but for a diffusion model to find all the details in its exploration is much harder than finding something that is semantically what it wants to generate. So this is the reason why we're giving the decoder the task of conveying all the low-level details, all the perceptual elements. And the role of the decoder is to take that latent representation and to make it an output that is full of details, that is realistic and that is matching what we want. And for that reason you have the encoder and the decoder which you you would think is symmetric. But in practice the VA that are used in conjunction with diffusion models are such that the decoder is actually bigger than the encoder. Does that make sense? So we we don't have a lot of time. So, one thing that I put down there is a longer form version of everything I mentioned in some very interesting articles which I recommend reading but not mandatory as part of this class which is um some results from flux tool which is a text to image model and uh the LDM paper the latent diffusion model paper that is at the bottom of the slide. Yep.
So, uh the question is what should we think of the role of the encoder and the decoder? So, at the end of the day, what you want is to have an easy time generating images at the end of the day. And what you want is to not have too high of a computational burden for your diffusion model to operate. So what you want is a latent space that is low in dimensionality and that is easy for you to learn. So what does easy for you to learn mean? It means having a space that is not super spiky but that is more smooth. And in particular, in order for you to obtain such a space, it is actually something that in practice something that you can obtain if you focus on the semantic meaning of these images as opposed to capturing all the lower level details. So in other words, what I'm saying is even if two images are perceptually not very similar, but if they represent the same thing, what we want for these images is to be located not too far from one another in the latent space. So we want the encoder to mostly capture the semantic meaning. So the encoder what we want is to make the job of the diffusion model easy and so so this is the learnability you will see in those those paper learnability. So once you have images that are semantically similar that are close to one another you do your diffusion process you obtain let's say a point that you say okay this is the latent representation that I want to have in the pixel space so now the job of your decoder is to make sure that all the details that you want to have in this final image are there so in that sense In contrast to the encoder, the decoder's job is to make sure all the perceptual details are present in the final decoded image in the pixel space. So in that sense, you have these two different roles of the encoder and the decoder. So you want your diffusion model to have an easy time learning this thing and you want a space that has some semantic information which is what I mentioned the encoder can do. But what you want at the end of the day is to have an image that contains the details that you want and that part is mostly something that the decoder can help with in the decoding stage. Okay. I'm 10 minutes over time but with that I'll give it to Shervin. Thank you Ashin. So now we're going to see together how to represent the conditions that we have at the input of these generation models. So the first kind of condition that we can have is a text one. So the text world when it comes to representing embeddings has a long history and I'm not going to go through the whole history of how we represent embeddings. I'm just going to focus of how people do it today and the way they do it is with transformerbased architectures. So let's first see how we can decompose the input sentence. So we have this concept called tokenization which is the practice of decomposing any piece of text into atomic components. So you can have something that operates at
an arbitrary level, word level, subword level or character level. And a sub level is usually um algorithms that learn how to best decompose the the tokens with respect to the data by focusing vocabulary budgets on those tokens that most frequently occur. And the tokenization technique that's most used these days is subword. But for simplicity, the one that we're you're going to see in the slides is more towards arbitrary, but it's just for representation purposes.
So who knows about transformers? Okay, so most of you. So there is a sister course that talks about transformers at length. I'm not going to talk about transformers at length, but just give the main pieces of information that you should know. So it is an encoder decoder architecture that is centered on the concept of attention and it came out in 2017 and most of the language models that you see today are based off of it. Even vision models you have many of them that are based on a transformer. And the good thing about it that made it work well is that its inductive biases are based on the concept of attention that prove to be generalizable.
And let's focus now on this concept of of attention. So if we had to summarize in a single sentence what it is, it is the practice of representing a given piece of input as a function of all the others. And here with the example a cute teddy bear is reading. If we want to represent in the space of attention the token teddy bear what we do is that we have embeddings for all these tokens. You have learnable matrices that will turn them into the space of queries Q keys K and values V. And what you're going to learn is a distribution of keys of of this key over keys. And the weights that you obtain are going to be those that you place on the corresponding values. So in summary, you're able to represent the embedding of teddy bear with respect to uh the others in this sentence in that way. And you can write this down in matrix form which is this soft max term that represents this probability distribution. So the square root of DK is just a normalizing constant that prevents the sum from becoming too large the dimension of this uh queries and keys and multiply it by the value which is which carries the embedding of the of the values.
So you can find these attention layer at three different locations. So two of them are uh self attention layers. So in the encoder and another one masked self attention which conveys the fact that tokens can attend only on those that were predicted before. And then you have another layer where attention occurs which is called cross attention which will make these queries attend on other encoded um keys and values.
So this is the high level and in just a few words the main component is that uh your input tokens are converted into embeddings with a learnable layer. You add some information regarding the position of your tokens. Then you make it through go through an encoder that will apply this concept of attention and also add on top of it some learnable mechanism to add some um like complexity for each token's representation which is those feed forward this feed forward network. And then at the decoder level when you want to generate let's say a sentence you start with some placeholder to token that you uh for which you add position embeddings as well that you self attend and then perform cross attention with respect to your encoded tokens and then you also have uh feed forward there to finally obtain your output.
So I'm telling you all that to mention that there is there are locations where embeddings can be extracted out of all of these. So if you use the encoder part of the network, you can find the end of the encoders, a location where you have an encoded meaning of your inputs tokens. And this is typically one of those locations where you could say these are the embeddings of my inputs.
Okay, great. So, um, now doing the same for images. Let's say you have a condition that is based on images. One thing that is pretty nice is that everything that we mentioned just now can be reproduced with images. So all these token representation are just numbers and you could well represent these numbers. uh you can associate them with patches of images and this is what the uh VIT vision transformer paper does which follows the same principle but instead of learning embeddings on tokens you learn uh embeddings of on patches of images. So everything say stays the same and you can perform tasks such as um image classification with it.
And something that I haven't done in the text space that we can do here together in this image world is to see how these pieces come together in in this example of a teddy bear reading a book. So here you decompose your image into these patches that we mentioned. Each of these patches you have a learnable layer that gives you an initial representation of what they mean. So in the case of text you would have a dictionary of tokens that are mapped to embeddings. But here you have something that is more continuous. So you can have a linear layer, you can have a convolutional layer, you can have everything you want. So you have something at that stage that gives a representation of dispatches and then in uh a classification setting you have a token. So for example here cls that will carry information regarding the class of your image that you want to predict. So each of these can be represent represented by embeddings and we mentioned that we have position information encoded in them and uh next class we're going to see more about position encoding and and all of this. So I'm not going to expand too much. So you have a representation for each of these patches of your input that you then feed your VIT and exactly as we mentioned at the output of the encoder you can locate one specific embedding that carries the semantic information of your input image. So here it would be the output corresponding to the CLS token um that here you project to learn uh a classification task for example but let's say you are at representation stage of your of your workflow you would just take the embedding that you see uh in orange at the top as your representation of the image.
Okay, great. So, something that I didn't tell you is that for text doing that process is cheap in data because the task that we use to learn these models is self-supervised by nature. We anchor on next token prediction in the world of text and you have a bunch of text in the internet. It's easy to have such data and um and it's easy to to learn. But in the image world, if you want to handcraft a similar task that is as meaningful as what we have in the text world, we have a a bit more trouble. One of these candidates that I just show shown is the classification task for example for which you need to gather labels. So it's it's quite expensive. And I just want to say that one follow-up work that works on embedding of images that we're not going to see in details today is uh dino self distillation with no labels for which the concept is to have a teacher and a student model that receive this different kinds of inputs and they aim at learning similar probability distribution in the output that are prototypes. So I think it's a very interesting paper to see how we can build representations out of no labels just out of the images themselves.
So okay great. So now we went through how we can represent text and images. Uh, so did everything make sense so far? So we went slightly fast, but I just want to make sure we got the gist out of it. Yeah, the comment is uh in the image world, do these uh images become tokenized? So you can think of them that way. The only difference is that we don't have a dictionary of tokens, but rather we look at them in a continuous manner. Because in the text world you would just have a lookup but here learnable. So that's exactly right. Yep. So the comment is depending on how you cut your image you can have errors. Yeah to totally right. And this is why it's a hyperparameter. This VIT model comes with different patch sizes and this is one of the trade-offs. Yeah. I mean absolutely. Okay. Awesome.
So now that we have seen how to represent text and images separately, you might wonder, okay, hey, I want to generate a new image with conditioning that depends on both, let's say, or I want to condition on either, but I don't want to have to deal with two spaces. So how would you bring them together? So what we want is to learn a relationship between text and image that has this nice property of when you look at the embedding space you want similar concepts either image text to be grouped together and dissimilar ones to be pushed away from each other with respect to some metric. So let's take the example of our teddy bear and uh something totally unrelated. Let's say a water polo ball. So the teddy bear with the text token teddy bear is similar. Uh same with the water polo ball image and the water polo token, but the opposite pairs are not. And we want a way to quantify each situation and learn it automatically. So how can we do that? So let's like build a matrix of these two. So this is just phrasing in a different way what I've just said. And what we want is teddy bear and the image of teddy bear to be of high similarity.
So what we're going to do is project these embeddings that we talked about in the same space. So we're going to build these vectors with the representations that we just saw before. So what we're doing here is the same. So we have our patches of images alongside the CLS token that you pass through the VIT architecture. And then once you get the last embedding representation, what we're going to do is to project it in a space that we're going to have it shared with the text space. So the property of that projection layer that we built is that it's going to output a vector of same dimension by construction as the one for text. So we do that here and then we do the same for the text world. So by the way I mentioned to you that we would look at embeddings at the end of the encoder layer. And here I'm going to I'm showing another procedure to get embeddings of text which is to pass all of these tokens to the decoder and decode the one corresponding to the last token and by construction of the attention mechanism and the transformer this embedding coming out of the last token is going to represent the whole sentence. So it's another equivalent way of showing the representation of tokens and we're going to take that representation and project it as we said in that same shared space and by construction this U and V are now comparable. You can do dot product. You can compute similarities. And we define this similarity s variable that is the scaled dot product. And one thing that we can do now is to transform this notion of similarity into some score that can be seen as a probability distribution through the softmax operator. So if you compute the exponential of the similarity between image teddy bear and token teddy bear over the sum of the exponentials of this image teddy bear and all the other text tokens, you find the probability distribution that this image corresponds to that piece of text. So okay, you can do that from a fixed image to the set of texts and you can do that on the other hand with a fixed piece of text token to all the images. So you can have both ways. And what we do is that we can define two losses that look symmetric that represents respectively how similar like the penalization of any error of a fixed image based on all pieces of text and similarity error of a piece of text with respect to all the images. So it's symmetric cross entropy and then when we sum these two losses we can obtain a final formulation that represents what we care about.
Right? So that is the key of the clip model that uh trains the representations of both image and text worlds into um like with this way of training things. So what we do is that um we scrape a bunch of images and captions off of off the internet and then for a given batch we say that a given image has its associated caption as the true label and then all the other captions in that batch are negatives. So it's called in batch negatives and you do the same thing of captions with respect to all the images and you obtain that way um final embeddings in the text and image encoders here that will generate the representation that you're looking for.
Okay. So uh here just to give some order of magnitude the paper talked about 400 m u pairs of data and it's great you you reach uh great accursive and unsupervised tasks. Uh but the issue here is this soft max. So every time you have a huge batch of data, you need to build a similarity matrix of a fixed image with respect to all the captions and symmetrically. And this is not uh great computationally. And also from a meaning standpoint, what you're truly saying in your loss is that you want to maximize the similarity of your true image caption pair, but you don't truly enforce the fact that all the other similarities should be each very small. So some follow-up work reframes this problem into sigmoid loss one. So instead of saying I want this one image to be to have a probability distribution that matches that of the true caption. We take one caption and one image separately and we ask ourselves the question do they match or not? And by doing so, you don't have all these similarity matrix complexities and also you have a stronger statement at at your loss level that's uh like pushes the positives together and the negatives further apart.
Okay. So, so any questions on this contrastive learning part? Yep. The question is do you start from a pre-trained encoder and what weights do you change? So I think in the paper they start from scratch and they train both encoders from scratch. You could very well uh freeze or like start with some pre-train encoder freeze all the layers let let the last layer I think typically you would just train everything. Yeah. And in real life I think you just wouldn't train anything. You would just pick something out of hugging face. But uh yeah in practice since it's easy to get all this data. So typically you would scrape the internet look at images and look at alt tags in HTML. So you can get all these pairs. So I think you're in such a setup where doing it from scratch makes sense but what you suggested makes sense as well. Yep. So I think it depends on what works best. Yep. So the question is is the transformer architecture a prerequisite for this task? Not at all. So you can do it with any encoders. And the reason why I talked about transformers is to ground the way of encoding into what people do in practice. But you don't have to have this architecture as a prerequisite. Yeah. Okay. Awesome.
So now we're going to reach the most exciting part of the class which is finally how do you guide our generation model with the condition and we're going to go back to thinking about what we discussed this past lectures. We had this diffusion mechanism where you are at this given step here XT + one and you want to find the less noisy version of the image XT. And the thing that we want to do now is to do that with a condition that I called Y but that you might see called C in papers. So you want to guide the generation towards generating a sample that follows Y. So a first idea that you might have is uh okay it's a classification. There is a concept of classification. I can come up with a classifier of weights five and try to make it work. So how could you make it work? You start from this probability distribution that you want to sample from and then by applying B rule uh like the conditional version with the right arguments for A, B and C, you obtain this product of two probability distributions that you can see on the left represents the one-step generation process that we know and on the right side you have the probability distribution of Y given XT which would typically be what the classifier would give you. So now let's see how we could sample from this. So taking a look at the probability distribution on the left. So if we recall the first lecture let's say in the DDPM world you know that this probability distribution follows a normal distribution of a mean that we learn and of variance that is following a noise schedule. So we don't learn it. It's something that but that we have. So we know the expression of this. If you look at the probability distribution of of this, it's uh something that we know. So we have that on one hand and on the other we look at our uh probability distribution at the end of the classifier. So here why are we not happy with that? I mean we have the data of the probability distribution. You just have to put the noisy image to your classifier and then you you have it. You have a probability distribution. Why are we not happy with that? It's because we cannot plug it into that product equation that we saw and then easily sample from it. We want to reframe it in a way that we can have this nice gaussian distribution that we had for example at the uncondition generation step. So what we do is that we have a trick that expands this probability distribution with a tailaylor expansion and then you come up with like a first term that is a function of XT. And when you plug the probability I mean the log of P5 that we just found into this product, you see that the probability distribution that you want to learn is simply the same one that you had but with a shifted mean. Yeah. So I just want to take a pause for a second. So we went through a process that where we tried to find an easy probability distribution to sample from. So we went to through a series of equation but all I want you to know from all of that is that we tried to get to this state where we can uh sample from an easy distribution and that's what we reached. So in some way we're we're happy. And then here what's good about it is that you don't need to retrain your unconditioned model. So your mu theta and your sigma t are things that you know and there is a downside to it. Now I mentioned classifiers. You need to learn a classifier that operates on noisy images. And and the process here is that you you need to have that classifier that predicts some probability distribution and then you need to take the backward pass on the network in order to find the gradient. Did that make sense so far? Okay, great.
So when we uh sample from the distribution that I mentioned in practice, one thing that people see is that it doesn't produce samples that follow the Y condition enough. So this is why you see that W factor being introduced that can be tuned to force the generation process to follow the classifier's gradients more. And u this is what we call the guidance part. So we saw that we don't have to retrain the uncondition model in order to sample from this distribution. But on the other hand, you need to have a classifier that operates on top of a domain that can be composed of noisy images. And the bad thing about it is that you typically don't have that out of the box. So it's typically something that you have to train as part of your training process which is a downside and also the fact that you need to compute the backward pass of your classifier is quite expensive which is also not great. And something else that I will mention is that you not only need to have that extra classifier and spend all these budgets to compute the gradients, but you need to make sure that the u like that the signal of your gradients scales well with the the coefficient w that you introduce. So you need some hyperparameter tuning to make sure that the operation that you're doing is meaningful.
So one question that you might ask yourself is it possible to do this guided generation without the needs of a classifier and the answer is yes. So if you use base formula you can see that you can define an implicit classifier so called pi of y given XT based on the generation like the unconditioned generation of your image and the conditioned generation of your image. So condition on Y which means that if you have a network that can generate an image without any conditions and that can generate an image with a given condition then you can solve this problem by plugging this equation back into what we had which is what we do here. So the equation that I uh that I showed regarding the like the distribution of the next sampling step can be written in epsilon space. So I mentioned this uh gaussian distribution of shifted mean. you can rewrite the noise that you want to predict into one that takes into account that classifier signal. And what the authors of the paper of the classifier free guidance paper did is that they plugged in the pi expression with the implicit classifier expression that I just mentioned before. and they obtain this very nice expression that is a function of the noise that we want to remove based on a model that isn't fed any condition and one for which a condition is provided and what you can see is that there is this W term in the middle that is that has about the same interpretation as the one that we saw before that will tweak the generation to be more or less towards the condition that you bring. So there's that and then now we're happy because we no longer have to train a separate classifier fi. We can just rely on the uh generation uh networks weights theta.
So how does that look like in practice? So with a certain probability called a unconditional probability, what you can do is sample an image from your training set and a time step and noise and try to predict the noise from it. So which is what we saw in the past and with probability one minus P unconditional. What you can do is do the same, but then use the embeddings that we saw in the parts just before to feed that network with this additional information and predict the quantity of noise to remove with the condition Y. So you do that one minus p unconditional times and what you get is something interesting uh which is if you look at how your predictions look like. So let's say the noise that you want to remove is the point top left and then the noise that you want to remove when you have a condition is the one in the middle. then the guidance hyperparameter W will push the prediction of the noise more towards the noise of the condition sample. And the reason why we do that is that because in practice if you don't have this hyperparameter you tend and if you just use the conditioned sample noise as the one that you use to denoise your noisy latent you might obtain images that do not follow your prompts very closely and this is why we have such a hyperparameter in the first place. Uh in terms of numbers, the guidance parameter is typically set to like just a few I think three in the paper. So typically more than one and I just want to put a disclaimer here that the equations that I showed with the with the classifier guidance hyperparameter might have several definitions. So you might see formulas formulas with W in there W + one. So all these values of W that you take are dependent on the equation that you operate from. So this is one. And then the other thing is that in practice at training time the proportion of time for which you show unconditioned generation is typically 10 20% of time. And uh this time you do not need to have an additional classifier at hand and to do an expensive backward pass. But you still need to do two inference steps because we saw that the equation has this epsilon theta of XT and no condition and the one with the condition which is two forward passes. So you still have some extra cost although it's better than the other technique. And this technique uh CFG is the one that's most used today.
Okay. So, any questions? Okay, great. With that, I hope you have a great weekend.