Covariance and Correlation in Probability

Welcome back. Okay, so today we're going to introduce a really important set of concepts that we're going to use all the time in probability and statistics, especially when we deal with data, machine learning, kind of fitting with data, fitting models with data. Those concepts are covariance and correlation.

Correlation is a term that comes up all the time. You've heard that correlation does not imply causation. Often times you try to see if two variables or two events are correlated, or if data has correlations. So the principal component analysis, singular value decomposition, kind of the basis of higher dimensional statistics, is really very largely based on correlation and covariance. Roughly speaking, the covariance; correlation is sometimes just described as like a normalized covariance. So we're mostly going to talk about covariance.

Covariance can be approximately thought of as quantified in the joint dependence between two variables, X and Y. So we know that we can have a joint probability distribution; we can have conditional expectations of X and Y, conditional probabilities. The covariance kind of talks about how does the variation in X depend on the variation in Y? And we're going to want a nice property; we want the covariance of X with itself to just equal the variance of X. That's going to be a really, um, kind of important property. So we want whatever we define this covariance of X and Y to be, if I plug in two copies of X, I want it to equal the variance of X. Okay, so let's define this. We'll talk about some examples, and then we'll define correlation.

Good. So the covariance of two random variables X and Y is pretty easy to define. It's the expected value of X minus its mean; we're going to call that μ_x, that's the mean or the average value, the expectation value of X. It's the expectation of X minus its mean times Y minus its mean, where I'll just maybe label this in orange: μ_x is the expectation value of X, and μ_y is the expectation value of Y. It's the mean or average of Y, the mean or average of X. And you'll notice that this is almost identical to the definition of variance. The variance of X is the expectation value of X minus its mean, quantity squared. So if I plug in two copies of X here, I recover the expectation of (X - μ_x)², which is the variance of X. So this is good; at least this is, you know, very close to the definition we're familiar with of the regular old variance of a single random variable. This is how two random variables X and Y co-vary; the covariance of those two random variables. This is a pretty simple thing to compute and to work with and to analyze and to understand. It's an intuitive, an intuitive notion, is that the ex, we take, you know, the variations of the samples X from the mean. So if I have a distribution of X, not all of the values will land perfectly on the mean, and my expectation, you know, I have some probability that they'll land away from the mean; there's some, normally like, some standard deviation of X, and the same for Y. And what we do is we compute the expectation of joint variations from their means.

Okay, um, I at this point might actually want to draw a picture for you, and then we'll write down some properties of this thing. So what do I mean by covariance? If I have an X variable and a Y variable, and let's say that my data looks, um, like, let me see if my data looks like, um, this. So I'm going to—we're actually going to code this up; we'll generate data—and I have a bunch of lectures on principal component analysis; it's all about these kind of covariances between variables, between two different variables. So you can kind of go to that principal components analysis SVD set of lectures if you want to just immediately jump to high dimensional, um, you know, joint distributions and covariances, covariance matrices. But for now, just imagine that I have data that is roughly kind of a Gaussian; let's just assume it's kind of a Gaussian, but it's an oblong Gaussian. So there's a preferred direction, and it has a non-zero angle in this XY plane. So this collection of data, if I actually, you know, I would compute its mean of X; each of the has a joint distribution; there's a PDF in X, and there'll be some PDF in Y, and there would be some joint distribution. And if I find the mean value here, I could literally compute how do these points, how does the variation in X from its mean relate to the variation of Y from its mean? So if I pick a little test point here, let's pick a little, a little test point here; this would be, you know, X - μ_x, and this would be Y - μ_y. And for most of these points, if I have a large X - μ_x, I will also have a large Y - μ_y because this thing has a positive slope. And the fact that this data kind of has that, um, that slope and this tightness to the distribution indicates that this, this data is going to have a large covariance. Okay, so there's a large covariance in this data.

Let me do another example. Another example: let's say I have data where it's a little bit less steep and a little bit fatter. So I'll try to draw something that is, you know, just a tiny bit more, okay. So now I've got a wider distribution and a little bit less steep of a correlation here. So this would still have a covariance between X and Y, but it's a less strong covariance. If I have a large positive X variance from its mean, I don't expect as large of a Y variance from its mean. And then in the kind of extreme case down here—and again, I actually encourage you to generate these kinds of point clouds in Python and actually compute this sample expectation, compute this expectation averaged over all of your 100 or thousand data points and convince yourself that this covariance is higher than this covariance is higher than this covariance—where down here I'm assuming that I have kind of a symmetric Gaussian in X and Y. Let's see if I can draw this; so it gets less dense as you go farther away. So you have this kind of symmetric, um, Gaussian in X and Y. This is probably going to have covariance almost zero between X and Y. There's, there's really no, um, you know, there's no correlation between deviations in X and deviations in Y. Okay.

Good. So this is just pictorially what I mean by by covariance. If I have a big slope and a tight, you know, not very much spread, I should have a lot of covariance. If it is a lower slope and a fatter spread, it'll be lower covariance. And eventually, if I have, you know, no preferred direction and a lot of spread, it'll be zero covariance down here. Okay, and it doesn't matter where my mean of this distribution is; I could center it anywhere because we're already subtracting off μ_x and μ_y. Good. And I'll just label here μ_x and μ_y. Good. So let's write down some properties here. Some properties of my covariance. One of the useful properties, properties, and I'll do this one in orange I think. So I'm actually just going to expand this out; I'm going to like multiply these two and expand it out and come up with a cool formula for the covariance in terms of expectations of X and Y. It's a pretty useful formula. So we're going to say covariance of X and Y; I'm just going to rewrite what we already have here. This is the expected value of (X - μ_x)(Y - μ_y). This is a little dim, so I'm going to switch to my brighter orange. (X - μ_x)(Y - μ_y). Okay. This is just taking and expanding this out into all of its four terms. Now we know that the expectation value of a sum of quantities, even if there is joint dependence between X and Y, the sum of these, they split into four different sums of four different expectations. So I can write this as expectation value of XY—the brackets versus round brackets, square versus round doesn't matter; it's just whether or not I have too much stuff inside of here; sometimes I use square, sometimes I use round—minus the expectation—this μ_x is a constant, so I can pull it out of my expected value—so this is minus μ_x expected value of Y minus μ_y expected value of X plus the expected value of this constant is just this constant, so it's just μ_xμ_y. Good. And I'll switch colors again here. So this guy is just μ_x is the expected value of X, so this is expectation of X times expectation of Y. This guy is expectation of X times expectation of Y, and this guy is expectation of X times expectation of Y. So all three of these terms are, you know, minus expectation of X expectation of Y minus expectation of X expectation of Y plus expectation X expectation Y. So I get two minuses and a plus; this adds up to equal a single copy of expectation of X times expectation of Y. So this equals expected value of my variable X * Y minus the expectation of X times the expectation of Y. And so this is a nice property; this is a nice formula for the covariance of X and Y. This is something you can derive; we just derived it, that I can write my covariance of two random variables X and Y in terms of these expected values of X and Y. And you'll notice right off the top, right off the bat, if X and Y are independent, then this covariance is equal to zero. We know that if X and Y are independent, then this expectation is the product of the expect, and this term will cancel this term. So maybe I'll just write that down. Obviously, if X and Y are independent, then my covariance is equal to zero. Then covariance X, Y = 0. Again, because for independent X and Y, this expectation splits into the product; expectation X times expectation Y, which cancels this term, which gives a covariance equal to zero. So if I have independent variables, the covariance is definitely equal to zero. The reverse is not always true. I can have a jointly distributed PDF; I can have two variables X and Y that are dependent on each other and still have a zero covariance. I can definitely have a covariance of zero and have X and Y be dependent, but if X and Y are independent, I have to have a covariance of zero. If I wanted an example of something that had covariance zero but dependent X and Y, I would make X uniform, I would make X a discrete random variable that's uniform on -1, 0, and 1. So it has a one-third probability of each of these values, and I'd make a Y variable that is X². Okay. So two discrete random variables; clearly Y is dependent on X; these have a joint PDF; they totally depend on each other. But if I compute the covariance of these two matrices, uh, you can go through the math; the covariance of X and Y—and this is actually pretty easy; you literally just sum up over the very few possibilities of X and Y; there's three possibilities of X and two possibilities of Y—the covariance of this is equal to zero, even though X and Y are not independent. Okay, so that's just a really, really, really easy counter, like, counter example. If X and Y are independent, then the covariance is definitely equal to zero, but if they are not independent, you can still have zero covariances sometimes. The reason is is because X—this is kind of an even function over kind of an odd domain; if I, you know, add up all of these, I'm getting like an equal amount of negative numbers and positive numbers—you'll see; just work this out; it's really easy; there's six things you have to add up; the probabilities are easy to compute; this is a pretty good exercise here. Okay.

Good. So other properties. We have definitely already seen that the variance of X equals the covariance of X with itself. And literally, if I just plugged in an X and X here, I would get the definition of variance; the expectation of (X - its mean)². So this is a property that's definitely true. We also have that the variance of a variable X + Y, the variance, not the variation, the variance of this variable Z = X + Y is equal to Var(X) + Var(Y) + 2 covariance(X, Y). So I'd want you to convince yourself of this; I want you to like actually take X + Y and plug it into both places here, expand it out, and convince yourself that you get Var(X) + Var(Y) + 2 covariance(X, Y). Okay, and then I want you to think how does this change if X and Y are independent, and how does it change if X and Y are dependent? Pretty simple; if they're independent, then this is zero, and you get Var(X) + Var(Y). If they're dependent, you get this extra covariance term. Good.

What else do I want to show you? This is the main stuff. This is a way of quantifying the joint dependence between two random variables. So if the two variables are highly correlated, meaning that like a variation in X implies a variation in Y, there will be a high covariance. If there is a low correlation between X and Y, there will be a low covariance here. This is kind of colloquially speaking. I guess I should define what I mean. I defined covariance; now I'm going to define correlation. The correlation, um, is essentially just a normalized covariance. Correlation of X, Y, and I'll go back to pink for the probabilities; the correlation, we define corr(X, Y) is equal to the covariance of X and Y; it's equal to covariance of X and Y divided by the standard deviation of X times the standard deviation of Y. Okay, divided by standard deviation of X times standard deviation of Y. And the reason we normalize by this is because I can actually take this distribution and I can give it a larger covariance just by making all of the numbers bigger. If I scale this thing up, remember if you scale up a variable X, its variance scales squared; it's like a, it scales with X². And so if I make this, this distribution just bigger, if I make my numbers bigger, if I convert from feet to inches or meters to centimeters, my covariance will be a bigger number. And so I divide by the standard deviation of X and Y to normalize that covariance. This is just a normalized covariance. And essentially, sometimes we call this σ_XY; sometimes we call this covariance σ_XY divided by σ_x * σ_y; that would also be a way of writing this if you like. And there's a nice property; this is not true of covariances, but it is true of correlations; this nice property that the correlation of aX + b and cY + d is just equal to the correlation of X and Y. So essentially what this means is that I can take my distribution of X and Y, and I can shift it by b and d; I can shift it over and up by (b, d), and I can stretch it out by a factor a and c in the X and Y directions, and that doesn't change my normalized covariance; my correlation. It will definitely change my covariance; in fact, you should compute what is the covariance of this transformed data; it'll be interesting. But the correlation doesn't change when I do this linear transformation of my data, which is pretty, pretty nice and pretty useful property of correlation. So that's why we often want to deal with this normalized covariance.

Okay, super important property in probability and statistics; the notion of covariance and correlation. This tells me some notion of kind of joint variation of two variables, and this is going to be very useful in high-dimensional statistics when we have a lot of data. Let's say I, you know, poll 10,000 people, and I ask them each a 100 questions; I can find correlations in their answers, and I can maybe infer patterns in that data. Okay, that's the basis of principal components analysis, singular value decomposition; that's a whole set of lectures later, for linear regression and multi-linear regression and modeling, and this is kind of the foundation of that. Okay, thank you.