Transcription
Welcome back. Okay, so today I'm ready to state one of the most important uh theorems in all of probability and statistics: the central limit theorem.
So last time we wrote down the law of large numbers, and we actually proved it uh using Chebyshev's inequality. This is a very intuitive statement that if I have data, if I have data from an independent, identically distributed random variables uh from n different identical independent random variables X1 through Xn. Again, these could be n coin flips, n dice rolls, n uh questionnaires you ask n different people, all kinds of things—the heights of n different, you know, uh Americans or n different Texans—it doesn't matter. This is uh n independent random variables that you sample.
Then the sample mean—literally the average, the numerical average of those samples—will converge to the mean of the distribution that they belong to as the number of samples goes to infinity. Very intuitive results. Um, we have a gut feeling for why this is true. Okay. Um, if I collect more and more data from uh identically distributed random variables and I average them, that average will converge to the expected value of that distribution. Makes total sense. Okay. Um, I should probably have written "sequence of identical uh independently distributed random variables." Sometimes this is called i.i.d., identical independently distributed, and that just means that it's the same fair coin being flipped n times.
Okay. Now the central limit theorem is more powerful; it's a more powerful result and it's much more useful. What it says—it's the same basic idea—it says that this, this sample mean x-bar converges to Mu, but specifically that it is distributed as a normally distributed random variable with a calculable variance and mean. So it says—I'll actually just write down what the central limit theorem says here, maybe in yellow—the central limit theorem has the same basic assumptions: we have n identical independent random variables. And what it says is that the sample mean, the sample mean, which again is just um, you know, x-bar sub n = 1/n sum i = 1 to n of all of my independent trials—it's just literally the the mean of all of my data—this is going to be a normally distributed random variable. This is normal with mean mu and uh standard and variance Sigma squared over n. So the variance of this, this sample mean uh is Sigma squared over n, meaning its standard deviation, the SD, is just Sigma over root n. Um, that might make you feel more comfortable. This is the mean. It means that not only does the sample mean converge to the expected value of the distribution mu, but it does so in a way that as n gets large, this distribution of ex of sample means will start to be distributed as a Gaussian random variable, as a normally distributed random variable.
This gives you very, very tight estimates on how good your estimate of the sample mean is when you're dealing with data, and this is super important. So, for example, if you um are trying to measure some physical quantity—let's say you're trying to measure the speed of light, or you're trying to measure um, you know, the mass of an electron or something like that, something hard to measure—you might end up doing like 30 random 30 experiments. You might, you might actually do the experiment, you know, 30 times, and you'll get slightly different values for the speed of light or for the mass of the electron in all of those 30 different experiments. And what this says is that regardless of what the actual distribution of these random variables are—maybe they're uniform, exponential, placon, whatever they are—if you add them up and average them, that average value starts to look like a Gaussian random variable where the mean is the actual mean of the distribution and the standard deviation gets smaller and smaller and smaller as you increase the number of experiments you do. This has profound implications for modeling experimental error, how to do statistics to tell something about the large population from a small sample mean, um, and it's one of the most important theorems in all uh probability and statistics.
This is one way of stating it in terms of the sample mean, which is the law from the law of large numbers. Before, I'll just give this a little subscript n because this changes as n goes to infinity. We'll start to do some numerical experiments on this soon. We'll actually compute this thing for n increasing. We'll do this over and over and over. We'll do this like, you know, we we'll compute these sequences for n equals let's say 100, and then we'll repeat that experiment maybe 50 times, and we'll see what is the expected variance of this distribution. We'll show that it converges to a Gaussian distribution on a computer, and we'll also prove it using the moment generating function.
The proof of this um is actually pretty challenging. Um, it's, it's going to take me a little while to build up the math to prove this, and the proof is interesting, but it's not essential. You, if you can take my word for this, if you can understand how to use this result, you don't necessarily need the proof, but it is quite interesting, and I will present that later. The basic idea is that it's based on something called the moment, um, the moment generating function, um which, roughly speaking, the moment generating function is the Laplace transform of your probability density function, and the moment generating function of a sum of random variables, um, let's say the moment generating function is called M of T given uh some probability distribution, some PDF uh p of of X. Then if I have the sum of a bunch of random variables, if I had the sum of a bunch of random variables, this xn is essentially the sum, the weighted sum of a bunch of random variables, then the moment generating function of that sum is the product of the individual moment generating functions of the uh individual uh independent trials. And so what we're going to do is we're going to approximate this moment generating function of this, this thing we want out of the moment generating functions of the thing we have, and we're going to show that this uh kind of aggregate moment generating function converges to that of a normal distribution. So dot dot dot uh more soon. That's the thumbnail sketch: is that we're going to use the moment generating function. It's a nice way of showing that this PDF converges to this PDF. Actually, technically the cumulative distribution of this converges to the cumulative distribution of this. We're going to show that using the moment generating function. Very, very powerful idea, kind of an advanced topic, um but I'm excited to show you that soon. Okay.
Um, this is one statement in terms of the sample mean, and I chose to write it this way because this connects to the law of large numbers. There is another way of writing it where you could say that just the sum of all of these variables, the sum uh we'll call it S sub n equals just sum from uh k = 1 to n of x sub k. So I'm not dividing by n, I'm not taking the average; I'm literally just adding up all of these random variables. We can also say that this sum S sub n is normal, is a normally distributed random variable, and now its mean is n mu and its uh variance is n Sigma squared, or standard deviation is root n Sigma. So this and this are essentially completely equivalent statements, but sometimes you're going to see the central limit theorem just uh stated as how the sum of these variables is distributed. I think it makes a lot more sense to think about how the average, how the sample mean is distributed, because this is um what's going to be useful for understanding experimental error, design of experiments, how big of an n do we want, do we need if we want, you know, our variance to be less than than a certain amount, you know, all kinds of interesting things there. How big of a sample do I have to, how big of a a poll do I have to do to tell the outcome of an election? Okay, things like that, those are important uh concepts here, and it's all related to n and to the Sigma squared over n in the central limit theorem.
Now, again, the really profound thing, and I'm going to say this again because it's so profound: this doesn't matter. It doesn't matter what the distribution of x is. It doesn't matter uh what the distribution of my x uh i's is, and that's a pretty wild fact. So it doesn't matter how these are distributed, as long as they're identical and independent with a mean and a variance, you can make this very, very powerful statement um of the central limit theorem. Okay. Um, I think that's essentially what I want to tell you—mostly just to state it, to indicate that this is um a powerful generalization of the law of large numbers. The law of large numbers just says that your sample mean converges to mu, and that the variance get smaller and smaller um as n goes to infinity. This says something much more powerful: it says that this variable becomes distributed as a normal distributed random variable; it has a shape and uh and you can quantify its mean and its variance um in terms of the mean and variance of the these independent random variables, regardless of what their distribution is. Just super profound, kind of unexpected result, cornerstone of probability and statistics. We're going to use this all the time when we do hypothesis testing, experimental design, you know, confidence intervals, basically anything where we are, you know, using data and trying to make some statistical statement about that data. Okay, thank you.