Transcription
Welcome back. So in the last lecture, we derived the binomial distribution for the number of successes in N independent Bernoulli trials. For example, if I flip N coins, what's the chances that I get K heads? Um, that would be a random variable that satisfies the binomial distribution. So the number of heads in N fair coin flips would be a binomial variable. And what I alluded to at the very end of last lecture is that in the limit of large numbers of samples, this binomial distribution converges, um, actually pretty quickly to a normal or a Gaussian distribution. So we've all seen a Gaussian distribution, the normal distribution, the bell curve; we've, you know, seen this all over the place. And this is a natural limit of a binomial distribution. So what I want to do today is write down the binomial distribution, write down the normal distribution, discuss how they're related, and then code up in Python these two distributions and plot them to see how good the agreement is. Okay. And I'll just mention here, this is for large N, for large numbers of independent samples, and roughly speaking, N * P * Q not too small.
And I'm just going to remind you, the binomial distribution is a random variable X where the probability of X equaling some number K is equal to N choose K times the probability of success to the power K times the probability of failure to the N minus K. And I should probably remind you, this random variable X, binomial N, P, is the number of successes in N independent Bernoulli trials, where each of these Bernoulli trials has a probability of success P. So, you know, essentially X = B1 + B2 + ... + BN, where B sub I is Bernoulli with probability P. Okay. So this means a coin flip is a Bernoulli random variable. You know, like if I flip one coin, the probability of it being heads is 50%; the probability of it being tails is 50%, if it's a fair coin. And so that would be Bernoulli 1/2. Okay, the pro—that would be a single event, and X is binomial if it is the sum of a bunch of Bernoulli independent trials. So if I flip 10 coins, the number of heads in 10 coin flips is a binomial distribution with 10, 1/2; those would be the parameters: N = 10, P = 1/2. Similarly, if I roll dice, um, now the chances of getting, let's say, a six—let's say my Bernoulli random variable is: did I get a six? That's a success or not; that's a failure. So P would be one and six; there's a one in six chance of rolling a six. And then the number of sixes I roll in 20 dice rolls would be binomial 20, 1/6: the number of trials, the probability of success. Okay.
We derived this all last time, and in the limit of large N, this converges to the normal distribution. This converges to the normal distribution. X is, we say, distributed as a normal random variable where the mean is going to be N * P. This is the expected number of successes for large N; it's just the chance of getting a success times N. And this is going to have a variance, or a standard deviation squared, of N * P * Q. And remember, we defined here Q = 1 - P; probability of success of one of these trials is P; probability of failure is just 1 - P, that success didn't happen. Okay. And roughly speaking, this is my mean mu and my standard deviation squared; this is squared, or my standard deviation squared, also known as the variance. Okay. And so the probability for large N converges to this normal distribution. And I'm actually going to write out what's the probability of X. So the probability of X equaling x, this kind of distribution function P(x) = little x, this is equal to—and I'm just defining this; we haven't seen this before, and we're going to come back to this; it's such an important distribution—this is 1 over the square root of 2π times sigma; the sigma is outside of the square root here; times e to the minus (x - mu)^2 / 2 sigma^2. Okay. This is the formula for a Gaussian curve, for a normal distribution with mean mu and standard deviation sigma. And here you would just plug in N * P for mu and N * P * Q for sigma^2, and you would get this normal distribution that is a really, really good approximation to this binomial distribution. So, for example, if I flip 100 coins, that's a large N, the exact number of heads is distributed as binomial 100, 1/2, but it's very well approximated by a normal distribution with these parameters here. Okay. And we're going to, in a subsequent lecture, we're going to talk about how do you actually compute things using this normal distribution; how do you compute the probability that the number of heads is between 20 and 50, or less than 70 in 100 coin flips? That's a lot easier to compute using this normal distribution than adding up a bunch of things from this bin distribution. So way easier to compute here. I'm actually going to make a note of that: it's easier to compute with the normal distribution. And sometimes we just write curly N(mu, sigma^2); that's the normal distribution; so much easier to compute things down here. Okay.
Why don't we code this up and just show visually that for large N, this distribution and this distribution look almost identical? And maybe we can play around with, you know, as N gets small, as P gets small, things like that, and see what happens. Good. Um, I already coded this up. You can, you know, download this yourself; you can reproduce this; you can write it yourself; it's really quite a simple simple code. And then you can analyze and play around with it. Okay. So let's see. I'm just going to do a couple of basic things. I'm going to import NumPy; I'm going to need SciPy because the combinatorial functions, the N choose K, that lives inside of SciPy. So I'm importing SciPy as SP, and I'm also importing this special module because com—N choose K is inside of this. And we're going to plot some things, so I'm including Matplotlib. Good. Now I'm just going to cook up some really, really simple examples here, and I've already, again, pre-written this code. So we're going to start with a large N, N = 100; we're going to start with equal probability, so we're talking about coin flips; probability of heads is 0.5; probability of failure, or not heads, is also 0.5. And I'm going to compute the binomial approximation, the binomial distribution, and then the normal approximation to that binomial distribution. So maybe I'll just put in parenthesis here that this is exact and that the normal distribution is our approximation; that's important. And again, pretty easy to cook up; in fact, you could just ask GPT to make the blocks if you want. To compute the binomial distribution, I'm going for, you know, for all of the possible numbers of heads, so that's 0 to 100. So from K = 0 to 100, the probability of X = K is just N choose K; that's this comb function here; N choose K * P^K * Q^(N - K); super simple. And I'm just going to, you know, tack on these probabilities; so I'm going to get a big vector of probabilities here, PX binomial; those are my probabilities. And then similarly, I'm going to use this normal approximation. So I'm going to define mu as N * P, and I'm going to define sigma as the square root of N * P * Q. So sigma^2 is NPQ, so sigma is the square root of that. And then this function here, it looks a little ugly, but you can type it out pretty easily; it's just, you know, 1 / (sigma * sqrt(2π)) * e^((-x - mu)^2 / (2 sigma^2)); super simple. And we're going to plot these two things and just see how close do they look. Okay. I think I already imported stuff. Now I'm just going to run this code, and pretty remarkably, it's actually really hard to even see a difference here. You can see that these curves are almost perfectly overlapping with each other. If I change my limit a little bit to, let's say, you know, -40 to 40, sorry, 40 to 60, we'll zoom in a little bit, and maybe you can see a tiny bit of orange peeking out here. So the binomial and normal distributions are almost perfectly overlapping with each other; that's really—and that says for large N, like N = 100, this is a very good approximation of the binomial distribution. Okay; easier to compute down here; easier to analyze; really, really simple distribution that we see a lot.
Let's play around with some of these parameters and see what happens if N is small or if P is small. So those are kind of cases we might want to look at. Let's take N down. We're going to talk a lot about this idea that when you add up independent random variables—in fact, they don't have to be Bernoulli; if you add up independent random variables of almost any distribution—they will eventually converge to a normal distribution. This is one of the big properties in probability called the central limit theorem, and it's kind of profound and unexpected that under very weak conditions, if I add up independent, identical random variables, they will converge to a normal distribution. And the rough heris—the rough rule of thumb is that for N about equal to 30, this starts to become a very, very good approximation. So N = 30 is kind of a large sample size or a large number of independent trials after which their sum starts to really look very, very Gaussian or normal. So let's just try N = 30. If I plot that here, again, the discretization is coarser; there's only 31 bins in my histogram for the binomial, but the actual values are almost perfectly touching this normal distribution; it's a very, very good approximation. What if I make it smaller? What if I make it N = 10 or 5? Okay. So at N = 10, you can start to see that there's a little difference; it's honestly still surprisingly good even for N = 10, a very small N, but you're starting to see some discretization effect. So if I flip 10 coins, maybe it's easier to model that as binomial than normal, or more accurate. And let's try, you know, a real extreme case, N = 5. So we're going to flip five coins here, and now you start to see the real limitations; the normal distribution kind of approximation breaks down for really small N; it's just not a good approximation anymore. But for N, you know, 30 or more, the normal distribution is going to be kind of almost a perfect approxim of this binomial as long as the probability P is not too small. Okay. So I hope I've convinced you of that here. One last thing I want to show you, because this is going to come up later: if this probability P or this probability Q is really, really small, then this normal approximation also starts to act funky again. So let's go back and just remind ourselves that for N = 30, we get a very good approximation, but now let's say that my probability is really, really small; let's say my probability of success is, you know, 0.02; so only, you know, 2% chance of a successful Bernoulli trial. So now it's a little hard to see; I'll zoom in in a minute, but it's a little hard to see, but you can see that these distributions don't really agree very well anymore. So maybe I'll change my limit, so I'm like -5 to 5. And what you can see is that—well, so my normal distribution actually should be going from like -5 to 5. So why don't I just do that? Good. So for small P, remember P is now a small probability; even for large N, my normal distribution weirdly has a bunch of area to the left of zero, and I can't have a negative number of heads or events; I can't have a negative number of successes. So weirdly, for small P, my normal distribution just stops making sense. And in that small P or small Q case, we're going to need something called the Poisson distribution. Okay. So this just is kind of a cartoon that if P or Q is small enough, you get this weird non-physical behavior that the normal distribution predicts negative successes; that doesn't make any sense. And so we're going to need to replace the with something called a Poisson distribution for those cases. But for all normal, regular kind of probabilities where you have, you know, a large N and a reasonable probability like 0.5 or even 0.35, our normal distribution is going to be a very, very, very good approximation to our binomial distribution. Okay, good. So that was really what I wanted to show you: that the normal distribution is a natural limit of the binomial distribution in the large N limit; it's very, very accurate even for, you know, N = 10, 20, 30, and it just gets better and better as long as P and Q aren't too small or too rare. And this is a really important consequence of something that we're going to come back to called the central limit theorem. This is like one of the highlights of probability: that if you add up, under fairly modest conditions, if you have N independent random variables of almost any distribution—they can be uniform, they can themselves be Poisson, geometric, whatever—if I have N independent random variables that are, you know, identically distributed and I add them up, their sum is going to start to converge to a normal distribution for relatively small numbers, and you know, 30, 40, 50. Okay. That's the central limit theorem; we're going to use it all the time; it's kind of a generalization of the law of large numbers, and it's super, super, super, super important. This is the first and most kind of important and basic example of the central limit theorem. All right, thank you.