Transcription
Welcome back. So we were looking at the binomial distribution and showed that under pretty modest, uh, assumptions for large n, this starts to converge to the normal distribution. But that is only if the probability of my event p is not too small or too large. Okay? Remember the binomial distribution is the number of successful events out of n independent Bernoulli trials. So, for example, this could be flipping n coins or rolling n dice, um, things like that. Then this binomial distribution converges to normal, but if my probability of this Bernoulli event is too small or too big, then this normal approximation actually starts to fail. Okay, and I want to just show you how, and I want to show you, uh, the solution. So the solution is what's known as the Poisson distribution. So I'm just going to write that over here: the solution is the Poisson distribution. And I'm pretty sure that Poisson tried this approximation out for small p, and he realized something fishy was happening. So let's do like just a little thought experiment. Let's pick, uh, n equal 1,000. So we're doing a really big number of trials, but let's say that p is super duper rare. Let's say the probability of each of these Bernoulli events is one in a thousand. Okay, so 1 in 1,000. So in this case, with n equals 1,000 (big n) and very small p, now my normal distribution X is distributed as normal; the mean is going to be 1,000 * 1/1,000; the mean is 1; and the standard deviation, uh, squared is going to be again approximately one. Q is basically 0.999; it is close to one. So n * p is about 1. So this is a normal distribution with mean one and standard deviation squared, or variance of one, also standard deviation of one. And if I plot that distribution, uh, maybe I'll make my axes in blue here. If I plot that distribution, and let's say this is, uh, x equals 0, this is X, and this is X equals, uh, 1, and let's say negative one here, what we find is that this thing has mean one. I'll do this in yellow; this has mean one. But because the standard deviation is one, that means a decent amount of the distribution is left of x equals 0. A decent amount—about 16%, I think it's—yeah, I think it's about 16%. You can compute the exact amount; maybe it's 14, maybe it's 15. I think it's about 16% of this distribution is left of zero, and I can't have a negative number of coin flips. I can't have a negative number of dice rolls. I can't have a negative number of successes, uh, at least not in probability. Um, and so I think Poisson realized that it's pretty fishy if 16% of your distribution is non-physical; doesn't make any sense in this normal approximation. So the normal approximation is great for large n and moderate probabilities p and q, but if p or q gets really, really small, then this approximation—about, you know, 16% of it—just doesn't make any sense; you know, it's non-physical. But the solution is actually really, really simple. So the answer is this Poisson distribution. So I'm just going to derive it here; it's super easy to derive. We're going to derive the Poisson approximation, um, and maybe I'll, I'll do this in yellow and blue as well. So what we're going to do is we're going to let, uh, we're going to define this new parameter Lambda equals n * p. So it's essentially just like the mean, uh, of our normal distribution; it's the kind of expected number of, of successes, the expected success rate. And we're just going to take this binomial distribution, expand it out, and then use some properties of maybe n * p not being too large or something like that. So the binomial—so if x is binomial, uh, with n, comma p—then the probability of X equaling, um, K—because this is an, this is a discrete distribution, so K is like an integer, the number of heads equals 15, let's say—the probability of, of, of X equaling K is n choose K; we've already written it down; I'm just rewriting it again—times my probability to the K times, uh, 1 minus my probability, 1 - p to the n minus K. And we can actually expand this out in terms of Lambda. Okay, so what we're going to do is we're going to do some like magic here, just some manipulations, and we're going to find that something simple simplifies in some limit. Okay, so what we're going to do is we're going to say this n choose K—this is not an approximation; this is just n factorial / K factorial * n - K factorial—and p is just Lambda / n. So p to the K is Lambda over n to the K, and 1 - p is 1 - Lambda / n to the n minus K. Now this—if I take the limit, um, one of these limits—this might start looking like an exponential. Okay, um, and we're going to play around with this a little more. Um, I'm going to write out this factorial and this factorial, and I'm just going to manipulate some things around here. Okay, so this equals Lambda to the power K divided by K factorial. So I've kept Lambda to the K and K factorial here. And now what I'm going to do is I'm going to take this n to the power K and this n factorial, and I'm going to group those together. So I've got, um, my n * n - 1 * n - 2 dot dot dot * n - k + 1; that's n factorial over n - K factorial divided by—for each of these terms I have an n; there's K of these terms; there's K of these terms—n to the k, n * n * n time dot dot dot times n. Bear with me; why am I doing this? Because Poisson said this is a good idea, and it ends up working. Okay, so I've done all of these terms, and now I just have this term left. And what I'm going to do—all I'm going to do is split this exponent—I'm going to say this is 1 minus Lambda over n to the n * 1 - Lambda / n to the -K. Okay, so now this is exactly the same as my—I didn't make any approximations; this is exact binomial up until this point—and now what we're going to do is we're going to take the limit as n goes to infinity. So as n goes to infinity, some interesting things happen. Lambda to the K over K factorial; there's no n's; this is, this doesn't change. So I get Lambda to the K over K factorial. As n goes to infinity, this is basically K numbers that are very close to n divided by K copies of n; this fraction converges to one in the limit of large n. So you'll have to convince yourself of that, but this fraction here in the middle goes to one because it's only K terms divided by K terms, and they're all almost identical to n for really, really big n. So this is times one, um, and then I have in the limit where n goes to infinity, this is 1 - 0 to some power; this goes to one. And it's this term here: the limit as n goes to infinity, this becomes e to the minus Lambda. This is in fact the formula for e to the minus Lambda is the limit as n goes to infinity of 1 - Lambda / n to the n. So all of this ends up being e to the minus Lambda. And I'll just write out the things that equal one; um, this goes to one, and this goes to one in the limit as n goes to infinity. And so what's left is this Poisson distribution: the probability that x equals K is equal to Lambda to the k divided by K factorial e to the minus Lambda. And just like we plotted the normal approximation and compared it with binomial, we can do the same thing with the Poisson distribution. We can plot this, uh, for small p and large n and show that this actually converges to the binomial distribution in that other limit. So I'm just going to cross this guy out. So this is what happens if we have small p or q—sometimes q, but let's just say small p. Okay, um, and this is a super duper useful distribution. This is essentially the distribution that, uh, determines the number of rare events that will occur. Rare events are super important: catastrophes, um, ideally accidents and bad parts in a factory. You know, if you're, if you're building, um, a widget and you need it to be super duper reliable, then the probability of, of failure—one minus the probability of success—should be very, very low. You're going to need a Poisson distribution, not a normal distribution. So really simple example, um, here: let's say that I am in a factory and I have, you know, fluorescent lights, and my fluorescent lights are pretty reliable. The chance of one fluorescent light going out in, uh, in a day—let's say the probability here, I'm going to actually say like probability of failure, uh, um, the probability of one light, uh, going out—let's say that that's, uh, 0.002. Okay, two out of a thousand will fail, um, will fail, you know, in a day. Okay, and let's say that n—let's say that n equals—I have 10,000 of these lights in my factory. I have 10,000 of these fluorescent lights in my factory. So p equals 0.002, and n equals 10,000. So Lambda equals the product—product of these two—which is 20. Then in that case, the probability of exactly k lights going out in one day is equal—or is distributed as Poisson with Lambda equals 20. That's how we would write this. So the probability in one day, if the probability of a light going out today is 0.002, or 2%, and I have 10,000 lights, the probability of exactly k lights going out is Poisson distributed with Lambda equals 20. And so you could compute: what's the probability of five lights going out? What's the probability of one light going out? What's the probability of 20 lights going out? You can compute that almost exactly using this Poisson distribution. Okay, um, I think in a follow-on lecture I'll plot this and show you that it's similar, uh, to the binomial in the small p limit, and I think I'll give you some historical examples, actually. Um, there's a really cool example, I think, from the Prussian Army where they had a weird number of horse deaths, and a statistician—you know, they ask someone: is this expected? Is this a reasonable amount of dying horses just assuming that it happens rarely, but it could happen, or is there something bad happening? Are they mistreating the horses, or, you know, is the management, uh, causing bad horse outcomes? So, uh, I'll give you some examples; I'll plot this; and I'll give you some cool historical examples. But basically, the Poisson distribution governs rare events, or events that happen, you know, uh, sporadically in time, and we're going to dive a lot more into this later. Okay, thank you.