The Moment Generating Function

Welcome back. So I'm really excited today to introduce you to one of my favorite concepts in probability: the moment generating function. I remember when I was learning probability, you know, ages ago; this was one of the kind of biggest "aha" moments for me that connected a lot of different topics. So the moment generating function is roughly speaking going to be a nice function that helps us compute moments—things like the expectation, the variance, and higher-order moments—things like skew and kurtosis—in a pretty nice, compact formula. And it's also going to be central in proving some of the most important theorems in probability and statistics, like the central limit theorem. Okay, so what is a moment in probability? The moment of a distribution, the moments of a probability distribution, are the expected values of my random variable X, or X², or Xⁿ. So there are actually infinitely many moments: there's a first moment, a second moment, a third moment, fourth moment, and so on and so forth. And they are the expectation value of my random variable to the power n.

We've already seen examples where this is super important. The mean is the first moment; the mean of my distribution is the first moment. And the variance of my distribution is closely related to the second moment; it's the second moment minus the first moment squared. And these are extremely important for characterizing my probability distribution function. So if I know that my distribution is, let's say, a normally distributed random variable X, or a Gaussian, then it is uniquely determined by the first and second moments. So the first and second moments uniquely determine kind of the mean and the standard deviation—this is, let's say, μ + σ and μ - σ—the mean and the standard deviation of that normally distributed random variable. But higher-order moments are actually also important; there are lots of distributions where you need more moments to uniquely determine that distribution. So the third moment, the expected value of X³, is known as the skewness of the distribution. So if I have a distribution that's a little bit lopsided, something like this, it's going to have some skewness. The Poisson distribution, for example, is going to have some nonzero skewness. Um, if I have a distribution that has some, you know, kind of fat tails, I might need my fourth moment, or my kurtosis—the expected value of X⁴. Um, so I can draw, you know, a picture of that; maybe I have something that looks kind of like my normal distribution, but I've got some bumps out here in the tails; there would be some fourth moment, or some kurtosis, in that distribution. And these moments are unique identifiers of my probability distribution. It's a lot like the derivatives of my function in the Taylor series approximation. So if I want to approximate a function with a Taylor series, I can, you know, collect all of the derivatives—the nth derivatives of my function—and uniquely write the Taylor series for analytic functions. Same basic idea applies for probability distributions; you can uniquely determine the, let's say, the cumulative distribution function, and sometimes the probability distribution function, in terms of this kind of infinite series of moments. So this is kind of like the Taylor series for probability and statistics—super, super fundamental concept and very, very useful.

Okay, so let's define this moment generating function, which is going to allow us to compute these moments very efficiently. Okay, um, so I want to do this in pink, I think, because I like my probabilities to be here. So the moment generating function is going to be defined in terms of an entirely new variable; usually these are defined in terms of a variable X; here we're def—we're introducing a new variable because we're going to do some transformation from probability space into this moment generating space. And this is defined as—I'm going to write it for a discrete variable X and for a continuous random variable X. Um, so for a discrete random variable X, it's defined as the sum over all possible states X can take of eᵀˣ times the probability density function P that big X equals little x. Um, this is for X discrete, for a discrete random variable. Okay, this is just how it's defined; I'll tell you why it's defined this way and how to use it in a minute. And for continuous random variables, for X continuous—things like the normal distribution or an exponential distribution or a gamma distribution—this is defined as the integral over all possible X's—so, you know, let's say generically from negative infinity to infinity—of eᵀˣ times my probability density function f(x) dx. Now, first off, those of you who have studied differential equations, um, control theory, dynamical systems, you'll recognize right off the bat that this is the Laplace transform of my PDF; this is the Laplace transform of my PDF, my probability density function f(x), which is pretty wild that this very useful function for approximating these moments happens to be the Laplace transform of my probability density. This actually maybe shouldn't be so surprising; Laplace was one of the absolute founders of modern probability and statistics; in fact, Bayesian statistics I think should really be called Bayesian Laplace statistics, because even though it was discovered by Bayes a couple of years earlier, Laplace independently discovered it and went way farther in developing it. So most of our modern theories and kind of language and thinking about measurement error, probabilities, and statistics are actually responsible because of Laplace. So Laplace must have been thinking about this transform both in terms of probabilities and differential equations, and there's some deep, deep connections. If you think about stochastic processes, like, um, you know, radioactive decay, you have a differential equation and a corresponding probability density function, and you can take the Laplace transform of both, and it means something. So lots of deep connections. But zooming out, this moment generating function is something you can compute; it's the Laplace transform of your probability density function, and it has this very, very useful property that the moment generating function can easily help you generate the moments—the higher-order moments—of your probability of your random variable X.

So the way that looks—and maybe I'll do this in blue—this is a theorem I'm going to prove next time, in the next video, but I'm going to state it here. So there's a theorem that the nth derivative of my moment generating function, evaluated at zero, is equal to this expectation of Xⁿ. This is super useful. So if I take the first derivative of this function and evaluate it at zero, I get my expectation—my mean. If I take the second derivative of this function and evaluate it at zero, I get the expectation of X². The third derivative evaluated at zero gives me something related to my skewness; fourth derivative related to kurtosis. So all of my moments I can—I can generate—I can easily kind of grab from this moment generating function—super, super useful because these are quite difficult to calculate generically. Um, and once I have this function, it might be much, much easier to do this. And again, this is going to uniquely characterize—maybe I'll write this down—this moment generating function M(t) uniquely determines the cumulative probability distribution—the cumulative probability distribution—specifically, the probability that my random variable X is less than or equal to some little value x. Okay, and this is something we'll—we'll see later. Now, notice that I didn't say the probability density—the PDF—I said the cumulative distribution function, the CDF. There is a reason why I talk about CDFs often in terms instead of PDFs; I'll tell you later, but just gloss over this fact and just say this moment generating function uniquely determines the distribution—your probability distribution. Okay, and it's very, very useful for computing these moments that are again kind of the fingerprint or the Taylor series kind of approximation of your distribution; these uniquely determine your distribution. Great. So I want to give you some examples: How do you do this for Poisson? How do you do this for normal? How do you do this for exponential? And next time I'll prove this theorem that my moments give me these—that my moments can be drawn from this moment generating function. Okay, good. Um, so I think from now on I'm just going to be doing some examples. Um, is there anything else I want to tell you? I think that's pretty good. Moments are important; they uniquely determine your distribution, and you can pull them out of this kind of magic moment generating function, which happens to be the Laplace transform of your PDF—pretty profound, pretty powerful stuff. Let's do some examples. Um, so I'm going to do Poisson, normal, and exponential. So example: Let's start with Poisson.

The Poisson distribution. So my PDF here—this is a discrete random variable—so let's say X is Poisson. Um, we'll say, you know, X is Poisson with some λ, and of course that means its PDF is going to be—what's its PDF?—it's λᵏe⁻λ/k!. That's the PDF of Poisson. So the moment generating function is going to be the sum of eᵀᵏ, sum over k—this is my moment generating function of t. Okay, so here it's defined in terms of some dummy variable X; here my dummy variable is k because that's what we normally do for Poisson, but it's this expression here, and it's not that hard to manipulate this and get a pretty easy expression. So my moment generating function is this, which is the sum—I'm going to pull this e⁻λ out of my sum because the sum is over k—this is e⁻λ sum of eᵀᵏλᵏ/k!. I pulled out my e⁻λ, and then I combined these terms into eᵀλ, all of that to the power k/k!. Pretty simple stuff. Now this expression here, the sum of this thing to the k/k!, this is the definition of e to the stuff inside the parentheses. So this equals e⁻λ * e to all of the stuff in the parentheses—eᵀλ—all this sum here over k—this is the definition of the exponential of all the stuff in this parenthesis. So it's e⁻λ * e to the λeᵀ. So this equals eλ(eᵀ - 1). Okay, so this is the moment generating function M(t) for the Poisson distribution, and I can take its first derivative and evaluate at zero to get the mean; I can take its second derivative, evaluate at zero, to get something related to the variance, and so on and so forth. This is an easy function; I can work with this. This is how you compute the moment generating function of a discrete random variable like Poisson. Okay, let's try another one; let's do a normal distribution.

Okay, example two: Normal distribution. So let's say that X is in normal, and let's make it really easy on ourselves; let's say it's mean zero, standard deviation one. I don't want to deal with, you know, some hairy normal distribution—just a basic standard normal. Um, okay, we're going to write this out; we're just going to do it; it's not going to be that bad, and it's going to save us time later when we want to compute higher-order moments and do things like that. This is also really important for the central limit theorem. So the moment generating function of this PDF is the integral from negative infinity to infinity of eᵀˣ times my PDF dx. Um, so it's integral negative infinity to infinity of eᵀˣ, and my PDF of my normal has a 1/√(2π); I'm going to pull that out because it's just a normalization factor—1/√(2π) times e⁻ˣ²/2. Okay, so I pulled my normalization factor out for my Gaussian normal distribution, and this is the PDF of my normal distribution, and this is my Laplace transform integrating factor. So this is the thing I need to compute, and pretty easy—I'm literally just going to combine these exponentials, do some really basic algebra, and integrate this thing. Okay, so one tiny fact that's going to help us do this is that x²/2 - tx—that's just the minus of these exponents—when I multiply this—this thing is equal to -(1/2)(x - t)² + t²/2. Really, really simple. If you take (x - t)² * (1/2) and then you subtract off the t² term, you're just left with an x² and a 2tx. Okay, this is really, really simple; just, you know, expand your polynomial—whatever—this is just an identity—to take this exponent here when I multiply these two and write it in terms of these squared exponents that are going to be a little bit easier to work with. This is just an identity. And so now this integral here is equal to 1/√(2π) integral minus infinity to infinity of e⁻⁽¹⁄₂⁾⁽ˣ⁻ᵗ⁾²⁺⁽ᵗ²/₂⁾ dx. So this is um—tx - x²/2—which is the minus of this identity. So I get the minus of this identity, which is -(1/2)(x - t)² + t²/2. We're just doing integration now. And now I can split these integrals up; I can split this—this product up here. So this equals my eᵀ²/² doesn't depend on x, so I can pull that outside of my integral. So now I get one—I get eᵀ²/² * 1/√(2π) * 1/√(2π) integral minus infinity to infinity of e⁻⁽¹⁄₂⁾⁽ˣ⁻ᵗ⁾² dx. Now this is the probability distribution for another normally distributed variable with mean t and standard deviation or variance equal to one. So this thing also has integral probability equal to one. If I integrate this from negative infinity to infinity, the law of total probability says this integral equals one, and so my moment generating function is just this stuff on the outside; it's just equal to eᵀ²/². This is my moment generating function for a normal—pretty simple stuff. Okay, I mean, like, there was a little hairy math in the middle, but I think you followed; this is calculus; it's easy; we know how to do this. And now we have this really simple function for the moment generating function; this is actually a nicer function than my PDF, in a way; it's—it's closely related, but it doesn't have all my normalization stuff. Okay, let's do one more example before we close out.

So the last example I'll do is my exponential distribution. Okay, so now I'm going to do example three: Exponential. And my exponentially distributed random variable is pretty closely related to radioactive decay and things like that, so you know that's where your ODEs and your Laplace transforms are going to come in if you really wanted to follow up on this. So the moment generating function is again equal to the integral, in this case from zero to infinity, because my exponential distribution only is for positive times. Um, and here I'm going to introduce a different dummy variable s for my Laplace transform because my exponential already is defined with a variable t, and it's going to get confusing, so we're going to say this is eˢˣ times my probability distribution—an exponential—which is λe⁻λᵗ dt. And now I'm integrating this with respect to t. Um, and now my moment generating function is actually in terms of a dummy variable s; this is actually more like a Laplace transform because my exponential we're already using the variable t; I'm integrating, you know, out dt, and my Laplace variable now is going to be s, and I'm going to get a moment generating function in terms of s—totally fine; you can call these variables anything you want. Um, and this thing is really easy to compute. So I can pop my λ out, and I get—what do I get?—I get λ integral zero to infinity of e⁽ˢ⁻λ⁾ᵗ dt. And this is really easy to compute this integral; this is just how you do Laplace transforms in general. This is λ/(s - λ)e⁽ˢ⁻λ⁾ᵗ evaluated at the bounds of integration infinity and zero. So this is only defined if s is less than λ. Okay, if s is bigger than λ, this thing is going to blow up; we're assuming real-valued s for now, and so s has to be, you know, less than λ for this to—to work. So this is defined if s is less than λ, and if s is less than λ, then the infinity bound this goes to zero, and the zero bound this goes to -1, so this equals λ/(λ - s). This is the moment generating function for an exponential distribution, and it's only defined for some values s, depending on λ. Okay, three different examples of how to compute this moment generating function; it's pretty easy for discrete variables; for continuous variables the functions aren't too nasty, and you can use these to calculate the moments—the expectation, the variance, the skewness, the kurtosis—the fingerprint of your distribution for really, really, you know, a wide variety of probability distributions. And again, last thing: This will be useful when we start to prove the central limit theorem—that if I add up n independent but identical random variables—if I add up a bunch of the same distribution random variables—they will start to converge to a normal distribution. We're going to use the moment generating function to prove that; we're going to show that that sum—that the moment generating function of that sum of random variables starts to converge to the moment generating function of the normal distribution. So we're going to use this exact result to prove the central limit theorem. Okay, thank you.