The Expected Value (Mean) of a Probability Distribution

Welcome back. So today, I am going to introduce one of the more important concepts in probability and statistics: that of the expected value of a random variable X. Sometimes this is called the expectation value, um, and it's kind of, if you were going to randomly sample from this distribution a bunch, what would you think the average of those samples would likely be? That's essentially what this expected value is.

And so for a given distribution—this is my kind of regular Gaussian distribution over some variable X—um, in this case, the expected value is actually going to coincide with the most likely value, the kind of mean of the distribution, μ. But that's not true for every probability distribution, for every random variable X. Sometimes you get counterintuitive or even misleading results, and I'm going to tell you about that in a minute.

So, approximately speaking, the expected value is the center of mass of your distribution, of your probability distribution, of your probability distribution. Okay, this probability density function over your random variable X. And the way we compute it is by essentially taking a weighted average of all of the values of X, weighted by the probability of actually finding that value of x.

So, um, I'm just going to write this out in math. The expected value of x, in—I'm going to start with a discrete random variable, something like a Bernoulli or a binomial or a Poisson, something that has a discrete number of elements—and this is going to be: we're going to sum over all elements of X. So, uh, we're going to sum over all of the possible values this variable can take on, and generally these are going to be like integers. I'm just going to say like sum over all of K, um, the value x_k that that this—the random variable could take on. So I'm adding up the actual value of my random variable times the probability of X equaling that specific value, x_k. And if I wanted, I could write this a little bit more carefully, and I would say that this is the probability that my random variable x equals a specific value, little x_k. Okay, so this is literally just a weighted average of all of the values, little x_k, that my random variable could take on.

So let's say I'm flipping coins. Um, you know, I flip my fair quarter 100 times, um, and my random variable X is the number of heads, then I would sum up over all possible numbers of heads. So if I flip it 100 times, I could get zero heads, one heads, all the way up to 100 heads. So I'd add up, you know, 0 to 100 are the values here times the probability of actually getting that specific number of heads, which would follow the binomial distribution, and I could read those values off of, let's say, Pascal's triangle, for example. So this is computable, and it tells me my expected number of heads that I would be most likely—I would—I wouldn't be surprised at all if I got 50 heads. So I'm guessing this should be something like 50 for the binomial distribution with 100 coin flips.

I can also write this in continuous random variables, things like my Gaussian, normal distribution, where X is a continuous variable, and now this is going to be an integral. So for continuous variables, I'm going to have—my expected value of x is now just going to be the integral over all possible values X can take, so generically from minus infinity to infinity of x times my probability density function f(x) dx. Okay, again, just a weighted average of every little x that I could possibly take in this distribution times its probability of actually hitting that little x * dx. Okay. Um, and so this is expected value for discrete variables and continuous variables here.

Um, and there's another interpretation. This is a very, very, very useful interpretation, and again, this right now I'm talking kind of as probability, but there is this notion of statistics. If I actually collect measurement data from the real world, I measure a process, I actually flip that coin 100 times, or I go on the street and I ask a thousand random people what's their height, um, then I'm going to be getting sample data that maybe will approximate these distributions. And so if I sampled a bunch of—if I sampled x a bunch of times—if I sample X, I'm going to say x_j n*, if I sample x_n independent times and I average and average, then this mean x̄ = 1/n * the sum from j = 1 to n of all of my independent trials—this is my sample mean—I'm going to put this in parentheses—this is my sample mean. Okay, my sample mean will converge as n goes to infinity to the expected value of that random variable. Then this sample mean will converge—the limit as n goes to infinity of x̄ will equal—will converge to—this expected value, which I'm going to call μ. It'll equal my expected value of X. And this is a really important result. We're going to come back to this. This is actually—this is the law of large numbers. We're going to—we're going to prove this later, um, but essentially this is just a statement of the law of large numbers: that if I sample my distribution enough and average that sample, it should converge to the analytic expected value of that distribution. I should converge—my sample mean should converge to the true mean of that distribution, and my, you know, kind of variance around that μ will shrink as n goes to infinity. Very, very, very important result here, and it's another kind of example of what this expected value means: it's the limit of the average of a bunch of trials of this random variable. And later we're actually going to code this up. We're going to do 100 coin flips, and we're going to see how that starts to converge, and we're going to do that a bunch of times and see the variance of those sample means, and that's going to tell us lots of things about the statistic, and we can always invert that and ask questions like: if I gathered 50 samples and here is my mean, how likely is it that it is a—that it's actually being sampled from this particular distribution? How likely or unlikely is it that I got this sequence of samples, um, you know, given things like its mean and its variance? Okay, good.

Um, this is super useful, super simple, but it also can be quite misleading. Okay, so I want to point out a couple of things that can be pretty misleading here. This is a Gaussian distribution where the mean μ is the expected value. Here is another completely different distribution in yellow. This distribution has the exact same expected value, the exact same center of mass of the distribution, and in fact, that value has zero probability of actually—of—of actually sampling an element that has that value. All of the weight is in these two kind of these bimodal peaks here. And so this yellow curve has the same expected value, but it's a completely different distribution, and in fact, that distribution you would never expect to actually sample an instance of X that had that value, μ. So that's kind of weird. Weird things can happen with oddly shaped distributions.

Okay, I need other numbers, other parameters to define this distribution and to distinguish these two. One of those numbers is going to be the standard deviation. So the expected value is kind of the average; the standard deviation or variance is going to measure how much spread my distribution has. So the yellow one clearly has more spread than the pink curve, and that would distinguish these two. And this is called your first moment. The expected value is your first moment, kind of like your moment of inertia. It actually looks a lot like a moment. The variance and standard deviation are related to the second moment, and it turns out there are higher-order moments: third, fourth, fifth, and so on. And taken together, those higher moments almost are like a fingerprint for your distribution. So if I know all the moments of this yellow distribution and all the moments of my pink distribution, I can say a lot about them, and I can distinguish them. It's almost like the Taylor series for a function; it's like an expansion of your probability density in terms of the mean, the standard deviation, the third moment, fourth moment, fifth moment. So I'm just going to make a little note of that: that um, this is the first moment; these are called moments—μ, this expected value; the second moment would have to deal with the standard deviation or the variance; but there are more and more moments... and these moments are like a fingerprint or like a Taylor series kind of expansion of your probability density. And we're going to use this later; it's fascinating stuff. This is related to the Laplace transform of your PDF—Laplace comes up everywhere in probability and statistics—and this is one of the coolest places is in this moment-generating function that generates these moments. But that's an aside.

Really what I was trying to tell you is that I can have distributions with the same expected value that are completely different, and the expected value doesn't actually mean that it's even likely that I find my distribution at that point. Kind of weird. Okay, so it's not likely the most—it's not necessarily the most likely value of x.

Um, so in fact, I'm going to define a couple of things here. So the most likely value of x is called the mode. This is the most likely value of x to find my function—literally the value of X that has the highest probability. Okay. There is the mean or the average; that's what we computed here; that's the expected value. This is the kind of weighted average. And then there's a third value called the median, and that's actually often times the most useful for statistics where there's outliers or weird distributions. The median—and this is what we call the middle of the distribution. Okay, so it's literally the value that's in the middle of the distribution, which weirdly in this yellow case is actually still at μ, so this is maybe not the best example of median. Um, and I'm going to write out and define what these are in a minute, but I want to point out this notion of the median being robust. This is a robust way of doing statistics. So if you have outliers, the expected value is highly sensitive to outliers; the median is very robust to outliers. What do I mean by outliers? Um, let me give you an example. So let's say I have the distribution of wealth, um, let's say like the amount of money people have in their banks in the US. Okay, and let's say that nominally it looks like this: there's kind of a fat tail, but there's a peak and there's a distribution. This is, you know, the amount of wealth people have in a given country, but there are on the far, far, far, far ends of the tail people like Jeff Bezos and Bill Gates and Elon Musk, and there are only a few necessarily of these people that are like ultra-wealthy, you know, hundred-billion-dollar net wealth, but that actually moves the mean significantly. So the mean of the distribution actually gets well off of what you would expect, kind of if you didn't have these outliers, what the mean would be, but the median, which is robust, is actually going to do a good job of capturing this peak here. Okay, and I actually looked up the numbers; it's pretty shocking. The average—sorry, the median US household wealth—the median is about 200k; that's kind of approximately the peak of this distribution. The average household wealth is 1 million—1 million dollars—and that is almost entirely because of these huge outliers. So these outliers aren't just moving the mean a little bit; they're moving it by a factor of five. That's how much money there is in these tails. This anomaly, these kind of rare events or outliers out here, are shifting the whole expected value of this distribution. So the median, the middle of the distribution, is much more robust to those few outliers on the other side.

So I'm going to write out what this means. The middle of the distribution is X such that the cumulative density function equals 1/2—literally half of the probability is left and half of the probability is to the right. The mode is the most likely; it's literally the X such that my probability density of X is maximized. You could write this as the argmax of P(X) or F(x); that's so fine. And the mean or average is this expected value of x, this expectation of X that we're calling μ here. Okay, good.

Um, that was a lot. That's a lot of information. That's probably all I want to tell you. Um, essentially the expected value is a very useful quantity in probability and statistics; it is one of the most important numbers that characterizes a distribution, but it's not the only important number. I also need to know the variance and the higher-order moments. It is highly sensitive to outliers, so if you think you have outliers or rare events, the median might be a more robust choice. Um, but expectation is easy to calculate, and it also will converge to the—the sample mean will converge to the expected value um in statistics in the large n limit. That's the law of large numbers, and we'll prove that later. Okay, thank you.