Transcription
Hello and welcome to this introduction to Bayesian statistics. My name is Woody, and together in this course, we're going to go on a journey exploring Bayesian statistics from scratch. This course is for anyone interested in probability and statistics, whether you're a student who wants to know more about probability theory, or if you're interested in using this professionally as a data scientist or a programmer.
We're going to begin by asking some pretty profound questions, like what probability even means. We'll think about it in the context of simple events like dice rolls, but also one-off events like horse races. We'll then look at conditional probability and how it can help us solve some pretty unintuitive problems. We'll use things like tree diagrams and also consider distributions like the normal distribution. We then tackle Bayes' theorem head-on, trying to keep things as visual, colorful, and intuitive as possible throughout. Finally, we have a look at the puzzle that kicked off this whole thing 250 years ago, when Thomas Bayes proposed a puzzle involving a billiards table. I really hope you enjoy this course, and it helps you understand Bayesian probability deeply and intuitively, and that it helps you understand the world a little bit better.
Hello and welcome to this first lesson on Bayesian statistics. We're beginning with what might seem like an obvious question: what is probability? Now, it's worth remembering that probability is actually a relatively new area in mathematics. It was only really thought about in the 17th century in the way that we think about it now. So, what I want to do in this lesson is establish how we can make sense of what it means, for example, to say the probability of, say, rolling a dice and rolling a six is one and six. So, we're going to think about two different ways that you might characterize this, and only one of them is really going to be compatible with what we want to achieve in this course.
So, the first way we could think about this is from an objective standpoint. So, why, for example, do we say that the probability of rolling a six on a dice is one-sixth? Well, one way to characterize this is that we all think it's one-sixth on a fair dice, and that's because this reflects some objective reality. Now, one way to cash this out is from a frequentist standpoint. Let me explain what that means. Let's say we're considering dice. You might roll the dice once, twice, a million times, maybe even an infinite number of times. What you can then do is you could take a random sample, maybe even an infinite sample, from those rolls, and you could say, what proportion of those did a six land face up? And we know that as the number of rolls tends to infinity, the proportion is going to tend towards one-sixth. So, one way to explain what the probability is, is in terms of frequencies. We would say, if the event was to happen infinitely many times, what would the proportion be of the thing we're interested in? Okay, so what could be a different point of view? Well, a different way to think about it is a subjective approach, which is to say that each person has their own reasons for believing a certain probability. So, this subjective stance is the one that's going to form the foundation for what's going to become our Bayesian model.
So, which is it then? Is it the subjective Bayesians, or is it the objective frequentists? I think a lot of people's intuition at first is to say that the subjective one seems a little bit too wishy-washy, a little bit not set in stone enough. So, as you can probably guess, since you've signed up to do a course in Bayesian statistics, I think there are some very good reasons for taking a subjective Bayesian model. It's not subjective in the sense that you can think whatever you want, but it's objective in the sense that each person might have a different answer to the question. Now, I'm going to give you two reasons for thinking this: one is to attack the frequentist position, show that it doesn't work in all cases, and then a different situation, I'm going to show you a great reason for thinking the Bayesian subjective model is actually really good.
So, this frequentist model is perfectly good when we're thinking about dice. No problem. We could imagine rolling the dice many times. We could at least hypothesize, wrong, infinitely many times. But what about when we've got a specific one-off event? Take, for example, a horse race. Let's say it's the Grand National, and, uh, here's our horse, and if you'll excuse the pond, let's say his name is Bayes' Camp. So, what's the probability that Bayes' Camp is going to win? It's the sort of thing that anyone who's going to bet on the horse race wants to know. It's the sort of thing that statisticians want to know. Now, there are some pretty good reasons for thinking that the frequentist account doesn't really work here. So, here's what the frequentist is going to have to say about this horse race. We're going to have to hypothesize, uh, an infinite set of horse races. So, we're going to have to imagine this single horse race happening infinitely many times. Now, from this, we're going to have to draw out a random sample of these horse races, just like we might consider a random sample of dice being thrown, and maybe this sample is infinite. And then, what we're going to do with this is we're going to count how many times did Bayes' Camp win. [Music] And we're going to divide that by how many races there were, and whatever number that is, that's going to tell us the probability that Bayes' Camp will win the Grand National.
Now, a lot of people think there's quite a lot wrong with this model for two reasons. The first is, what does it even mean to imagine an infinite set of a single event like the Grand National happening? It's not something that can happen more than once. It's pure imagination to speculate all of these races. And then, what does it even mean to draw a random sample from a hypothetical set? So, it seems like there are some pretty good reasons for thinking that the frequentist model is no good in situations that are just one-off events.
But let's look at some reasons why the Bayesian model might actually be really good. So, suppose I told you that my partner was expecting a baby, and we didn't know if it was going to be a boy or a girl. And just for the sake of simplicity here, we're going to ignore some biology and assume that everyone is either male or female. So, what can we say? Well, here's Alan, and he's my uncle. Now, Alan thinks that the probability that it's the child will be male is 50. Now, he has no reason to think it should be male, should be female, so he's going with 50. So, that seems a pretty reasonable thing for him to say. Would we say that this probability is correct? I think a lot of people would.
Now, here's our friend Anna. Now, Anna is a doctor, and Anna has just written an article for the World Health Organization showing that in my area, 51.2 percent of all children born are male. So, is Anna's view more correct than Alan's? She certainly has more information. But is it fair to say that she's right and he's wrong? It would be unfair, I think, on Alan to say this. Anna has more information, and no doubt, if Alan read her article, he might change his mind. But as things stand, it would be irrational for Alan to say 51.2 because he has no reason for thinking that the probability is higher than 50%.
Now, let's throw another character into the mix here. Here's Sarah. Now, Sarah is our midwife, and Sarah has just performed a scan. Now, the scan isn't 100% accurate, but it looks to her like it's male. She thinks the probability the child is male is 95%. So, what do we say? Now, is it right to say that Sarah's probability is correct, and Anna and Alan's is incorrect? Most people's intuition is that this is not the right thing to say. Sarah certainly has more information. But if Alan was to suddenly say, "No, I think it's 95% likely the child will be male," without that information, that would be irrational.
So, the Bayesian approach is to say that actually, all three might be correct. And really, the probabilities represent each individual's degree of belief, or a measure of their uncertainty. That's the sense in which probability is subjective. It's not that you can think whatever you want. So, Sarah can't look at her scan and say she still thinks it's 50. That would be irrational. Alan can't, in the absence of any other evidence, say he thinks it's 95% likely to be male. That would be irrational as well. Each person has a certain amount of evidence which forms their degree of belief, or the measure of their uncertainty. So, that's the Bayesian approach to this. So, degrees of belief are going to form the underpinning of everything that we're looking at in this course. I hope you found that first lesson interesting, and I hope it gave you a framework with which to think about probability for the rest of the course. See you in the next lesson.
We're now going to look at conditional probability, and we're going to begin this by looking at a really simple example, but one that helps build some intuitions which will help us build up to looking at Bayes' theorem in full later on in the course. So, let's start with a super simple and kind of silly case. So, we've got these 12 characters here, and there are loads of different probability questions we could ask if we were thinking about randomly selecting one of them. So, we could categorize these guys in loads of different ways, but one way we could do it is according to whether they're a dog and whether they wear glasses. So, that guy in the middle is the only one who is both, who both a dog and wears glasses, and we've got some dogs who don't wear glasses, and some non-dogs who do wear glasses, and so on.
Now, that we've got this visualized, we can ask some probability questions about it, ranging from the really, really simple, such as, "What's the probability that a randomly selected character is a dog?" And there's no tricks here, really. It's just there are three dogs out of 12 characters, so three out of 12. To a slightly more complex question, such as, "What is the probability that a randomly selected character is a dog given that they wear glasses?" Now, for this, we need to think about conditionality. So, the condition here is that the character is wearing glasses, which means we're going to restrict ourselves to only thinking about these five guys who wear glasses. So, we're zooming in on that, ignoring everything else. So, given that they wear glasses means we are looking at a probability out of five. So, there are only five characters who wear glasses, and the question is, how many of them are dogs? And the answer is, of course, one. So, one out of five is the probability that a character is a dog given that they wear glasses.
Okay, so far so simple. Um, how can we formalize what we've just done and make a general rule about it? Well, in this example, we're just counting, and really, we want our rule to relate to probabilities, and that's what we're going to practice actually in the next video, but for now, we can say that the one guy in the middle, that guy who represents being a dog and wearing glasses, that region is the can represent the probability of a character being a dog and wearing glasses. So, that region there can represent that probability. So, that's the probability of dog and glasses. Now, the whole, uh, the whole region is obviously the probability of a character wearing glasses, um, since it was that was what was the condition. So, formalized in this way, we've now got it wrapped up in terms of probabilities, which is going to be useful for us later on.
Now, I'll call this for now, "baby Bayes' theorem." It's most of the way to formulating the full Bayes' theorem, but it's not quite there yet, but it's really useful. We can already solve some pretty complicated stuff with it. Um, just a little note on the letters that we use. Typically, I'll be using H and E rather than, you know, dog and glasses and D and G. And the reason for that is that we're often thinking about the probability that a hypothesis is true given some evidence. So, in general, we're going to be using H's and E's in most cases.
So, now that we've derived this baby Bayes' theorem, let's apply it directly to an example that uses probabilities and not just counting. It's a pretty simple example, and I do encourage you to pause the video and try to solve it yourself, just to check you understand what's going on here. All right, so in order to work this out, we're trying to work out the probability of H given E. So, it's often easiest to begin in these cases with the probability of, uh, of E. So, what's been given? So, um, what is the probability of E? Well, it's not just 0.2, it's the entire bubble. So, it's everything included in it, so 0.2 plus 0.15, so we've got 0.35 here. And the probability of H and E, well, that's just this intersection here, so that's 0.15. There's our answer, and you can do this in your head if you want, or you can use a calculator, but it's 3/7. So, there's a pretty simple example of what's the probability of H given E when we've got the information presented to us as probabilities in a Venn diagram.
Now, let's take a look at a much harder example here. So, in this one, the challenge at the beginning is to actually just visualize this, which will involve drawing a Venn diagram for ourselves. So, here's the question: Amira and Jane are sometimes late for school. Seventy percent of the time, neither of them is late. Amira is late 20% of the time, while Jane is late 25% of the time. Last Monday, Jane was late. Find the probability that Amira was late.
All right, so, uh, the first thing to do is just give a bit of terminology or use a bit of terminology to formalize what's what we've actually been told. So, okay, what's our evidence here? Well, our evidence is that last Monday, Jane was late. So, let's say that, uh, E for us is the represents Jane, Jane being late. So, Jane is late, and our hypothesis, the thing that we are interested in testing, is whether Amira is late. So, H is going to be Amira here's late. All right, let's now try to write down using this notation some of the information we've been given. So, 70% of the time, neither of them is late. So, there are actually different ways to write down that something hasn't happened. Um, you will often see, for example, if we wanted to write that Jane isn't late, um, you'll sometimes see it written as E prime or E dash, like that. You'll sometimes see it written as E complement. You'll sometimes see a sort of hook E like this. I'm going to use this one, the E dash. So, what we've been told is the probability that Jane isn't late, so E dash, and Amira isn't late, so that's H dash, is 0.7. And we've also been told that the probability that Amira is late, so that's the probability of H, is 20, so that's 0.2. And we've been told that the probability that Jane is late, so the probability of E, is 0.25.
Now, can we represent this all on a Venn diagram? So, really, this is the main challenge. So, so this bubble here can represent H, so that's Amira being late, and this bubble here can represent E, so that's our evidence, and that's Jane being late. Now, the first thing we can do is put a 0.7 on the outside, because that's the condition that neither of them are late. Next, what we can, what can we say? Well, I know that the entirety of this H bubble is 0.2, because the probability that Amira is late is 0.2. So, this entire bubble here has got to sum to 0.2. So, if all of this bubble sums to 0.2, and we've got 0.7 here, the only part unaccounted for is this section here. Now, if I've got 0.2 here, let's just keep track of this, and 0.7 on the outside, that takes us to 0.9. So, the only unaccounted for region is this, and it's got to be 0.1, because all of the probabilities need to sum to 1. Well, I now can work out the rest quite easily, because I know the probability of E is 0.25. Now, I've got 0.1 here, so there is 0.15 remaining. And I know that probability of H is 0.2. Now, I've got 0.15 here, and so 0.05 remains. Okay, so there's our picture. We've got the, the information represented in the Venn diagram. We're now in a position to actually go, go for the main question. So, what is the probability that Amira is late, H, given the information that Jane was late, given our evidence that Jane was late? Well, using our baby Bayes' theorem, it's the probability is the intersection, so that's the probability that they are both late, which we now know, 0.1, divided by the probability of E, divided by our evidence, so that's the probability that Jane is late, and that's all of this here, so that's 0.25, 0.2, and that's going to simplify down to three-fifths, or alternatively, 60%. So, we now know that, um, when Jane is late, Amira has a 60% chance of being late. So, most of the time, she's late, 20%, or rather, uh, in general, her probability of being late is 20%, but if Jane is late, there is a 60% chance that Amira will be late as well. So, one way to think about that is that Jane being late, if we learn that Jane is late, we update our idea about what the probability that Amira is late. In fact, Jane being late seems to make Amira more likely to be late. And it's that kind of thinking which is going to help us sort out Bayes' theorem later on, this idea of updating our probability given some new evidence.
Let's now have a look at an example where we apply our theorem to examples that can't easily be solved using simple Venn diagrams. So, here's a question: Tanya loves to play tennis, but especially so when the weather is good. When it is sunny, the probability that she plays tennis is 80%. When it is not sunny, the probability is just 35%. There is a 60% chance that it is sunny on any given day. Last Saturday, she played tennis. What is the probability that it was sunny last Saturday?
Okay, so obviously, we think that there is a 60% chance that it is sunny on any given day normally, but we've got some new evidence, which is that she played tennis, and that's going to be relevant and could lead us to update our opinion of what the probability was. Let's try to model this. Let's first decide what's our hypothesis and what's our evidence. So, the hypothesis, the thing that we are investigating, is whether or not it was sunny last Saturday. Our evidence is that Tanya played tennis last Saturday. So, can we represent this on a tree diagram? Well, either it was sunny last Saturday or it wasn't. Now, the probability that it was sunny is 60%, that's, uh, before we know any other evidence. So, this, this diagram is going to represent a full picture of what could happen. Well, therefore, there's a 40% chance, or 0.4 probability, that it was not sunny last Saturday. Now, if it was sunny last Saturday, we know that she might have played tennis. We're calling that E with probability 80%, because when it's sunny, there's an 80% chance that she plays tennis. There's also a 20% chance, therefore, that she does not play tennis, so E dash. Um, and we could also look at what about when it was not sunny? So, the branch below, well, she might have played tennis. The probability of that is 35%, and that means the probability that she didn't play tennis that day is 65%.
Now that we've got our tree diagram, we can apply our baby Bayes' theorem. So, let's try to work out the numerator first. So, probability of, um, H and E. So, that's the probability that it was sunny last Saturday and she played tennis. Well, hopefully you remember how to work with tree diagrams. We're just multiplying the branches. The probability that it was sunny and she played tennis: 0.6 times 0.8, so 0.48. Now, what about the numerator? It's a little bit trickier to work out here. What's the probability of our evidence in general? So, what's the probability that she just played tennis? Well, if she played tennis, either it was, uh, sunny and she played tennis, and we already know the answer to that, or it might not have been sunny and she played tennis, which I'm representing by the blue branch of that tree. So, that's the probability that E is true, that's the probability that she played tennis. Either it was sunny and she played tennis, or it wasn't sunny and she played tennis. So, we can calculate these, sum them together, and we get 0.62 as our probability that she played tennis.
Right, we now have the numerator and the denominator. So, it's kind of easy to work it out from here. We'll just plug it into our formula above. So, 0.48 is our probability that it was sunny and she played tennis, and 0.62 is our probability that she played tennis. And therefore, there was a 77% chance that it was sunny, given that she played tennis. So, once again, you can see that this theorem has helped us update our idea of what the probability is. We think in general that there is a 60% chance that it's sunny, but if you learn that Tanya played tennis on a certain day, you can update your idea about what the probability is. It's now 77% likely that she played, sorry, that it was sunny that day. So, that's what Bayes' theorem is ultimately going to do. It's going to help us update our idea of what a probability is, given some information.
Let's now have a look at an example where you can put this into action. So, do pause the video and try this before me if you're feeling confident with it. All right, so what's the information given here? Well, during a, during recessions, there is a 40% chance that Tom will lose his job. Otherwise, there is a 5% chance. On any given year, there is a 10% chance of a recession, and last year Tom lost his job. Find the probability that there was a recession.
So, uh, as usual, let's just kind of formalize this by saying the thing we're investigating, our hypothesis, is was there a recession? So, H is that there was a recession. [Music] And our evidence that we can bring to bear on this is that Tom lost his job. Okay, so let's try to represent this. So, we know that on any given year normally, either there is a recession or there's not. So, either there is a recession, and the chance of that is 10%, so 0.1, or there is not a recession, which is 0.9. So, that's like our, that's our sense of what it is before we have any evidence. It's actually called a prior. It's like our prior, uh, thinking. More on that later. Um, now, if there is a recession, so if that hypothesis is true, then the probability that Tom loses his job is 40%. Now, if there is a recession, there is therefore a 60% chance that Tom does not lose his job. If there isn't a recession, well, uh, not lose his job. Now, if there isn't a recession, then Tom might lose his job, but it's very unlikely, 0.05. And if there isn't a recession, we therefore know that there's a 95% chance that he will lose his job. Okay, so we're looking for the probability that, uh, H is true, there was a recession, and he lost his job, divided by the probability that he lost his job. Okay, so probability that there was a recession and he lost his job, that's simple. That's just this branch here, so that's 0.1 times 0.4. And the probability of E, so that's the probability he lost his job, so that's either that there was a recession and he lost his job, or that there wasn't a recession and he lost his job. So, that is going to be that there was a recession and he lost his job, plus the probability that there was not a recession and he lost his job. Now, we can fill in all of these values. So, up here, it's 0.04. And down here, it's 0.04 plus 0.045. So, in total, we have 0.04 divided by 0.085. And our final probability, depending, well, there are, see, lots of different ways you could write this down, but let's just express it in one go as a percentage. We get 47%.
Okay, so that's quite a big jump, isn't it? So, last year, we needed to find the probability that there was a recession. Well, we know that there's normally a 10% chance of a recession, so this is an unlikely event. But if you've learned that Tom has lost his job, and you haven't learned anything else, well, in that case, you know that you've got to update your idea of what the probability is, and it's now 47% likely that there was a recession last year.
Let's have a look at how what we've learned about conditional probability applies to normal distributions. All right, so here's a little girl who loves to run, and let's say that for girls her age, the time it takes them to run 100 meters has a normal distribution with these properties. Now, don't worry if you don't know about normal distributions. The terminology looks a little strange, but all it means is that the average time is 15.5 seconds, and that 1.2 squared, MIP, that all that means is that most girls will run it within 1.2 seconds of that average.
Now, here's the question, or an initial question that we want to answer: What's the probability that this girl runs 100 meters in less than 13.5 seconds? Well, to answer that, we, we just use a computer or a calculator, and it turns out at 0.048, so 4.8%. And that's modeled by areas under the curves. But suppose that we had some more information. So, suppose that we also knew that to get into the school running team, you need to be able to run 100 meters in less than 14 seconds. Okay? And let's say that we also know that this girl, whose name is Sophia, is in the school running team. So, we've got some new evidence to bear. We know that in general, the probability that she can run, uh, 100 meters in less than 13.5 seconds is about 4.8%. How can we change that if we know she's in the school running team?
Well, we can use our baby Bayes' theorem. So, our hypothesis is that Sophia runs 100 meters in less than 13.5 seconds, and our evidence is that she is in the school running team. So, let's see how this is going to work. So, we're looking for the probability of H given E, so the probability that our hypothesis is true given the evidence. And we've got our formula for this. So, can we investigate these things? Well, the numerator is this slightly weird expression. It's the probability that she runs 100 meters in less than 13.5 seconds and that she's in the school running team. But being in the school running team means that she can run it in less than 14 seconds, because that's the condition to be in the school running team. So, both of those have to be true. And we're dividing by the probability that she's in the school running team, which is just the probability that she can run it in a time that's less than 14 seconds. Okay, this numerator needs sorting out. What does it mean for her time to be less than 13.5 and less than 14? Well, if her time is less than 13.5, and less than 14, it's just less than 13.5. There's, there's nothing to say there. If it's less than 13.5, then we already know it's less than 14. We don't need to bother with that. And we're dividing by that by the probability that it's less than 14. Now, with probabilities represented as areas with normal distributions, we can say that the numerator is this area, the probability that her time is less than 14 seconds, which you can compute using a calculator or computer as just over 10%. And the probability that she can run less than, uh, 13.5 seconds is our 4.8% from before. Now, do that calculation, and you get around 45%.
So, again, the evidence made us significantly update our thoughts about whether Sophia can run 100 meters in less than 13.5 seconds. We thought it was about 4.8% before. Now that we know that she's in the school running team, we've got some evidence which forces us to update our view, and we've now got to say that the probability that she can run in less than 13.5 seconds is 45%.
We're now going to have a look at some really counter-intuitive results that can follow from a good understanding from conditional probability. So, let's think about IQ. Now, IQ is famously normally distributed. It's got a mean of a hundred, so that's the average IQ score, and a standard deviation of 15, meaning that most people have an IQ within 15 points of a hundred. Now, uh, let's say that to be counted as a genius, you need an IQ of above 171. And I picked this because it actually turns out that's roughly one in a million. So, other people might have some other definition of a genius, but let's just say that this kind of one-in-a-million IQ, above 171, that's the definition of a genius.
Now, suppose there was a pill that you could take, and this pill, um, all it did was it shifted the average IQ for those who took it up by two points. A tiny little improvement in average IQ for the people who take the pill. Could we answer this question: How much more likely is it that your child will be a genius if they take this pill? I think most people's intuition is it's not going to have a particularly big effect on the chance. Adding two points on average to their chance of being a genius feels like it's not going to have a significant effect on making them more likely to be a genius. It's not going to hurt, but it seems like it won't help too much. Let's investigate this claim.
So, what can we say so far? Well, we could imagine conducting, uh, an experiment. An immoral experiment, but it's a thought experiment, so don't worry. We're going to take a large group of children, and we're going to give half of them the pill. And this red pill, uh, is going to shift the average IQ, uh, for those children up by two points. We'll let them grow up, and then we will just see which of them turned out to be geniuses. This, this thought experiment will help us answer this question. All right, so let's investigate.
So, here's the conditional probability claim. So, what is the probability that they took the pill, given that they are a genius? That's actually the question we want to answer. So, we see, given they're a genius, so we look at all the kids who turn out to grow up to be geniuses, and we say, how many of them actually took this red pill? All right, so let's use our baby Bayes' theorem to answer this question. Our hypothesis is that they took the red pill as a child. That's the thing we're interested in. And our evidence is that they are a genius. All right, let's work through the numbers. Now, oh, by the way, I should say that I normally like to represent these things graphically, you know, someone pointing out where you need to be to be a genius, but to be honest, it is so far up that way that we're just not going to bother with, uh, graphical representations, because the regions are just too small. All right, let's work through the numbers. All right, so we know that we, what we need to do first is we need to find the probability of H and E. So, that's the probability that they took the red pill and they became a genius. And then we're going to divide that by the probability that they are a genius in general. Okay, so the probability they took the red pill and were genius. Well, half of the kids were given that red pill, and then of those who are given the red pill, what's the probability that they became a genius? Well, remember the distribution for those guys. So, our kind of IQ plus had a normal distribution with a mean of 102 and a standard deviation of 15. And really, what we're doing is we're answering the question, what's the probability that this kind of IQ plus group is greater than 171? And that turns out to be pretty small. So, it's 0.000000211. So, it's a little over two in a million. And let's now think about what's the probability that in general a child is a genius. Well, it's either that they took the pill and were a genius, so that's the same as what we had above, okay, 0.000000211, or that they didn't take the pill, the red pill, and that was a probability of a half that they were not one of the chosen ones. And then the probability that if they don't take the pill, they become a genius. Remember, that was one in a million. Um, actually, I lied. It's just fractionally above one in a million. It's, um, there we go, 1.1 out of a million. All right, so crunch all of these numbers, uh, cancel the halves, if you like, or don't bother. We end up with 0.6. So, okay, so 66% of, uh, the kids who are geniuses took the pill. That's what we've worked out here. Okay, the probability that they've taken the pill, given they're a genius, is 66%.
Now, think about this for a moment. What this means is that out of all of those geniuses, okay, out in our experiment, out of all of those geniuses, okay, so here are our geniuses living their very clever lives. We know that the number of the of these people who took that red pill is about double the number who didn't. Okay? So, like, if I think of which of the ones who took that red pill, it's going to be out of this sample, maybe these guys. So, in your room of geniuses, twice as many of them would have taken that red pill as didn't. Think about that for a second. This was a pill that bumped up your average IQ, your expected IQ, by two points. Just two points. And to be a genius, you need an IQ of above 171. And yet, it doubles your chance of being a genius. Now, I know this is a bit of a contrived example, but I use it to point out the fact that using conditional probability can help you understand some really counter-intuitive results. For example, that in the tails of distributions, tiny changes in the mean can have a really big effect on probabilities. So, just graphically, we can see that this shift is really, really small, and yet it has this really big effect.
Now, let's imagine there was a different pill, just a completely different example. Um, and this pill, instead of shifting up the chance of a child, uh, the expected IQ from 100 to 102, it just changes the standard deviation from 15 to 20, which means most of the kids have an IQ that's within 20. Now, graphically, that looks like this. It just means a little bit more wild cards, more likely to be a bit cleverer, more likely to be a bit less clever. Now, what effect does that have? Well, in this case, it's actually pretty astonishing. I'm not going to go through all the calculations this time, but they follow exactly along the same lines as before. So, this one here is actually pretty astonishing. So, taking this pill, which doesn't make you on average more clever, it just makes you more, more kind of more spread out in terms of what your ability might be, um, this one actually, we get the probability of having taken a pill, given that you end up a genius, is an astonishing 99.4%. Okay, that does not mean that if you take the pill, there's a 99.4% chance of being a genius. It means that if you look at all of the geniuses in our study, and then say, what is the probability that a randomly selected one of those people has taken this pill? It's 99.4%. So, that's giving you a whopping 174 times, uh, improvement in your chance of being a genius from taking this pill.
We now have enough background information and intuitions developed to tackle Bayes' theorem head-on, and we're going to do it looking at one of the most famous examples of Bayes' theorem in action. So, disease X affects 10% of people where you live. Now, you've experienced some symptoms, you go to the doctor, she performs a test, and the test says you have disease X. Now, this test that she does is not 100% reliable. In fact, it gives the correct result 80% of the time. Now, you want to know, what is the chance that you actually have the disease? Most people's intuition is it's 80%. You thought it was 10% previously, that was your prior understanding. But now you've had the test, which tells you you have it, and that test is 80% reliable. Most people would update their view and say they now are 80% confident they have it. Let's check if that's correct.
So, we're going to do it by imagining a representative sample of the society. Now, we know that 10% of the people have the disease, and they're randomly distributed. Now, here are the 10 people, but let's place them down here just so that we can have a good picture of what's going on. Now, we also know that 20% of the people tested get told the wrong answer. So, I've drawn a dotted line there. The people to the right of that, which is 20% of our sample, they got told the wrong thing. So, let's think about what these different groups are. These guys here, they're healthy, and they got told that they don't have it, which is correct. These people here, they're healthy, but they got incorrectly told that they have the disease. Let's color them red just to make things easier. These people down here, well, they have the disease, but unfortunately, the test told them they didn't. Let's color them green. Okay, we can label things up just to make things a bit easier. So, here are the healthy people. Here are the people who have disease X. Here are the people who got told the correct result in their test. And here are the people who all got told an incorrect result.
So, the test says that you have disease X. Which of these people might you be? Well, you're not these people because they are healthy and they got told they're healthy. So, you're not one of them. If they, if you got told you have it, you're also not these people because they are people who have the disease but got told incorrectly they don't. So, that means that you must be either these yellow guys down here or these red guys up here. So, the question is, which one, in which group are you? What's the probability? Well, there are eight people who have the disease and got told they have it, and there are 18 people who don't have the disease who got told they have it. So, what's the probability that you have the disease? Well, it's 8 out of 8 plus 18, so that's 31.
So, let's just go back to the original question to see how surprising this is. The doctor does the test, it says you have the disease. You know this disease affects 10% of the people where you live. You know this test is 80% reliable. But in fact, there is only a 31% chance that you have the disease. So, our prior understanding of having the, the illness, which was 10%, got updated by this mostly reliable test, and now we know the chance is 31%. You still probably don't have the disease.
Let's just piece things together again and see if we can formalize. So, here was our calculation, and you can see that, uh, the reason why you still probably don't have it is there are simply more people who don't have the disease but got told that they do than people who have it and got told they did. Now, let's see if we can formalize this to see where these things come from. Start with these yellow people. How did we determine there were eight of those? Well, we took the number of people in our sample and we multiplied that by the probability of having the disease, which was 10%, so that shrunk things down to those 10. And then we multiply by the probability of having a correct test, or 0.8, and that shrunk it down to 8. And just, just to be clear, that P of T given X, that's the probability of having a positive test given that you have the disease. Now, what about our, uh, so those are the yellow people. Now, what about our red people here? Where did they come from? Well, to calculate them, we took the number of people in our sample, we multiplied by the probability of not having the disease, so that was 0.9. Then we multiplied by the probability of getting a positive test result, given that you don't have the disease, which was 0.2. That gave us that group of 18 there. So, now we have an expression for the yellow people and the orange people. Let's try to put together our calculation. So, we took the number of people times the probability of having the disease times the probability of getting a positive test result given the disease. That was our eight people, and we divided by those eight people, the yellow ones, plus the red ones. Okay, that looks like a horrible mess. Now, let's try to clean it up. Notice that we can just cancel the number of people, so it's not relevant how many people are actually in our, in our trial. Okay, algebra time. First thing to do is, uh, just clean up our, our terminology to make it consistent with what we've been doing before. So, rather than X, let's call it H, our hypothesis that we have the disease, and rather than T for test, let's use E for evidence. And now, just by convention, we tend to write it this way round, and that, ladies and gentlemen, is Bayes' theorem.
So, now that we've got this theorem, let's relate it back to our picture, just so we're crystal clear about where each bit comes from. This numerator, well, that represents these yellow guys down here. So, they're the ones who have the disease, um, and who they've been told that they have it. So, it's the probability of, um, the evidence given the hypothesis, as the probability of having a positive test result given that you have the disease, multiplied by the probability of having the disease. And on our denominator, it's those yellow guys again, plus the people who got told they have the disease even though they don't have it. So, that's the probability of the evidence, so probability of a positive test result given that you don't have the disease, multiplied by the probability of not having the disease.
Let's now have a look at an example that will help build some detail into our Bayes' theorem and even more understanding. The example we're going to look at is one that was made famous by this guy, Daniel Kahneman, and he talks about it in his book, "Thinking, Fast and Slow," and it's something that he developed many years before with his collaborator, Amos Tversky. So, here's the situation we need to consider. Here is Steve, and Steve is a very shy, is very shy and withdrawn, invariably helpful, but with very little interest in people or in the world of reality. A meek and tidy soul, he has a need for order and structure and a passion for detail. Now, here's the question that Kahneman poses for us: Do you think he's more likely to be a farmer or a librarian? Well, it's obvious, right? He's way more likely to be a librarian. This is everyone's intuition. Let's see if we can use Bayes' theorem to, uh, to undermine this, to actually, uh, figure out what's going on in more detail.
So, the thing that we all forget to do is to consider how many farmers and how many librarians there actually are. Now, the precise numbers don't matter, but there are way more farmers in the world than librarians. In fact, in pretty much any country wherever Steve's from, he's there are probably more farmers than librarians. In my example, there are 10 times more. So, here's the, here's the evidence about Steve: he's shy and withdrawn, and he's helpful, and he's tidy, and he's meek, and he likes order and structure. So, we could say, well, what fraction of farmers, well, let's say it's 30%. Let's say it's a fairly low percentage of farmers that meet this criteria. Now, it's going to be much more likely among librarians. So, let's say that 90% of librarians meet this condition. Well, Steve must be either one of the green farmers or one of the yellow librarians, okay, because they're the people who meet.
That condition. Well, we now know that Steve must be in one of these groups. So, we could maybe look at the ratio of these groups. What's the ratio of the farmers who are meek and tidy and shy to the librarians who are meek and tidy and shy? Well, what is it? Well, it's 30 to nine in my example, which is 77 to 23. So, is he more likely to be a farmer or a librarian? Much more likely to be a farmer. And this goes against our intuitions because the thing that we forget is how many farmers there are.
So, let's see if we can phrase this all in terms of Bayes' theorem. So, our hypothesis, the thing that we are investigating, maybe is Steve is Steve a librarian? And ultimately, wants to know, given the information, what's the probability that he's a librarian? Our evidence is that Steve is shy and tidy and like structure, all of that stuff. So, here's the question: What's the probability that he is a librarian, given all of that evidence? Well, how do we calculate it? Well, we first take the probability that he is a librarian. That's our prior probability. That's what we thought the probability is regardless of any evidence. And then we multiply that by the probability of him, uh, being shy and tidy, given that he was a librarian. That shrunk it down from from 10 people down to nine people. And then we divided that by the total number of people who could be shy and tidy. So, that's these same, uh, shy and tidy librarians, but also the shy and tidy farmers. And we found them by taking the probability that someone was a farmer or not a librarian and multiplying that by the probability of them being shy and tidy, given that they are not a librarian.
Okay, let's put some numbers in. So, the probability of, uh, Steve being a librarian was 10 out of 110. There are 110 people in the sample, and 10 of them are librarians. And then we multiplied by the probability of being shy and tidy, given that they're librarians. That shrunk that down by a factor of 90%. So, nine, nine over ten times by that probability of being a librarian. We divided by the same amount again, plus all of the shy and tidy farmers. So, we took the probability of a randomly selected person being a farmer, 100 out of 110, and we scaled that down by the probability of those farmers being shy and tidy. So, we multiplied by the probability of being shy and tidy, given that you are not a librarian. That is, given that you're a farmer. So, those are those are the numbers. That's how we, that's how we can apply this to actually plug in some numbers and work some things out. And that's where we get our 23 from. So, it's, it's much lower than people think. People hear that information, they think, yeah, I'm pretty sure, I'm pretty sure that Steve's a librarian. But actually, he's probably not. He's about three times as likely to be a farmer.
Just a word on some of the terminology that's used in relation to Bayes' theorem. And I've been using this a little bit as I go, but let's just see how each of these parts is actually named formally. So, this bit here, probability that our hypothesis is true, this is called the prior. So, this is what we think the probability is prior to any new evidence we have. So, in the case of our librarians and farmers, you just think, what's the probability a randomly selected person is a librarian? Well, in our sample, it was 10 out of 110. That's before we have any information about the situation. That's what we think the probability is.
Um, now, in contrast to that, this part, the answer, is called the posterior. So, posterior meaning afterwards. So, after our evidence, what do we now think the probability is? So, there's this contrast of what do we think it was before we saw the evidence, the prior, what do we think it is after the evidence, the posterior? The other parts of this, well, this is called the likelihood. So, that's the probability of seeing this evidence, given that the hypothesis is true. And in the case of our librarians and farmers, this is essentially, what's the fraction of librarians who are meek and tidy? So, this was the 90 in that case. Finally, this, uh, the whole denominator is called the evidence. It's sometimes called the marginal likelihood. So, um, really, that, that whole expression could just be written as probability of E. Probability of the evidence. That's what we did in our baby Bayes' theorem. And it's just the general probability of seeing the evidence that we've seen. And, and just as it's split up into the chance of seeing the evidence if the hypothesis is true, and then also the probability of seeing the evidence if the hypothesis is not true. So, that's the kind of terminology that you'll see when you're working with Bayes' theorem. And often, it is easier to write in the shorthand of just probability of E. But there are situations where the only way you can work out the probability of E is to have it split up into these two groups. So, you'll sometimes see Bayes' theorem written like this, and you'll sometimes see it written in this shorthand.
Here, we're now going to look at a substantially more complicated example, but it's a really nice one. And it's the example given by Bayes himself in his writings about this when he was first developing this way of thinking. Now, I'm adapting this, uh, to make it a little bit more simple. And I'm following an adaptation done by by Eddie in his article, "What is Bayesian Statistics?" So, what we're going to do is we're going to imagine a game between Alice and Bob. Now, the game begins by this, a ball is rolled to a random position on the table. What this does, it divides the table into two sections. Alice and Bob then play the following game. So, balls are going to be rolled onto the table randomly. If they land on Alice's side, she gets a point. If they land on Bob's side, he gets a point. And the rules are, first to six wins. The only thing is, Alice and Bob, and let's pretend us, uh, we can't see the table. We don't know how it was divided in the first place. Okay? So, we're not sure where that blue line was. So, let's say that the game begins, and Alice gets the first point. The ball lands on her side. And then maybe Alice gets another point, and the ball lands there. So, Bob wins. And the game proceeds. Alice gets a point, Bob gets a point. Now, all they know is the points they're getting, not where the balls are landing. And Bob gets another point. Then Alice gets a point. And then Alice gets a point. So, let's pause at this point. What is the probability that Bob will win this game? And what should Bob think it is? What should Alice think the probability is? Since they don't know where the blue line is, they don't know where the other balls landed, all they have are the results. So, how on earth can we solve this problem?
Well, what's the probability that Bob gets a point on any given go? That's going to be a useful starting point for us to think about. So, we'll begin by actually thinking about this problem from a frequentist standpoint. If you remember way back in the first video of this course, we looked at the difference between frequentist methods and Bayesian methods. And this is a great example to contrast them and see how the Bayesian method can solve the problem more elegantly and actually more accurately. So, here's what the frequentists would do. The frequentist would have no problem answering this question. What's the probability that Bob gets a point? Well, he's got three of the points so far out of eight goes. So, that's what the probability is that Bob wins a point. Now, in order to win the game, he's going to need to win the next three points. So, the probability of that happening is 3/8 cubed. So, it's pretty simple for the frequentist. They can just carry out that calculation. And what we get is about 5.3%. All right. So, that's how the frequentist would think about it. And there's some pretty good reasons for thinking that this might work.
So, another way that we could think about it would be this. You could say, well, imagine that the probability that Bob gets a single point is X. Well, then you could say, what's the probability of seeing these kind of results? Well, out of a total of eight turns, what's the probability that Bob would get three of them? Well, um, anyone who's studied binomial probability will recognize this as a binomial probability question. So, the probability that the number of points that Bob has got is three, and the number of points that Alice has got is five, can be calculated using 8 choose 3. Now, don't worry if you've not seen this before. This is a binomial distribution for events where there are two possible outcomes with fixed probabilities. So, eight choose three ways that it could happen. And X is the probability that Bob wins any given point, that happened three times. And one minus X must be the probability that Bob doesn't win or Alice wins, and that happened five times. So, this here is going to give us a function that will give us the probability of finding ourselves at 5-3 to Alice for any given probability. And in fact, once we calculate the 8 choose 3 is 56, we've got this nice-looking polynomial. Now, we could graph this polynomial. And when we do so, we could say, well, what's the highest point on that? Well, that highest point turns out to be 3/8 as well. Now, all of this is just to suggest that the most likely explanation for finding ourselves at a position where Bob has three points and Alice has five points, the most likely explanation for that data is that the probability of Bob winning each point is 3/8. And if that's the case, the probability that Bob wins is 5.3%. It sounds compelling. That's the frequentist approach. Let's have a look at the Bayesian response.
So, what's the Bayesian going to say with regard to this game? Well, what is the probability that Bob wins? Well, let's use our notation that's familiar by now. Our hypothesis is that Bob wins the game. And our evidence is that the current score is five for Alice and three for Bob. So, let's try to apply Bayes' theorem to this. So, what do we know? Well, we know that the probability that Bob wins, given that Alice has five and Bob has three, is, and this is a direct application of of Bayes' theorem, it is the probability of seeing a score of five-three to Alice, given that Bob wins, multiplied by the probability that Bob wins, divided by our evidence, which is the probability of finding a score of five, Alice has five, Bob has three. Now, I'm going to try to give a sense of how we can use Bayes' theorem for this. Um, I am simplifying things a bit, but my aim here is to build some intuitions about how Bayes' theorem can be used in continuous distributions. So, we will skip over a few things, but hopefully, you'll leave with a sense of how we could extend Bayes' theorem further.
All right, let's start then with the denominator. So, what's the probability of seeing a score of five-three to Alice? Now, remember that we don't know where this blue line is. So, we don't know what the probability is that Bob wins any given game. Okay? It could be really big. It could be really small. It could be as or anything down to zero and anything up to one. So, in light of that, we're going to have to think, what's the probability of seeing a score of five-three in favor of Alice in general, across any given probability of Bob winning a single point? So, let's name that, and let's call that X. Let's say that X is the probability that Bob wins any given point. We saw from before that the probability of seeing this evidence is, uh, binomial probability, 8 choose 3 times X cubed times 1 minus X to the 5. Now, we want to know, what is the sum of all of these different possibilities for all values of X from 0 to 1? Now, if you've studied calculus, you'll know that if you have a continuous distribution and you want to know what the sum of all possible results is for infinitely many values, we can do that using calculus. So, what we're looking for is the integral of this function between 0 and 1. That is, finding the area under the graph will tell us what the probability is of seeing this evidence of Alice 5, Bob 3, for all possible values of X between zero and one. If you worked out that area using calculus or using a computer, you're going to find that it's one-ninth. So, what this tells us is that in general, there is a one-ninth chance of playing this game and at some stage in the game finding yourselves in the position of Alice 5, Bob 3.
Okay, let's move on now and have a look at this numerator. Okay, so this is the part where I will simplify some things, and that's just in order to aid intuitions. So, this first part, probability that A equals 5 and B equals 3, given that Bob wins. Well, can we come up with a function that could capture that? Well, we know what the function is. We know that the function that describes Alice having five and Bob having three is this binomial expression here, 8 choose 3. And then Alice has five points. Well, let's do Bob's first. Bob has three, so X happened three times. Alice has five, so that's 1 minus X, and that happened five times. And that's sort of independent of the fact that Bob wins for any given probability for Bob. Um, we know that this expression captures that. And I am simplifying, but hopefully, this makes some sense. Now, probability that Bob wins. Well, if Bob is going to go on and win from here, he will need to win the next three games because it's first to six, and he has three points. He's going to have to win all of them, which means we're going to need a further X cubed. So, that represents Bob winning, and Bob winning, and Bob winning. Now, this expression here, since we've got X cubed and X cubed, can simplify to 8 choose 3, X to the power of 6, and 1 minus X to the power of 5. So, just as before, in order to figure out what this probability is for all values of X, we'll use calculus. So, here is our function that represents the, uh, probability distribution of that numerator, that of the score being 5-3 to Alice, given that Bob wins, multiplied by Bob going on to win the next three points. So, in order to do that, it's the integral of this function between 0 and 1. And that's going to capture the area under the curve, the area under that function between 0 and 1. If you calculate this integral again, you can do that using using calculus or use a computer, you get 1 over 99.
So, now we know the numerator, and we know the denominator. It's just a question of plugging those values in. So, our numerator was 1 over 99. Our denominator was 1 over 9. That simplifies to 1/11th. So, that's what the Bayesian says the probability is that Bob wins. So, to contrast, the frequentist, who just looked at what happened and said, "Okay, so Bob's chance of winning each point is 3/8. The probability is 3/8 cubed." It's easy to do, but it gives us an answer of about 5%. Using this much more complicated Bayesian approach gives us 9%. It's almost twice as much. Now, I think everyone's response to this is to say, "Okay, but who's right?" Well, in the next video, I'm going to go through a simulation that hopefully can give some evidence that the Bayesian has got this one correct.
All right, so in this video, we're going to use good old Excel to test who was right. So, remember what the situation was. Um, given that we found ourselves at a position in the game where Alice was 5-3 up, what's the probability that Bob goes on to win? The frequentist said the answer was 5%. The Bayesian said the answer was 9%. I'm going to set up a simulation here where we're going to simulate thousands of games many times and see, uh, what actually happens. Okay, who's whose answer was closest to the to what happens in our simulations? That should give us some sense of who's correct. So, the first thing I want to do is just to say, well, what was Bob's chance of winning on any given game, on any given term? So, it's really, where did that initial ball land? So, that's Bob's chance. Oops, Bob's chance of winning any given point. And this is just a random number between zero and one. And I'm expressing that as a percentage here. So, let's just generate a few. So, there we go. And each time I click a button, I can just regenerate. So, those are some different probabilities for Bob winning a point.
Now, the next thing I want to know is, what's the, did it, did the game actually reach 5-3 to Alice? So, the first thing I want to know is, what's the probability of reaching Alice 5, Bob 3? Well, that's a binomial probability, which we discussed in a previous video. So, what's the probability? Well, uh, the number of times that Bob should have won is three. There were eight games if we got to 5-3. So, out of eight trials, the probability of Bob winning any of those was the number we generated here. And it wasn't cumulative. It's not the probability of winning anything up to three, it's winning exactly three. So, now we've got probabilities of the game reaching 5-3. So, you can see that when Bob was highly likely to win every point, it's very unlikely that we should find ourselves 5-3 to Alice. But in a situation where Bob is about 39% likely to win any given point, it's more likely that we reach 5-3 to Alice.
Now, did it actually happen? Did we actually reach Alice 5, Bob 3? Well, these are the probabilities of it happening. And a really nice way to just randomly see if it did happen is to say, if some new random number is less than what we got here, then yes, it did. Otherwise, no, it didn't. So, now I can see of these games, okay, here's a time where it did happen. So, Bob had a 40% chance of winning each point. Um, so the probability of the game reaching 5-3 was 28%, and it did actually happen this time. Okay. So, ultimately, we're only going to be interested in the games that did reach, um, 5-3 to Alice, like this one here. So, we kind of want to ignore all of the others, which we can do. So, what we want to know now is, if it did reach that point, what's the probability that Bob goes on to win? So, this is the probability that Bob goes on to win. So, the first thing to do is to say, if this is no, so if it didn't actually reach, um, 5-3 to Alice, then just do nothing. Leave the cell blank. [Music] Otherwise, what's the probability of Bob winning? Well, it's just whatever his probability of winning each point was cubed, because it needs to happen three remaining times. Well, this is blank because we never reached 5-3 there, and none of them did. But here we go. So, in this, in this simulation, Bob had a 71% chance of winning each point. So, the probability of it reaching 5-3 to Alice was only 4%, but it did actually happen in this simula, in this simulation. But then Bob did go on to win in that simulation. Um, all right. And now we want to know, did it actually happen? So, did Bob actually go on and win? Well, we do want to say that if this was blank, then we'll make this one blank as well, because we're not interested in the games that didn't reach 5-3. But otherwise, we can do a little random number check to say, if a randomly generated number is less than the number we see here, then Bob did go on to win. Otherwise, Alice went on to win. Okay. So, in this first game, let's just generate all of these. [Music] Let's wait until there are a few. Okay, so in, let's have one where there's, there's both. There we go. So, um, in all of these blank ones, these were games that never actually reached 5-3 to Alice. But in this game, Bob had a 70% chance of winning each point. The game did reach 5-3 to Alice. So, Bob had a 35% chance of going on to win, and he did. In this one, Bob had a 40% chance of winning each point. Um, the game did go on to reach 5-3 to Alice, but in that one, Bob failed to go on to win, and Alice won. So, now our question is, what's the trend then? If we were to do this loads and loads of times, how many of these games that reached 5-3 end up with Bob winning, and how many with Alice winning? Well, to do that, we need to generate more than how many we've got now. We've got 13. So, let's do that. Let's generate way more. [Music] Now, how many is enough? That's a pretty good question. Generally, 10,000. I mean, depending on what you're modeling, 10,000 simulations is a, it's a decent start. Anything less than 10,000, you're probably not going to get that reliable results. So, let's just generate absolutely loads of these games. So, poor old Alice and Bob are spending, you know, years and years playing this game. All right, so that should do. So, that's what, around 15,000, um, games. But obviously, the vast majority of them never actually reach 5-3. So, we have no result. What we're interested in is in this column. It's clearly more common for Alice to win. But what's the problem? What's the proportion of these games that Bob went on to win? Is it 5%, as the frequentist says it is, or is it 9%, as the Bayesian says? Well, we can count this way. So, the number of games that Bob goes on to win, I can say, count if in this column you see the word Bob. Okay, so Bob, out of all the games done, by 157 of them. And that number's going to change every time I do run another simulation. What about Alice? Well, I can say, count if you see the name Alice in this column. Okay, so Alice has won way more. But what's the probability of Bob winning? So, the probability of Bob winning, given that the game did reach Alice 5, Bob 3, is the number of games that Bob won, divided by the number of games, which is the number that Bob won plus the number that Alice won. All right, so what do we get? Eight. Let's convert it to a percentage. Make it easier. In fact, let's give it a little bit more accuracy. There we go. So, 8.2% that time. Let's simulate again. 9.1%, 8.6%, 9.2%, 7.8%. Okay, so it does vary, as we'd expect with the simulation. But does it seem to be varying around the frequentist answer of 5, or does it seem to be varying around the Bayesian answer of 9? It's pretty obvious that this is varying around the Bayesian answer of 9%. And what this suggests is that if I was to simulate maybe a million times, we would very reliably get 9% as an answer. So, that's called the law of large numbers, that the greater the number of simulations, the more this will tend towards the correct answer. So, running a simulation like this can give us some pretty compelling evidence that the Bayesians are the ones who've got this right. The Bayesians who said it was 9% are not the frequentist. So, there we go. Bayesians win, as I said they would at the start of the course. There we go. [Music] I really hope you've enjoyed this short introductory course in Bayesian statistics. I hope you've seen the beauty, the elegance, and the power of Bayes' theorem and seeing how it can help help you understand some really interesting, complicated things. I hope you go on to use Bayesian thinking to understand the world a little bit better.