Transcription
Welcome back. So today we're going to introduce one of the most useful ideas in all of probability, which is Bayes' theorem. This is used all across the board in statistics. Uh, practicing statisticians use Bayes' theorem all the time. This is a cornerstone of machine learning, inverse problems, and engineering inference problems. One of the most important ideas.
So we introduced this notion of a conditional probability that maybe I'm trying to compute the probability of event A happening, but I have some additional partial information about event B, some other event um that could have happened. And so I can update or improve my estimation of the probability of A happening given that I know that B also happened. So the probability of A given B is the probability that both things happened divided by the probability that event B happened. Um, this makes a lot of sense. Um, pretty pretty straightforward idea.
I'm going to write down one extra fact here I think uh we're going to need later: is that you can multiply these and say that the probability of A and B is probability of A given B times probability of B. This is just the multiplication law, but in statistics and in probability, often we want the opposite uh information. So maybe um A is, maybe B is some disease—whether or not you do or don't have cancer—and maybe A is a test result or some symptom that you present. So I could, you know, compute the probability of having that symptom given that I have cancer, but what's much more useful is: given that I have a positive test result or given that I have symptom A, what is the probability of an underlying cause like cancer? This is much much more useful, and this is called an inverse problem because I'm trying to compute the probability of something that's actually quite difficult to measure using uh some observable, some measurement that I actually can measure, this event A.
So Bayes' theorem is going to flip this on its head, and I want us to always be thinking: what can you measure? What do you have access to? Um, and what is an inverse problem or a forward problem? Because I mean I could always just switch the order of the letters, um and this becomes that, but that's circular. So you know, think about it this way: uh, what's the probability that I have cancer given that I have a positive test result or I have a symptom? Okay, that's the kind of thing you would want to estimate, and this one's hard to measure; this one is easy to measure, or expensive and cheap, or invasive and non-invasive. Okay.
And so Bayes' rule, I'm just going to state it; essentially I'm going to state it in in easy, medium, and advanced uh, is the following. So Bayes' theorem—maybe I'll write this in blue—says that the probability of B given A is equal to the probability of A given B times the probability of B divided by the probability of A. Now we'll we'll work through where this all comes from, how to do this in a minute, um and it essentially comes from the fact that probability of B given A and probability of A given B both have this term in common: this probability of A and B. So probability of A and B is probability of A given B times probability of B; it's also the probability of B given A times the probability of A. Okay, that's this is also true. I could literally just uh flip these two, and you know this would become probability of A and B divided by probability of A. Good. And so now I can rewrite this as um yeah, so I can set these two equal to each other, and I get the probability that B given A is all of this divided by the probability of A. Really really simple. Okay, so this is kind of a trivial outfall of of just the definition of a conditional probability and this basic multiplication uh law here. Okay, so you know this is true for any sets A and B, and this is true for any sets A and B. And so now if I have something that's hard to measure, B, I can do this inverse problem with the information that I do have access to. Okay.
Um, and these things have names in Bayesian inference, in the field of Bayesian statistics and Bayesian machine learning. So I'm just going to like remind you of what these things are called. This uh probability of B given A is um is essentially called your posterior, uh sometimes called your posterior distribution. The posterior uh probability of B is your prior. So again, if I'm saying like what is the chances that I have cancer if I don't know anything else, then I have the same chances of having cancer as anyone else with my basic approximate, you know, whatever demographic um age group, and that would be my prior is like a whatever one in whatever chance um population statistics. But then if I take a test or if I have a symptom, I can update my probability of having cancer given that extra information about A, and I do it using uh using this kind of this kind of update. So probability of A given B is my update; this is my prior; this is my best guess before I got this piece of information A; and this is my update; uh and then this is you know some normalization constant based on what's the probability of you know getting a positive test result period. And in its simplest form, this is Bayes' theorem. This is how you compute this inverse probability using things that I can calculate pretty easily and my prior distribution.
Now I'm going to have like a whole series of lectures on Bayes' theorem and Bayesian inference and Bayesian statistics um probably later, probably in the kind of Statistics module. But the reason this is called an update is because this can help me do all kinds of interesting things like um imagine I have a coin—actually I do have a coin somewhere here—I have a coin, and I'm going to start flipping this coin, and let's say I don't know anything about this coin. I don't know, you know, maybe I assume it's fair. Let's say I that's my assumption; my prior is that it's a fair coin, and you know the probability of heads is 50%, but let's say I flip this over and over and over again. I flip it 10 times, and I get tails 10 times in a row. That is new information that I can update my probability of it being a fair coin. And every time I gather new information, I take my prior and I update that probability, and then the next time I collect information, this becomes my prior for the next experiment. I gather that information and I update my probability; then it becomes my prior for the next experiment; I gather more information A, and I update my probability. I'm massively oversimplifying, but that's the basic idea of all of Bayesian statistics: is that I run sequential experiments, I gather data sequentially, and I have some initial guess, and I update that distribution or that probability or that that estimation using this new piece of information, this update that I got. That's a massive simplification, but that's the basic idea. Okay.
Um, so this is one way of writing it. This probability of A happening is sometimes also hard to compute, so I'm going to write another version of this. Let's uh let's write this kind of next version here. So now I'm going to say the probability of B given A is the probability of A given B times the probability of B; nothing is changing here, but now this probability of A, I'm going to use my law of total probability to say that this is the probability uh of A given B times the probability of B plus the probability of A given not B—A given we're going to call this B complement, that means not B—times the probability of B complement. That is is another uh formulation; probability of A, the law of total probability says that it is you know the probability of A given B times probability of B plus probability of A given not B times probability of not B. This is again kind of obvious, but maybe we don't know what probability of A happening is; we don't know the probability of of a positive test result. So I can compute it, you know, using the information I gather. And then a final generalization of this: if I have a bunch of disjoint sets B, a bunch of different disjoint sets, is that I can write the probability of um of B—let's say that I have you know a bunch of sets B that are covering Omega—the probability that event J happens given A is um the probability of A given this event B J times the probability of that event happening divided by, again, law of total probabilities: the probability of A is going to be the sum over all of these disjoint uh B's, probability of A given B times probability of B item J. Okay, so this is just three different levels of the same exact theorem. This is if I can compute the the probability of A, of my test result being positive. This is you know writing this probability using the law of total probability, and then this is a generalization if I have a bunch of different events B that I could be testing over. Okay, so realistically this is the one I mostly want you to be thinking about um is Bayes' theorem here. Okay, super super super super useful. Okay, and this is going to allow us to compute these inverse probabilities. Let's do some examples. I have two examples I want to show you that are going to make this really really concrete. Okay.
Um, maybe the first one I'll do is uh cancer screening. I think this is a really good example. I'm going to do it over here because this is where my notes are, uh and let me just draw a line. So the example I want to do first is cancer screening. So let's say that I have a test that is 99% accurate. I have a test A that's 99% accurate in detecting a rare form of cancer. So uh say my test is 99% uh accurate, but the disease I'm testing for, the cancer I'm testing for, is is fairly rare. Um, but the disease or the cancer is very rare is is uh very rare. So 0.001% of the population have this disease. Okay, so one in a thousand people have the disease; not 0.001%, I'm sorry, just 0.001—1 in a .1%— uh of the population have this disease, and the test is 99% accurate. Simple simple.
Um, let's now see what is the probability of actually having that disease given a positive test. So let's define our A's and our B's. Okay, so A is a positive test; I'm going to write that as positive; and B is I have the disease, the cancer, uh or not. So you know, again, disease or not disease. So the way we compute this, we want to compute the probability of having the disease given a positive test. Okay, so the probability of having the disease given a positive test—let's use Bayes' theorem here—actually we're going to use this second form of Bayes' theorem: it's the probability of having a positive test given the disease times the probability of disease okay divided by by this same probability: probability of positive given disease times probability of disease plus probability of positive given not having the disease times the probability of not having the disease. Good. Okay, good. Uh, and we can compute all of these things. So I want you to be thinking: could I compute this or this or this in this example? This happens to be the easiest form because I can compute all of these things. So let's just plug in the numbers. Okay, so the probability of having a positive test given the disease: it's 99% accurate, so this is 0.99. The probability of having the disease is 0.001; this is 0.001 divided by these things. So again, probability of positive test given the disease: 0.99; probability of having the disease: .001; plus the probability of the probability of a positive test given that you don't have the disease: this is only 1%; it's a very very accurate test, so this is only 0.01, but the probability of not having the disease is 999; 999 out of a thousand people don't have the disease. So you can multiply these out, just you know do this on your calculator, do this in Google, whatever, and you're going to find very quickly that this is um actually have the answer here: it's 0.0099 divided by 0.1098, which is about 0.09. So this is crazy. This means that even though this test was 99% accurate, because the disease is rare, if I have a positive test, the chance of me having the disease is only about 9%. That means 91% of the time it's a false positive, and this gives you some idea of why medical diagnostics and screening is so complicated and so challenging, because if you have rare diseases, testing for them can give tons of false positives, and so you can't necessarily make decisions based on one test alone. So you would actually do—this is actually called a screening test—if you have a rare disease, even if you have a pretty accurate test, typically if you test positive you are screened, and then you do some secondary test, another test, like maybe you actually do a biopsy, you actually grab a piece of tissue and look at it and measure it and see if it has cancer, a different more accurate test, because if you just treated people based on this information, 91% of the time you'd be treating healthy people for a disease that they don't have just because the disease is rare. Okay, so that's the kind of thing that Bayesian statistics can be super duper useful for, um and you need to be thinking about this all the time. You need to be thinking: what can I measure? What's the thing I actually want? What's the causal thing? The cancer causes the symptoms; the test score's not the other way around. And this one is hard to measure; it's invasive; it's expensive to measure; the test is easy and cheap and non-invasive, but it can sometimes, you know, give you misleading results. And in this case it points out that that sometimes you have to do screening followed by subsequent follow-on testing. Okay.
Bayes' theorem: super useful all over the place; we're going to use it all the time; it is synonymous with inverse problems and inference problems in statistics; um and we'll do examples; we'll start flipping random coins, and we'll start updating our priors of is that coin fair given this new information, and we'll build sequence; we'll build you know for loops in Python, and we'll start actually testing these with real data and statistics. That's going to be a bit later, um but I want you to be thinking about how you could use this in your own work. Okay, thank you.