Transcription
Welcome back. So we're talking about Bayes' theorem, which is one of the most useful ways of computing inverse probabilities or solving for inverse problems. Estimating something you want to know that's hard to measure from something that you do know that's easy to measure. And we gave an example of, uh, screening for a disease like cancer, um, using some kind of a test.
Um, and I want to give you one more example, just because I think this is such an important theorem. I want you to see how it works in different examples. And the example I'm going to do is about drug testing. Um, now this is kind of a fun meta version of Bayes' theorem. I'm curious what you think my views are on drug testing given information about me. So I'm not going to tell you my views on drug testing, but you should figure out what my views on drug testing are based on some information. So first piece of information: I'm from Texas. Second piece of information: I went to school in California. Third piece of information: I moved to Washington in 2012. And at the end of this lecture, maybe you'll have more information that tells you my views on, uh, on drug testing. Okay, so let's say that you're testing for some, uh, I don't know, illegal drug or narcotic or whatever, you know, some drug that you're testing for or you're screening for. And let's say your test, uh, is 90% accurate. And let's say that 10% of your population uses uses this drug, uh, and you have a 10% user rate. Okay, good.
And I think I'm going to define probability of B as whether or not you used or not, and probability A is whether or not your test was positive for that drug or negative for that drug. Okay, um, so I think yeah, I'll just actually write this out. So, um, B is: do you actually use? And A is: what is your test score? The test score. Because again, um, if you're drug testing, you're not going to go home with a person and follow them for 24 hours and actually see: did they use this drug? You're going to give them some test; you're going to, you know, they're going to pee in a cup, you're going to a piece of hair, you're going to do a test, which is cheaper, less invasive, and not 100% accurate. Let's say in this case, 90% accurate. So the probability, um, of—I'm just going to write down some probabilities—the probability of a positive score given that you are a user is 0.9. It's 90% accurate. And also the probability that you get a negative test given that you're a non-user is also 0.9. We're going to change this in a minute, but for now let's just say that these are the probabilities. So we're going to ask ourselves a very Bayesian question. Let's say, you know, I test my employee and the test comes up positive, and you know, I want to know, given that they got this positive test, are they in fact a user? What's the probability that they actually did drugs given that positive test? So the probability of use of it of the the person being a user given a positive test is the following: We're going to use Bayes' theorem. It's the probability of, uh, the positive test given a user, positive given user, times the probability of them being a user, which is 10%, divided by the law of total probabilities. It's the probability of a positive test given them being a user times them being a user plus the probability of a positive test given that they're a non-user times the probability of them not using. Okay. And we're going to plug these in, compute the probability, and come to some conclusion. Okay, uh, so the probability of plus given a user, this is 0.9. The probability of them using is 10%, so that's 0.1. And then the probability of plus given user, again this is .9 * .1 plus the probability of a positive test given that they're a non-user, this is 0.1. The probability that this test is wrong is 10%, so that's 0.1. The probability of a of a Mis of a of a of a false positive given that they're not using times the probability that they don't use. Only 10% of the population used, so 90% don't, so this is 0.9. And you can see pretty quickly that these are symmetric numbers, so this is, you know, uh, .09 divided by 2 * .09. This is 1/2, 50%. Okay. So what this means—I cooked this example up—so now you have more information about my views on this procedure. The probability, given these numbers—these are made-up numbers; I made these up maybe to make a point—the probability of being being a user given having a positive test is only 50%. Okay. That means, um, you know, like if I if I tested positive, there's a bunch of false positives, and so the chances of a person actually being a user given a positive test is only 50%. Now, if you really really cared, like if this is person is going to be, you know, an airline pilot or something, or if they're, you know, doing surgery and you really want to know that they're not doing whatever, you know, uh, sub—this is—this would probably mean you need to do another test on this population that tested positive. You screened; now you need to do more testing because this is not enough information; this is not conclusive enough. There's only a 50% chance the person was a user given this positive test score. And again, this is because of rare events and accuracy. If this was 99% accurate and this was, you know, 50% of the population, that would be a different story; you'd be much more certain given those test results. I cooked this up, you know, this this example up, and that gives you information about my views on drug testing. Um, but again, this can be used to calculate in polygraph testing, like if you polygraphed all of your employees for some, uh, some question. If you drug test all of your employees, you know, just because you have a test that that's moderately accurate, if you have a rare user base or a rare base of people that would, you know, fail that polygraph, you can get a lot of false positives that kind of mess up this inverse, uh, estimation because that's you're using this test score as a proxy for whether or not the person actually, you know, used this drug, and it's not, um, that accurate. Good. Let's do another example with slightly, uh, different numbers for these. I think this is going to be useful. I'm going to erase the board, uh, I'll be back in a moment.
Okay, so I want to do a slightly more, uh, complicated example where the test has different accuracies for the different groups, for the users versus the non-users. So let's say that the test is 90% accurate in telling if you did use a particular drug. So if you did use it, it's 90% likely to to test positive. And let's say that the probability of a negative test given a you're a non-user, let's say that that is 80% accurate. So let's say that if my my my non-user population, 20% of the time I get false positives. Okay, so it's kind of a bad test, um, but let's say that, you know, it's more accurate in detecting the thing, but it also like actually has a lot of false positives. 20% of of of the non-users will test positive is what this means. So this is called the sensitivity; this is how sensitive the test is to the thing you're actually trying to detect, whether or not someone used this drug. And this is called specificity, uh, which is a measure of kind of, you know, roughly how many false positives you're going to get in the non-user population. And this is important because, you know, what if I really wanted a test? I could make a test that's 100% accurate in detecting whether or not you used. I would just say that it's always positive. I would just like always say that the test is positive, and and then my probability of it being positive given you being a user is 100%; it would be 100% sensitive, but it would have a terrible specificity. Okay, so this is something that's really important, um, and actually this is important in machine learning, too. If I have training data and I'm trying to train on rare events, things that very rarely happen, I can get like 99.9% accuracy just by never saying that that thing happens. So I'm, you know, I I have terrible, uh, prediction of the rare events I actually want to be, um, detecting, but you know, my my accuracy would be very high. So you need to think about sensitivity and specificity and how rare an event is, and it totally changes your statistics. It changes how you do your machine learning, how you weight your data, how you train, all of that. Okay, so we're going to work on these numbers because this is kind of a little more interesting. This implies that the probability of a positive test given you being a non-user is 0.2 or 20%. And so again, we can use Bayes' theorem to compute the thing we want, which is you're using this test as a proxy for whether or not that person used this drug. So the probability that the person is a user given a positive test with these numbers is now, um, you know, probability of plus given user times probability that they're a user divided by probability plus given user given that they're a user plus the probability of plus given that they're not a user times the probability they're not a user. That's just the second form of Bayes' theorem. And I can use these numbers here. This marker is far too squeaky. And I get, uh, .9 * 0.1 because only 10% of the population uses. 0.9 * 0.1 plus now I have a 20% false positive rate, so that's .2 times my 90% of the population that doesn't use. .9. Now these numbers get even worse. This is one in three. This is, uh, this is one in three. So the probability that if I had a positive test I actually used that drug is only 33%. That's really really bad. That's not a good test, um, and so you know, I could I could make a test that says positive all all the time, and it would be like 100% sensitive, but it would be totally useless in inferring if you actually did that drug, if you actually had that underlying, you know, you know, uh, underlying use that that that should give you that positive test result. So hopefully this kind of illustrates some of the cautions, um, of computing these probabilities and and of testing and, um, you know, this applies generally. All of the examples I've given is where the thing you're trying to detect is relatively rare. This becomes a little bit better when the probability of B is is less rare, but for rare events, um, you really need highly sensitive and highly specific tests, or else you're going to totally botch, uh, your ability to actually infer the thing you wanted to infer in the first place. Okay, uh, intro to Bayes' theorem. We're going to get way deeper into this, but it's going to be a little while, um, so you know, think about how you can use this, think about how you can update if you were going to do sequential testing and things like that. Okay, thank you.