Transcription
Hello guys, this is Paul McQuarter with TopTechBoy.com, coming to you today from the shores of the mighty River Nile in order to bring you episode number 28 in our incredible new tutorial series where you're going to learn artificial intelligence, or you're going to die trying.
What I'm going to need you to do is pour yourself a nice tall glass of ice-cold coffee. That would be straight-up black coffee poured over ice, no sugar, no sweeteners, none needed. And as you're pouring your coffee, as always, I want to give a shout-out to you guys who are helping me out over at Patreon. It is your support and your encouragement that keeps this great content coming. You guys that are not helping out yet, take a look down in the description, there is a link over to my Patreon account. Think about hopping on over there and hooking a brother up.
But enough of this shameless self-promotion, let's talk about what I am going to teach you today. And what I'm going to do today is, it's a really natural follow-on to our lesson number 27, where we really kind of got a very, very good gesture recognition program going using OpenCV, Python, and MediaPipe. Now, the one thing that I didn't like, the one little improvement that I could see that we could make in that program, it's very accurate right now, but as we were doing our development work, we just had it set up that each time you would retrain it. So, like every time the program ran, it would ask you if you wanted to train it. Well, what I want to do today is I want to go in and kind of tweak it where when the program runs, we say, you know, you're going to kind of have a choice: do you want to open an existing training file, or do you want to train? And then if you want to open an existing one, which one do you want to open? And then if you want to train it, what do you want to call the file that you're going to put the training data in? And that way, if we want to use the program, we can just, boom, you know, if we got some really good hand gestures, we can just go load them up and not have to train every time. I hope that makes sense.
So, what I'm going to need to do is I'm going to need to get out of your way. And then what I'm going to need you to do is fire up your most excellent Visual Studio Code. And I can come over here, and I believe that you can see that, okay. I believe you got a good view of it. And then what we are going to do is I'm going to remind you that we are working in the, we are working in the Python folder. And let's see where were we, man? We're going to be on OpenCV 44. And so I need you to click the little plus icon. And then we are going to be in OpenCV-44.py. In the .py is kind of important. And boom, we have a fresh new Python program just waiting for you to write. All right.
But now, I will admit that we really don't want to start all the way from scratch. And we probably want to kind of pick up where we left off last week. And so what you can do is you can go to the most excellent www.toptechboy.com. And then I'm going to need you to use this happy little search feature over here. And I'm going to need you to search on "improved gesture recognition in Python and MediaPipe." And then you'll come up with this happy little page. And this is the code that we have developed so far. We can come over here and double-click on the little page, or just single-click on the double-page icon. That should give us the code. We come back over here, and we are going to paste it. And then I'm going to make sure that I'm still happy with the camera that I have set on. Yeah, camera 4, I believe, is my webcam right now. Let's go ahead and run this and just make sure the universe is in proper order and this thing still works. I'm not going to do a lot of training on this test run just because I want to make sure that it just runs. So let's just see if it runs. Okay, got some stuff going on down here. Looks like it's, uh, getting the TensorFlow going. All right. How many, uh, gestures do you want? We'll say five. And then, uh, we're going to call them.
Okay, maybe it doesn't like my camera. Why does it not like my camera? I said camera four. Sometimes when this thing boots up, it ends up moving my cameras around. And so let's come down here. Maybe I am on camera zero today. Let's see if I'm on camera zero today. The thing is, I have like six cameras, and it's not always numbered the same depending on, like, the last time I rebooted. A little bit annoying, but, you know, we'll adapt. So let's try it this time. Okay, how many gestures do you want? We're gonna go for five again. You can't quite see that, can you? Let me bring this up a little bit. There you go. Okay, so what's the first gesture? I'm just gonna go one, two, three, four, and five like that. Okay, now it's ready to start training. So let's pop this up. Okay, so it wants gesture one. So we'll give it gesture one. Okay, gesture two, gesture three, gesture four, and gesture five. Okay, five, one, two, three, four, five. Close. Far. Boom. Shazam. It's working. All right.
So that's good. Now, did you see how we had to, like, every time we run the program, we got to train it? Let's go in and see if we can create a training file. Okay? And then we'll either load that file or we'll train a new one. And this is going to be easier than, than you probably think it is. If we just sort of play nicely with the way the program already runs. And remember, kind of at the top of the program, what we were doing, kind of at the top of the program, before the while loop, what we would just say is we would just always say, "start with train equal true." So just always start in training. And then whether you were training or recognizing, everything was really happening kind of down in the while loop. Okay?
Well, what we're going to do different here is we're going to do a little bit more prep work upstairs, up in this part of the program. And then we're just going to drop down into very, very similar flow of what was happening before. But this time, what we are going to need to do is, let me do a little bit of window management here. Okay? So we're going to come after these key points. All right. Also, I'm not going to use "train equal true and false" anymore. I'm going to use a 0 and 1 because if I ask for data from the user and I ask him for true or false, then it could matter whether he uses uppercase or lowercase. So a zero or a one, I think, would be a lot easier.
So the first thing that we need to just do is we need to find out, does the user want to train or recognize? And so what I'm going to do is I'm going to say, "train is equal to an int," so it's going to be a number, and then "input," and then I got to give it the prompt. So I'm going to say, "Enter 1 to train, enter 0 to recognize." Okay, I think that's pretty good. So now it's a 0 or 1 whether we train or recognize. I think that makes a lot of sense.
So now what we want to do is we're going to say, "if train equal equal 1." So if we're gonna, that was kind of ugly. I don't know what I must have hit a page up or something. So if train, and it's little t, I like that little squiggly underline. Let me see if I can make this a little bit bigger for you. Okay, I think that's, I think that's good. I think you, let's see if I go one more. Ah, pardon me for a second, but I just want to make sure I make this as easy to read for you guys as possible. I think that looks pretty good.
Okay, so now we're going to say, "if train equal 1." So if I actually want to train it, what do I need to do? Okay, well, if train is going to be equal to 1, then a lot of this stuff will, first of all, I'm not going to say "if train is true" anymore because I'm asking it as a 0 or a 1. And now a lot of this stuff, the tolerance, I need to keep here either way. So the tolerance is going to be just part of the overall program. But now all this other stuff that I did just assuming that I was going to train, all this needs to be moved up inside of this "if," like this. Okay? And now I can get rid of this again. The tolerance, I need separate from the, you know, whether I'm training or not. So that's always going to be there.
Okay, then, so, uh, let's see where are we? This should be inside the "if" too. So I'm going to set this "known gestures" to, uh, to, uh, an empty array. Then the number of gestures, that also we need only if we're going to be training. Okay? And then we can get rid of it here. And then we need to get this. I'm going to cut it and move it up here. Okay? So remember, all of this stuff was stuff that we ran every single time before because we were training every single time. But now it's going inside that "if" statement. So we only do this, like, uh, setting up the gesture names array, getting the number of gestures, all that, we're only going to do if we want to train.
Okay, I'm going to get rid of some of these blank spaces here, blank lines. Okay, now what we're going to want to do is we are going to want to get the names of the gestures. If we're training, we need to get the names of the gestures. So we're going to cut that part. And then again, that we are only going to do, okay, we are only going to do that if we said that we wanted to train. Now you can see when I pasted it, it did not, uh, it did not indent these. So I need to indent these three lines to be in the for loop. Okay? And then the print is going to be inside of the train. Okay?
Now that I've said I wanted to train, then I said that I wanted to train up here. And then if train is one, this is what I do. I'm going to set the train count to zero. That makes sense. I'm going to create a known gestures array that's empty. I'm going to create a number of gestures from the user. And then the gesture names, I'm going to initialize. And then I'm going to go through this for loop and I'm going to get all the gesture names and I'm going to print them out.
Now, what else do I want to do, only if I am training? Well, if I'm training a new data set, I better give it a name. So I'm going to say, "train name." Okay? And that's going to be equal to "input." And this is going to be the file name that if we're saving the training data, it's going to need a file name. But I want that if you just click enter, I want it to just use the file name default. And that way, you can just click enter, enter, when it's asking you these questions, or you can put in a file name if that makes sense.
So, "train name is equal to input." And then what do I want to input? "File, file name for training data." Okay? And so I'm asking them for the file name of training data. And then I'm going to say, I'm going to say, "Press, press enter for default." Like that. "Press enter for default." And that way, that way you don't have to put in a file name if you don't want to. It'll just store it in the name "default." Okay, I hope that makes sense.
Now, what we need to see is if the train name, if the train name is equal equal nothing. What does that mean? That means that they just pressed enter. So if they just pressed enter, then what do I want to say? The train name is equal to what? Default, right? So I'm putting it in so they don't have to. So the train name is default.
Okay, now also, in either case, whatever it is, you know, if they entered a name, then I want to say, "train name is equal to train name concatenated with .pkl." Because what we're going to do is we're going to be creating pickle files. So I don't want the user to have to put that that extension. I'm gonna put the extension on it for them. So, and that should be a string. So we're gonna concatenate it. We're gonna concatenate it with the string ".pkl." So whatever name they put in, I'm going to put a .pkl on the end of it. Or if they just click enter and that's nothing and I name it default for them, then on default, I'm going to put the .pkl. I hope that makes sense.
So now, that is really all that we need to do. Now, because now it's kind of like we're right back to where we were before. When I drop into the while loop, I'm going to have everything that I need in order to train. And then I have "train" is set to 1. Now, the one thing I got to do is I got to go down and I got to change those "true"s and "false"s in the loop. I've got to change those "true"s and "false"s in the loop to ones and zeros. So, "if train is equal to not true," if "train is equal to 1." Okay? And then here, "if train is equal to false," that would be "if train is equal to 0." If that makes sense. And then here, where I set when I'm done training, when I'm drawing training and I set "train now" to false, I need to set it to zero. All right, just because I don't want to ask a user for true or false because that is the difference between the logical true and false and the string true and false and the integer true and false. If we just go with zeros and ones, it's going to be a lot more straightforward.
Okay, so I think that is really good. Let's come back up here and look at, look at what we did. So we have the case, "if train is equal to 1." Now you can see here, we're getting ready now to do what we've done. The "if train is equal to 1." Well, what do we do? What do we do if "train is equal to equal to 0?" Well, here I don't want to train, I want to use a file. Well, if I want to use a file, I better know what the file name is. So I'm going to ask them for the file name. So, "train name is going to be equal to input." And then, "What training data do you want to use?" Question mark, space. Okay, "train name input, what training data do you want to use?" So now I have a train name. Now, remember what we also want to do here, we want to just say, "Press enter for default." Like that. "Press enter for default." So if they just click enter, it'll run out and grab the default, and they don't have to do anything. So I think that makes a lot of sense.
So now, "if train equal equals 0." Now we have the train name, we've gotten that from the user. Now we've got, see, if he hit enter without putting anything, then if "train name equal equal the blank," I mean, not even a blank, just nothing, an empty string, then what do we want to do? We want "train name to be equal to default," the string. Okay, "default," the string, like that.
Now, after that, in either case, whether he input it or whether I set it to default, we want to say, "train name is equal to train name concatenated with what? .pkl." Like that. ".pkl." Does that make sense? That sure makes sense to me. We're just setting things up before we drop into that while loop. And that while loop is already set up to decide whether it's already set up to decide whether it should be training or whether it should be recognizing. And now I'm just kind of setting things up here to read the file if, if that was needed. Let's get rid of some of these silly blank lines. I like a few blank lines, but not a whole lot in a row like that.
Okay, so now we got to think, if train is, if, if train is not equal to zero. I mean, if, if we're not training, if we're going right into recognizing, we better go load that file. Because if you are down in the main loop and you're recognizing, it wants to know the gesture names and it wants to know the known gestures. And so since we're not doing that in the loop, we better go out and read the file here. Because we've said we're not training it. And if we're not training it, we better go load that data in.
So I'm going to say, "with open." Okay? And then "train name." That is what we want to open. And that box is getting annoying. Okay, uh, we're going to open "train name." And then we're going to open it as a read by. So we're going to read from it. And then we are going to open it as "f." You can call it whatever you want, but I'm just going to call it "f" for file. Now, what do I need to do? I need to get the gesture names. Now, how did I know gesture names? Well, I just got to remember when I write the training thing, I need to write gesture names first. And then I need to write known gestures next. And so I'm going to say, "gesture names is equal to pickle dot load." And then "pickle dot load from where? F." Like that.
Now, you see how it doesn't like pickle here? Do you see how it doesn't like pickle? Pickle is a library, and we haven't loaded it yet. So we better come to the top and we better import our friend, Mr. Pickle. And the number one reason people have trouble with pickle is because they can't spell pickle. If I were in charge, I probably would have spelled pickle P-I-C-K-E-L. But given that they spell it differently, we better go with their spelling. Okay, so now it should be happy down here when we say load pickle.
Okay, so we say, "gesture names is equal to pickle dot load." And then what else do we need to load? We need to load the "known gestures is equal to pickle dot load f." All right. Now, why did I, like, is maybe throwing you off a little bit because I'm loading in that training data before I've saved it? Why am I doing that? Because this is a little bit more clear here. It just says, if you are not going to train in the program, then what you need to do is you need to go out and you need to load the training data from a file. All right.
Now, what we have to think is, we need to go down and make sure that if we are doing a train, if we are doing a train, that we are creating that file. But we can't create it up here because the only way you can create it, you've got to go in and be analyzing images. So we've got to create your known gesture array. We've got to create that down in the loop because we've got to be grabbing frames and analyzing them. And so where we would do it is we would come here in the loop where "if train is one." Okay? We'll come down in the loop where "if train is equal to one." Okay? And then we say, "if hand data is not equal to nothing." So if we told it to train, and if there is data there, then I need to ask for a gesture. So I'm going to say, "Please show the gesture and then press T when ready." And then it's going to wait for the T. And then it's going to say, "known gesture is fine distances," okay, of hand data. And so that is going to be doing the analysis. And then "known gestures dot append." Because right, what do I want? I don't want just the training data for one gesture, I want them all. So, "known gesture" is the training data for one gesture. "Known gestures" are all of the gestures in an array. Okay? And then I'm going to say, "train is equal to train count plus one." Okay? And then, "if train count is equal to the number of gestures." All right, that makes sense. That if now we are done training, we've done all of them that we were supposed to, that we've gotten up to the number that we asked for. Now we say, "train equals zero." We're done training. But now what do we need to do? We need to save the training data.
So I'm going to say, "with open." With open. And then what do I want to open? "Train name." Which we set up up at the top. This time though, we are going to, man, that is so annoying. I've got to make it smaller so that box isn't so annoying. Okay, "with open train name." Word writing or reading? We are writing. So this would be write by WB. So we open that again as "f." Now we're going to do a "pickle dot dump." What do I need to dump first? What do I need to dump first? The gesture names. Remember the gesture names. Okay? And where do I dump them to? F. Okay. And now I need to "pickle dot dump." And then "gesture," and not "gesture names." "Known gesture" is it "gestures" or "gestures" plural? Because up above I built all those training data into one big array. So it is the plural because I want all of them. If that makes sense. "Known gestures," comma, "f."
How many mistakes did I make? Let me just look and see. I should do better to see if I have any of just those really ugly little, those ugly little underlined squigglies. And I don't. So I'm going to take a big swig of coffee and we're going to run this thing. And the first thing is, does it run without crashing? And then once it runs without crashing, does it actually train and load and that sort of thing? So let's run this thing.
Okay, so far so good. Don't worry, it always takes a lot. Okay, look at that. That makes me kind of happy. It says, "Do I want to train or recognize?" Well, I better train because I don't have any training files out there. So I'm going to say one. Okay, I don't like that formatting. So I'm not going to quit the program, but I'm fix it right now because I didn't have a space there. And we always want to have nice formatting. So where we come down here and we say, "Enter 1 to train or 0 to recognize," we should have had a space there. And then that would look a lot better. So I'm gonna train. So I'm gonna put in a one. And then, "How many gestures you want?" Let's do an easy one. Okay, let's just say five. Okay? And now, "Name of gesture one." One, two, three, four, five. Five like that. Okay, now, "File name for training data." I think that I just want the default. But again, do you see how I didn't leave a space? And so where was that? "Press enter for default." I should have put a space there. Like that. "Press enter for default." And so I'm just going to use the default. I'm going to press enter. Okay? And now it is ready to train. And what it wants is it wants to see a one. So I'll flash it a one. Okay, there's a one. And then a two. And then a three. And then a four. And then a five. Five. Okay, five, one, two, three, four, five. Shazam. Look at that. All right.
Now it's just working like it's working. It's recognizing. But now the question is, if I run it again, can I go load that without having to retrain? All right, so let's quit. Let's take a big dose of coffee. Caffeine is the fuel that runs the engineering world. Okay, let's run it. Okay, this time, do I want to, what? Train or recognize? I want to recognize. So I'm going to put in a, what? A zero, because I want to recognize. And then we create. Press enter. And then, "What training data do you want to use?" I hope I spelled it "extended" right the last time. Okay. And now it seems to be doing something. So let me fire this up. Okay. Okay. Unknown. One, two, three, four, five. Boom. Man, do you see that? Look, one, two, three, four, five.
Okay, what do we need to do now? We need to make sure that we can have more than one training set, more than one data set. Okay? And so I'm going to leave that as the default. But now I'm going to create another one and see if we can work with multiple, multiple training sets. So let's run this thing again. Man, I am excited. This is fun. This is really fun. Okay. Okay. Enter a one to train or a zero to recognize. Well, I'm going to train again. Okay? And how many gestures do you want? I want 12. Okay? And one, two, three, four, five. Okay, let's go guns up. Let's go high five. Let's go fourth and prosper. Okay, let's go hook horns. Okay? And then let's go, "A-okay." Let's go thumbs up. And let's, uh, let's go braid. Done. I probably shouldn't do that one. Probably shouldn't do that. What, what's the difference between being 16 and being 60? A young man, 16 years old, will say, "Wouldn't it be funny?" But a 60-year-old will say, "Wouldn't it be funny, but I probably shouldn't." That's the difference between being 16 and being 60. So we're gonna need another 12. I've done the high five, the go forth and prosper, the, uh, the A-okay, the thumbs up, guns man. Which one am I missing? Ah, the most important one. Resist the metaverse. Like that.
Okay, now, what do we want? Well, since this one has more, I'm going to call it "extended." Okay? I'm going to call it the extended. Okay? So now it wants to start seeing these gestures. So let's give it what it wants. One, two, three, four, five. Okay, guns up. High five is like this. Go forth and prosper. Hook horns. A-okay. You gotta kind of, it's different whether you show it the front or the back of your hand. So the thumbs up is a little harder. And then my favorite one. Resist the metaverse. Okay, look at that. Now, one, two, three, four, five. High five. Go forth and prosper. Hook horns. Guns up. A-okay. Thumbs up. Boom. All right, guys. Okay, so now we trained it on that. Now, can we go back? Now, can we go back and just load that? All right.
Okay, one to train or zero to recognize. Let's zero to recognize. What training data do you want? I hope I spelled it "extended" right the last time. Okay. And now it seems to be doing something. So let's come in here. Let's come in here. What is that? Unknown. One, two, three, four, five. Guns up. Hook em horns. High five. Go forth and prosper. Guns up. Hook em horns. Thumbs up. A-okay. Resist the metaverse. Okay. All right.
So you see, guys, now we can train it, and we can have different training sets for different applications. And it really, to me, I just think it is really working well. And so I am super excited about that, about this, because we're able now to have something. And remember how we did that adjustment so it works well independent of how far you are from the camera? And now we don't have to train every time. We can save our training data. We can get a good training set. And now what you can start thinking about is you can start thinking about, what can I do with this? My computer can figure out what I want based on what my hand is doing. And so now what you got to think is, what can you do with that? What can you do with that that can your computer start doing useful things based on what your hand is doing? Okay.
Oh man, guys, I hope you are having as much fun taking this class as I am making it. I am just so excited about this gesture data and I am really, really having a lot of, a lot of fun with it. What I hope is, if you enjoy this video, you'll give me a thumbs up. Also, it helps me if you leave a comment down below. If you have not already, go ahead and subscribe to the channel. When you subscribe, make sure you ring that bell, and that way you'll be notified when I have future lessons coming out. And what's really good is, man, share this with someone else. Share it with someone else because the world needs more people coding and doing engineering and fewer people sitting around watching silly cat videos. Paul McQuarter with TopTechBoy.com, coming to you from a rainy day on the banks of the mighty River Nile. I will talk to you guys later.