Transcription
Hello guys, this is Paul McQuarter with TopTechBoy.com, coming to you today from the shores of the mighty River Nile to bring you episode number 25 in our incredible new tutorial series where you're going to learn artificial intelligence, or you're going to die trying.
What I'm going to need you to do is pour yourself a nice tall glass of ice-cold coffee. That would be straight-up black coffee, poured over ice. No sugar, no sweeteners, none needed. And as you're pouring your coffee, as always, I want to give a shout-out to you guys who are helping me out over at Patreon. It is your support and your encouragement that keeps this great content coming. You guys that are not helping out yet, take a look down in the description. There is a link over to my Patreon account. Think about hopping over there and hooking a brother up.
But enough of this shameless self-promotion, let's talk about what I am going to teach you today. And just a reminder, what this is, is this is really the solution to the homework assignment that I gave you in lesson number 24. And that was to take the work that we had done in our earlier lessons in this series where we had learned how to use MediaPipe to find the hand and find the hands that are in a frame, and then to find each landmark on that hand, and then to have a nice data packet, a nice Python array that has the X and Y coordinates of all the landmarks on the hands.
Now, what you were supposed to do, now that we've already done that, we already have a nice function, a nice method, a nice class to do that, but to take that work that we had done and now use it to develop some sort of algorithm where we can learn gestures, and then based on the gestures we learn, we can find those gestures. How many of you guys had any success at all on this? If you had some success, put a comment down below. I am legend. And if you did not have success, I folded up like a cheap Walmart lawn chair.
Okay, if you folded up on this one, don't feel bad because this one, you really had to kind of think through. And I had to do a lot of thinking on this before I came up with an algorithm that I was really happy with. But what I really wanted was one that I could go through and just train it on different gestures. You know, to give it different gestures and to train it on those gestures, and then to sit and run and find those gestures. I found it a little bit tricky. And so what I'm going to do is I'm going to start by just kind of showing you how I thought about this. And, uh, to do that, I will come over here to my new, most excellent sketchpad view. And we're going to start. We're going to start this just by looking at what our, uh, what our, our, uh, basic configuration is. And that is, is that we have this hand, which is shown up in the up in the corner over here. And then for a hand, each hand has, uh, each hand has. Can't quite see the four. Let me see if I can move this. I'm going to live dangerously and see. Now, I don't think I can move that. Let's see. All right, I'm sorry. You're missing the, you're missing the four up there. That really bugs me. Let me, let me see if I can fix this before I move on. Give me just a second to see if I can fix this. I think I can bring the four back in. Yes, there. At least you can kind of see it. Okay, now let's come here. Sorry for that. I just had the four kind of chopped off, but now you can see it.
So you can see for a hand, there are 21 landmarks because 0 to 20 is 21. We start here at 0, and then we end up at the tip of the pinky, which is 20, which would be the 21st element. And so if we go back to that program that we had where we created that class to parse this data, what we ended up with is we ended up with an array. And that array had like X0, comma, Y0, which would be the position of the 0th position. And then it had X1, comma, Y1, and so forth, all the way out to the tip of the pinky, which would be X20, comma, Y20. Okay, so we have an array. We have an array of 21 different XY coordinates for all of those positions.
Well, based on that, then, how do we take that array and then look at a hand and look at the array, look at the array for a hand, and decide, is that hand in the Hook 'em Horns? Is that hand in the Hook 'em Horns? Or is that hand in the "Power to the People" sign? How do we break that apart?
Well, the first thing that I thought about doing is, it's really about distances. Okay, so what I wanted to do was go in and calculate the distance between every point and every other point. And just a reminder, like if I wanted to just say, like, the distance between this point, well, let's see, the distance between the zero point and the distance between the one point. Okay. And I'm sorry, I can't seem to draw. Like, let's see here. All right, maybe I can do it here. If I wanted to draw from the zero position to the one position, if I wanted to know that distance, how would I go about finding that distance? Well, the distance between the zero position and the one position is equal to, it's just our old friend, the distance formula. Okay. And so this would be X1 minus X0, the X position of the one point minus the X position of the zero point squared, and then plus Y1 minus Y0 squared. All right. And since we would be going from, like, not always zero and one, we would be going from A to B, it would be the distance between point A and point B would be X sub A minus X sub B squared plus Y sub A minus Y sub B squared. And then we could find the distance between any of those.
Well, the thing is, if we start at zero, if we start at the zero, then we would have an array of distances between zero and zero, well, that's going to be zero, between zero and one, between zero and two, zero, three, zero, four. And so what, what we're going to have is we're going to have a distance value, D0, D1, D2, D3, out to D20. Okay, so because there's 21 points, we would have all the way between point zero and point zero, all the way out to the distance between point zero and point twenty-one. But what you gotta see is that is just the first row, because that's the distance between this point and all the other landmarks. Well, there's also then, if we go to point one here at the thumb, now we would have the distance between this and all the other ones. So this array is going to have a second row. And so we're going to have a comma, and then this is going to be again, D0, D1, D2, D3, out to D20. But this time, it is going to be the starting point here, instead of the starting point here. And so what you can see is this is going to get really complicated. That we are going to start at row 0, and we are going to go all the way to row 20. And each one of those, each one of those is then going to have a column 1, a column 2, out to a column 20, or I should say column 0 here. So it's going to be like a 21 rows by 21 columns. And that is going to be the distance between every point and every other point.
Now, there will be some duplication because you can see the distance between here and here is going to be the same as the distance between here and here. So every distance is going to probably be in there kind of like twice. Like the distance between A and B is the same as the distance between B and A. But if we create this matrix, then we're going to have every single different possible distance. And so the first thing that I did, the first thing that I tried to make this work was to create that huge array, and then to train it on, let's say, train it on "high five." And then create all of those distances for "high five." And then when I come in with an arbitrary thing, compare my known "high five" distance matrix to my unknown distance matrix and see what the error is.
So I would, for the two, for the two arrays, let's see if I can just kind of, uh, simplify them here again. It's 20 by 20, but if I just sort of create a simple smaller one here, I would then create an error array, which was this minus this, this minus this, this minus this, the difference between all the corresponding, the difference between all the corresponding elements. And then sum up all of those differences, and that's the error. And then have a certain tolerance for error. And if you're better than that tolerance, then you've got the "high five." If you're worse than that tolerance, then you don't have the "high five."
Okay, it sort of worked, but not really well. And the reason it didn't work very well is it's never going to be the "high five" that you flash it later is never going to be the same as the "high five" or, you know, the "power to the people" that you flash it later is never going to be the same as the "power to the people" you flashed it to train on. And the problem is, like, when you're not, when your fingers are like this, it's all kind of like really where is it? So there's a lot of confusion in here. And so you're adding up a whole, whole lot of different numbers, like 20 by 20, say 400, 400 different numbers that you're adding up. And some of them don't matter near as much as other ones do. But like, what really matters? Like for the number one, you can see the big differences. What really matters is the distance between here and here. And then all the stuff that's going on in here is just noise. But when you add up all of those different things, it overwhelms what really matters, which is from here to here. So when I just did everything like that, it didn't work very well. I didn't get good results because there was just too much data, and there was too much data that didn't matter.
Well, then the second thing I decided to do was, I decided, well, let's simplify. And instead of comparing every single point to every other point, and every point to every other point, what if we simplified the data set? And then what I said that I would try to do, then I said what I would try to do would be to come in and what I would do is I would just come in and I would compare everything to the wrist position. So I would compare the wrist position to here, to here, to here, to here. And then my distance matrix would just be one row. And that's the distance between zero and zero, the distance between zero and one, the distance between zero and two, the distance between zero and three, on out to the distance between zero and 20, like that. And so I just came up with one row, and I just compared everything to here. So all of a sudden, you can see here, this is going to have a much, a very clean signal because this is one data point out of 21, and it's a big change. And then the other 19 are just kind of down here in the noise. But it's not going to overwhelm that one versus this one versus this one.
Now, that one actually worked a lot better. But then even there, what I found is, is that if you had something like, if you had something like the number five versus "high five" versus, uh, "go forth and prosper," that all of those would have very similar things. So just going from here to the tips, there wasn't enough difference to distinguish between those. And so in the end, what I did was, I thought, what would be the ones that would really matter the most? And let me see if I can. Hi. Let me see if I can come back over here. And I am going to get my eraser. Ah, that, that was a very bad idea. Okay, I'll just go to a different color. Okay, I will go to, I will go to green. Okay.
What I decided was to pick a smaller number. And I decided the ones that were really going to matter were the wrist, the tip of the thumb, and then we're going to go the tip of the fingers, and then we're going to go the kind of base of the fingers, like that. And now I have 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. I've reduced my data set down to 10. Okay. And then what I would do is I would go D00 to D01, D02, out to D0. And I, this would be if I've got 10 points, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, that would be D09. And then to do that between all the points, all the points of these 10 points. And so there would be 10 rows and 10 columns. So it's much, much smaller than my original approach. But it's the ones that really matter. It's the relationship between these tips and between each other, and where the fingers come out in this one. And you get rid of all these middle joints. And I thought there would be a much stronger signal there. And that actually is something that really worked, I think, and worked well.
And so it comes down to creating, first of all, we've got to create the distance matrix of for a given hand, what are the distances between those key points? Not all the points, but the differences between all of the key points. And then we have that difference matrix. And then I compare one difference matrix to another difference matrix. And I don't come up with an error matrix, I come up with an error number. So I take those two, find the error between all corresponding points in those two matrices, and then add all those errors up, and I come up with a single number called error.
Now, I hope that makes sense. And as we start coding it, I hope, I hope that you can sort of see what I'm trying to do. I feel like I need to say it again. Okay, I take a gesture, I train, I now have a distance, I now have a distance matrix. Now I put an unknown hand in, and now I have a distance matrix, unknown distance matrix, unknown different distance matrix for a training. And now element by element, I subtract element by element, and I get an error. And I add up all of those errors, and now I come up with an error number. I hope that makes sense.
Let's jump over here. Let's jump over here and have a sip of coffee. I'm going to need to go to my code view, which I do believe is right here. Now, what I will need you to do is fire up your most excellent Visual Studio Code. And what I want to do is I want to go out and grab that code that we already developed that has the class that does all the hand stuff. And so let's go to the most excellent www. And you want to search on "AI for Everyone Parsing Hand Position Data in MediaPipe." Okay. And then you come down with this one, which we're going to grab. And this one just has that MP Hands class, the simple MP Hands class that just returns the hands landmarks. And we're going to start with that. And so we're going to come over here. And then what we are working on is we are working in our Python working folder. And then I am going to create a new program. And that new program is going to be, I forgot which one it's going to be. It's going to be opencv-41. So it's going to be opencv opencv-opencv-41.py. And the dot py is kind of important. And boom, a fresh new Python program just waiting to be written.
But we're going to start by pasting that code that we just copied. And I hope that's it. Yeah, that looks like it. And now, just for good measure, I better put my camera in the right position. Today, we will be operating on camera number zero. And I should run this, and it should just kind of give me, uh, uh, you know, kind of execute that MP Hands. And then there it is. Okay. And remember, we can display whichever ones of the landmarks we want. And here we're just displaying four landmarks. All right.
So now, what we want to do is we want to come in, and I think the first thing I want to do is I want to define an array that is the key points array, if that makes sense. And so right in, you know, right before we go into the while loop, okay, right before we go into the while loop, I want to define something that I'm going to call key points. Okay. And the key points is going to be an array. And it's going to be an array of the key landmarks. And what are the key landmarks? We'll look at these. It's going to be 0, and then it is going to be 4, 4, 5, 9, 13, 17. And then it's going to be 8, 12, 16, 20. Does that make sense? So I'm going to come back over here and I'm going to tell it what the key landmarks are. And then we can always just go to those points. We won't be going to all of the points.
So let's come back over to the code view. And I'm going to come in. I'm going to define those key points. What were they? 0, the wrist, for the thumb, and then 5, 9, 13, 17. And then 8, 12, 16, and 20, like that. And let me make sure that's 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. So I've got 10 points there. And those are the 10 points that we are going to be working with. And so that's good. That's a great place to start.
Hey, just for fun, down here where we said for N in these points, where we kind of went up like that, what if we just said here, key points, just to see if we kind of got what we wanted? This would be a real quick little, uh, real good little quick test to see if we're getting what we want in those key points. Okay. Yeah, look at that. Okay. You see those were the points we wanted. And so do you see now I'm going to have a lot more clean signal? Now, still all this nonsense here is going on, but there's a lot less points that are nonsense points, and a much greater fraction that would really work.
Now, the other thing that we could try to do is we could try to take these out and just do the wrist, the thumb, and the end of the four fingers, and not put those tips of the palms. But let's try this one. And we could try, we could try it just with those other points because we're going to develop the code around this concept of these key points. Okay. So now we have defined those key points.
Well, what I want to define a couple of functions. So all the heavy lifting is going to be done by this MP Hands class. But now what I want to do is after the MP Hands class, let's come after we set up this. And let's just, let's go before we do the cameras, after we do the class, okay, before width and height, what I want to do is I'm going to define two functions. The first function is going to take that hand, and it's going to find all the distances between all the points. The distance between here and here, here and here, here and here, here and here, for those key points. And then it'll be, you know, then you'll move to the next thing, the next point, then from here to here. So it is going to be a multi-row distance array. Okay.
So how would we do that? Well, I'm going to define a function, and that function is going to be called find distances. Find distances. What does it need? It needs the landmarks, right? So it needs the hand data, which would be those landmarks that we're going to pass to it. Okay. Now, if I'm going to have this difference matrix, which is the difference between here and here, here and here, here and here, it's going to be a, a two-dimensional. It's going to have rows and columns. It is going to have rows and columns. So it's going to be a 2D matrix or a 2D array. Well, we need to create an empty one to begin with. So I'm going to create, and I'm going to call that the distance matrix, dis matrix, with an X. And I got to create an empty one, and then we'll go in and put the numbers in. But how do we do that? Well, I need my old friend, Mr. NumPy. So we're going to come back up here and we're going to say, import NumPy as NP. I tried doing this just with the standard Python arrays without using NumPy, and there were some unanticipated things that were happening. So it's just like, man, if we're going to be doing math, let's use NumPy because it always works. And I had some things that didn't work very well, just using the standard Python arrays. Okay.
So now I'm going to create this distance matrix, and it's going to be NP for NumPy. And I want NP to just create an empty array. So I'm going to say NP zeros. So that's going to create an array, distance matrix, that's going to be filled with zeros. Now I have to tell it how many rows do I want, and how many columns do I want? Well, I want length of hand data rows, and then I want length of hand data columns. Length of hand data rows and length of hand data columns. And then I want a comma. I've got to tell it what type of numbers I want. I want D type, and I want that to be floats. So all the numbers in there are going to start out 0. All the numbers in there are going to start out 0. Okay, but it's going to be a floating point 0, not an integer 0, because we might want 3.24 in there later. So we've got to make sure that these are zeros.
Now, I know what you're saying is, you are saying, why are you going all the way out to hand data? So that's going to be hand data rows, that's going to be 21 columns, and that's going to be 21 rows. I thought you were only going to do the intermediate ones. I'm only going to use the intermediate ones, but as far as creating this distance matrix, it's way easier to just do them all because there's just a few lines of code that'll do them all. And then when we come in to look at the error, we're only going to use the ones from key points. Now, you could try to do this where you're just calculating the distances that you're going to use, but I think it would be a lot harder than just going ahead, calculate them all, and then just use the ones you want.
All right, now what we have to do is we are going to have to step through. First of all, on the outside, we're going to step through the rows. On the first row, we're going to find the distance between all the points and the wrist. And then on the second row, we're going to find the distance between all the points and the thumb. And then on the third row, all the points. So you kind of see, you kind of see what we're doing there. But we're going to step through the rows. So I'm going to say for row in range. Okay. And where do I want to start? Well, I want to start at row 0, and I want to go where? All the way out to the last one, which is going to be the length of hand data. Now, why didn't I just put 21 in here? Okay, like just put 21 in here? Because maybe later I would do something different. So maybe I would pass it something different. And so just if I do it hand data, it will always work for whatever hand data is passed to it. So I'm going to start at length. I'm going to start at 0. I'm going to go all the way out to the length of hand data. And then I think that is all we need.
So I'm going to step through the rows. And what do I also have to step through now? I got to step through the columns. So I'm going to be at row 0, and then I'm going to step through all the columns. Then I'm going to go to row 1, step through all the columns. So the columns are on the inside. So now I'm going to step through the columns for column in range. I want to go from where? 0 to where? Length of hand data, like that. Okay, like that. All right.
Now, what do I have to do? Now, what do I have to do for each one of those? For each column and each row, for each one of those, again, this is going to be 21 columns and 21 rows. I have to calculate the distances. Okay. So now the first thing I'm going to do is I'm going to dist matrix. Okay, it's going to be the row value and the column value, right? And now I've got to populate it with what distances? So I'm going to have to calculate the distance. All right. And remember, it's XA minus XB squared plus YA minus YB squared, the square root of all that for all A's and B's. And so we're going to start with the X values. X of what? That would be hand data. Is the whole array that I'm passing? That's the landmarks. And hand data has the X position and the Y position. It's X, Y here, X, Y here, X, Y here. So I'm going to take hand data and then what I want is the row. So the row would be, let's say we're starting with this one, the row. And then remember, it's an X and a Y value. I want the X value, so that would be the zero value. So I'm going to take the hand data of the row minus the hand data of the column. So remember, like I said, we're going to go from A to B, and then it would be XA minus XB squared. Well, here the A is like the row, and the B is like the column. So I take the hand data row, and then zero is the X value, minus hand data column, and the zero value would be the, uh, the X value. Now I take that, and what do I do with it? I square it. So star star 2 will square it. I also need another opening here, another open parenthesis. All right.
Now, so that is X sub A minus X sub B squared. And now what do I need? I need to do the same thing with the Y value. I need to do the same thing with the Y value. And if this doesn't make sense, go back and look at the picture I drew, and it should make sense. So now what do I do? I add to that. What do I add? Hand data. Again, I'm still on that A, B thing. So this is going to be row and data row. But this time, I don't want the X value, I want the Y value. That is the second one, so I would put a one here. Okay. And then minus what? Hand data. And now column. And what is this column? Column one. All right. Now, what do I need to do to that? I need to, I need to close that parenthesis, and then I need to square it. Okay. So this is X sub A minus X sub B squared plus Y sub A minus Y sub B squared. And then what do I need to do with all of that? I need to take the square root of it. And the way to take the square root is to raise it to the 1 divided by 2, to the one-half. And just to be extra safe, I'm going to make that one point divided by 2 point, just to make sure that that's 0.5 and doesn't do some crazy integer math there. All right.
So now that should give me the distances. That should give me all of the distances between all of the points. So it's going to be a two-dimensional array with 21 columns and 21 rows. And now what do I want to do with that? I need to now return to whoever called it. I need to return what? I need to return the distance matrix. Okay. So now, basically, what I have is I have, uh, let me see if this is still alive over here. [Music] Where is the on button? Okay. [Music] Okay.
What I have now is I have this whole, I have this whole distance matrix, the distance between every point and every other point. And then again, I didn't do the key points, I did them all. Now, in the later part, I only use the ones I want. But right now, I have them all. So now let's come back over here to our code view. I now have that distance matrix. And I think mostly, like, kind of like half the work now is done. Okay.
Now, what did I say that I needed? I needed the distance between all the different points. Now, I also need to take two different distance matrices, one for the train data and then one for the unknown data. And then I need to calculate the error point by point by point between those and add up all those errors and come up with an overall error signal. So that's what I'm going to do here. I'm going to define find error. Define find error. And then what does find error going to need? It is going to need a gesture matrix that is one that you know what it is. You pass it something. You know, you pass it the gesture matrix that you know. That would be like, I know that this, this matrix, this distance matrix is "Power to the People." And then what I need is an unknown matrix to compare it to. An unknown matrix to compare it to. And then also, since I don't want to compare all of them, I don't want to take the error of all the points, I need, what I need to know which points we're going to look at. What is that? That's key points. That we defined down here. Does that make sense? I hope that makes sense. All right.
Now, what is this going to do? Well, this is really going to be very easy. First, we got to reset error to equal to 0. So we're going to set the error equal to 0. All right. And now we're going to do what? We're going to step through the rows and columns. We're going to say, in fact, we're going to step through the rows and columns so much identically that we're just going to borrow this again. Okay. We're going to take those two. So we are going to take the row for row in range 0 to length of hand data. And for column in range. Is that right? Uh, no, no, no, no, no, no, no, no, no, no, no, no. We don't want to do all the rows and all the columns. We want to go for row and what? Only the key points. Okay. And then for column, only in the what? Key points. And so now we're just going to step through these. And so we are going to have 10 columns and 10 rows because we are only stepping through the key points.
Now, what do we do? We calculate the error. Is equal to the error plus. Now we've got to be careful. Remember, you don't care if it's too big or too small, it's still an error. So you need to make all errors positive by taking the absolute value. And then what would it be? Well, it would be the gesture matrix, right? Of what? The row element and the column element. And then minus what? Minus the unknown matrix. So what? A row and the column. Okay.
Now, what do we end up with? I compare this known matrix to this unknown matrix, the difference matrix, the distance matrix, the distance matrix of the known, the distance matrix of the unknown. And I count the difference point by point for those key points. And I add them all up. And I end up with a, what? I end up with a single number. Unknown. I'd better spell that right up here. Unknown. Now it's right there, right? And now I end up with what? I end up with a single number. And I better return that single number, error, to whoever calls this. Does that make sense? Okay. I hope that makes sense.
All right, so now we need to get down into our program. And I gotta have the same program running whether I'm kind of like training or I'm comparing. Either I could be training on it, like I could teach it something, or I could be comparing that taught something to something new. But all of that has to be going on in this same loop because we got to keep the camera going, right? It's like a juggler, we got to keep reading a frame, doing something with reading it, doing something, showing it. So I've got to be in the same loop. But as I go through the loop, it's got to know, am I training or am I not training? And so before we go into this loop, what I'm just going to do is I'm just going to say train is equal to true. Because the first time that I go into the loop, it doesn't know anything, right? And so it needs to train. And then if it trains, then I'll set train to false. And then the next time through, it will start doing something different. But you see, there'll be kind of like two different chunks of code inside of this while loop that we will be executing.
Okay, now, in all cases, what we want to do is, in all cases, we want to, in all cases, we want to read it. In all cases, we want to find the hand data. Okay. In all cases, we want to read it, and in all cases, we want to see if there's a hand there and analyze it if it is. Now, if train equal equal true, so if we're still in the training loop, what do we want to do? Well, we want to say, if there's actually hand data. So if hand data is not equal to nothing. So like, if hand data is nothing, you don't want to train. If there's nothing. So only if you are training, and only if there was a hand there to analyze. So if there is hand data there, then what do we want to do? We want to, hmm, we want to train. Okay. And so what we want to do is we want to tell the person, print, we're going to say, "Show your gesture." So this is just going to train on one gesture. This is just going to train on one gesture. This next week, we're going to have to make it more sophisticated. But for now, let's just train on one, just to get our, our algorithm going. And then we'll have to do better later. So we're going to tell the person, "Show your gesture, and then press T when ready." All right. And so he's getting his hand in position. You don't want it training while he's getting his hand in position, but then when the hand is in position, then press T, and then it trains on that. Okay. So it tells it to press T when it is ready to go.
So now we gotta see, is did he press T? So we say, if cv2.waitkey. You guys probably copy and paste this so much, you forgot you can do it for other things. But if waitkey will wait for one milliseconds, and then remember we've got to put that crazy mask on it in Windows. So we say, and 0xF, like that, and 0xFF, if that is equal equal the ord of what? The ord of T, because that is what we were looking for. We are looking to see if he pressed T. And we better put the T in quotes, like that. Okay.
So he's got his gesture. First of all, we are training. Yes. And then there was, you know, there is a hand there to look at. Okay. And now if your hand is ready to go, then you press T. And then when you press T, what do you want to do? You want to train. So I'm going to say known gesture, and this is going to be that whole distance matrix. Known gesture, that whole distance matrix. How do I get it? By finding distance of what? The hand data. Now, remember, that hand data that is returned, it there might be a bunch of different hands. So that hand data is going to be an array of an array. And so we just got to tell it to do the first one, to do the first hand. Now, we don't want it confusing my face with a hand. So really, we only want to find one hand. So when we create this find hands up here, MP Hands, we said here to find two hands, up to two hands. Let's say, just find one hands. So that forces it, there will only be one hand there. But it will still be an array inside of an array. So we've got to do this. Go back and watch that earlier video where we did that, where we set this thing up. But this still takes the zeroth element here. And now find distance, it doesn't like because it should be find distances, which is what we called it. So now at this point, we should have the distance matrix, 21 rows and 21 columns, for that gesture that we just trained on. [Music]
Okay, now that we have trained, now that we have trained, so right here, when we get to this point, and we have this known gesture, and we go out and do that, we have trained. So now we're going to say train is equal to false, because we don't want to train over and over. Now that we know it, we're going to want to then go in and look at the other gestures. But just for fun, here, let's print, let's print our known gesture. All right, let's print our known gesture, just to make sure that it makes sense, that we haven't made a typo in here. Okay, I think this might actually run. Let's see if it'll run. What is this? Train true. Ah, you do not need a colon when you say train is equal to true. Okay, let's try it now.
Okay, now it says, "Show your gesture, press T when ready." So we're going to do the "Power to the People." Okay, we're going to do the "Power to the People," and then it's going to want T. Now, when I press T, it should train, and then it should print that matrix out. So let's see what happens. And it crashed. It didn't like my NP zeros. Why did it not like my NP zeros? NP zeros. Why does it not like that? This is perplexing. Ah, does anybody have any idea why it doesn't like that? NP dot zeros, length of hand data, comma, length of hand data, data type is float. Let's go back and look at that error message again. Line 54. Okay, so you see, I'm passing it only one hand data. So when it gets the hand data, it's just one. Floats is not understood because float singular. All right. Okay, so we learned to debug things here.
Okay, so now we're going to give it the "Power to the People." Now it wants me to press T. And now it should give me a distance matrix. So we're going to try that. Boom. It gave me a distance matrix. Now I'm going to actually quit out of here, and let's look at this distance matrix. So what you can see is this is, I need to bring that up a little bit, so you can see that better. This is really exciting to me because it looks really good. Okay. So this is the last row of distances. Okay. So this would be like the last row. It's looking at the distance between element 20. It's looking at the distance between element 20 and all the other ones. And so what this is saying is, what this is saying is, the distance between element 20 and element 0 is 62. The distance between element 20 and element 1 is 76. And the distance between element 20 and this is like 1, 2, 3, out to 20 is 0, because that's the distance between it and itself. And so this is element 19, and therefore the 19th element is 0. And the 18th is 0. If we go all the way to the top, this is the 0th element, and the 0th one goes to 0. So you see how the 0 is here, it's here, it's here, because that zero diagonal is always that point compared to itself. And so this really is kind of looking like it. It's looking very promising. It's looking like that is probably got a real set of data there. We don't see anything crazy happening. All right, that is really good.
So now, what I want to do is, so that's if training is true. What do we want to do if training is not true? So if train equal equal false. In that case, if train is equal to false, if train is equal to false, well, again, we better make sure that we have a hand. We better make sure that we have a hand. So if we're going to say, if hand data is not equal to the empty set. So I could train on the "Power to the People," and then I could drop my hand out. And then sure enough, now train is going to be false. But I still, I don't want to do something unless I actually have hand data. So if training is false, and if we actually have hand data, okay, what do I want to do? I've got hand data. Well, I want to find for that new hand, that unknown hand, I want to find what? I want to find the distances. But now instead of known gesture, what is this going to be? The unknown gesture. Okay. And the unknown gesture is going to be equal to, how do I get the unknown gesture? I do find distances. And what is it going to need? It is going to need hand data. And remember, we got to pass it just the first element, which is the only element, which is 0. And so now I should have, I should have the unknown gesture. All right, I should have the unknown gesture.
Now, to get down here, I can't get here unless I trained. And I can't get here unless I have an unknown gesture. So now I have a known gesture, and I have an unknown gesture. I want to find the, what? I want to find the, what? Somebody tell me. Okay, I want to find the error. And the error is going to be equal to find error. Do I need to? Oh, okay. This is where I pass it the key points. So what does it need to know? It needs to know the, it needs to know the known gesture. What else does it need to know? The unknown gesture. And then what else does it need to know? The key points. [Music]
Guys, now if we wanted to try different training and different accuracy, we could just come in and we could change this, and we could try it for maybe there would be even a smaller number of points that this would work for. Okay. But now we have the error. All right. So now, what let's do? What should we do? Let's just show the error. Let's just look at it. So at this point, we're just going to look at the error and see if we have any type of signal there. If we have any type of meaningful signal. And I think it would, instead of printing it, I think it would make more sense to do a put text. So we are going to do a cv2. And you know how much I hate the text, but I think that is the best way to do. We're going to do it cv2.puttext. And where we want to put the text, we want to
To put it on frame, okay? And then, what do we want to put? We want to put error. Error is the number. Put text wants a string, so we've got to take the string of the error. Also, it would be easier to read if it isn't, you know, if it's not 10 decimal places. So, we're going to round error. We're going to round error to zero places. So, this will just be an error 10, 11, 12, 13. It's not going to be 10.134567. Okay, so it'll round the error to zero and make it a string. Okay, now, what do we want to do? Where do we want it? Well, I think it would be good to put it over about 100 pixels and down about 175 pixels. So, this is where we want to put it. Over 100 and down to 75, I think would be good. Uh, let's make it 175 like that. Okay, now we've got to give it a font. CV2 dot font, I like Hershey's simplex. Okay, let's give it a height of three. Okay, let's make it blue. So, that's going to be 255, comma, zero, comma, zero. And then, let's make it pretty heavy. Let's give it a weight of eight. And now we've got to close that whole put text. Let's make sure we have our parentheses right. This opens the put text. This closes the put text. I open the string, I open the round, I close the string, I close the round. And that looks pretty good.
All right, let's see if this works. Okay, I found my hand as I'm drinking. That would not. Maybe this would be coffee lover. This would be the Chester coffee lover. Okay, so it says, "Show your gesture and press T." So, I'm going to give it the power to the people. And now I'm going to press T. Now, look at that. When it looks at my power to the people, I thought I told it zero on that crazy thing. Why is it still giving me a decimal? Round error to zero. I really wanted that rounded to zero, but it's going to round it to one decimal place. I'll fix that. I'll fix that. You know how I'll fix that? I'll make it an int like that. That will get rid of all those decimals. Okay, let's run it. Why is it not asking me for? Okay, it wants a gesture. Okay, now it tells me to press T. So, I'm going to give it the power to the people gesture. Okay, now, do you see the error? Looks like it is sitting around 100, you know, 100 to 200. Now, let me give it a different gesture. Like, let's give it the peace. Okay, the peace error is 4500. So, if it saw a peace, it would know that it's not power to the people. At least we've got the data there. Okay, now you see the power to the people now is not as good as it was before. All right, I think what you can see is it kind of matters that you keep the same distance. Okay, so you see my error is greater. All right, we'll, we'll worry about distance later, but my power to the people, my error now is about a thousand, but the number one is 3000. Power to the people is 1000, 4000, 1000. So, you see, you would know if you set a tolerance of say, 1500. I don't know why my errors are creeping up. Okay, so if you had a a tolerance of about 1500, you would know that this is power to the people and this is not power to the people. That is not power to the people. That is not power to the people. That is not power to the people. Guns up is not power to the people. Number one, two, three, four, five. Those are all not power to the people. But here, this is power to the people. Do you see how that works? Let's try training it on a different one. Okay, okay, so now what we're going to train on is, let's say, go forth and prosper. Okay, so go forth and prosper will be T, right? Let's compare that to the number one. Okay, let's compare that to hookum horns. You see that's 4000 for hookum horns, which is very different than go forth and prosper. Okay, and it's very different than power to the people, and it's very different than thumbs up, and it's very different than hookum horns. But here, you see that is below a thousand. So, it's like go forth and prosper. That is pretty cool, isn't it? That is really pretty cool. So, you see, now we've got something that it seems to recognize the thing that it was trained on. Well, what would we need to do now to take this to the next level? Well, what we've got to do now is we've got to create an even bigger array, and that bigger array is going to be your training set, and that bigger array is going to have the difference matrix for the first trained gesture, the difference matrix for the second trained gesture, the difference matrix for the third trained gesture, and so up front in the program, you're going to say how many gestures do you want to train on, and you would say five, and it would show me that it would like, like, show me the first one, show me the second one, show me the third one, the fourth one, the fifth one. Now your training is complete, but you've got an array of distance arrays. Okay, you're going to have an array of distance arrays. Now, what you're going to have to do is when you're looking at a gesture, you're going to have to take your unknown gesture and not just compare it to your one known, but you're going to have to step through those known, and you're going to have to create an error, and then if the error is below a certain tolerance, you're going to say, "Oh, it's power to the people." Okay, or, "Oh, it's peace," based on the things that you trained on it.
All right, guys, I've given you a pretty, pretty big hint, but what I want you to do is I want you to go away and now next week, come back with something that you can train very easily, and then it will find those things. All right, and right now, for right now, let's just kind of leave it like we're going to cheat a little bit, and we're not going to be coming like this because you can see here, all of those distances are going to be larger because my hand is closer, and all the distances would be smaller because my hand is further away. We'll have to worry about that then in the following lesson, but right now, we're going to keep a constant distance from the camera, and we're going to see if we can make this thing work.
Okay, guys, man, I hope this has been a little bit of a tedious lesson, but it really, it seems to me like it's really clear, and I was just kind of excited how you could sort of see for the known gesture, kind of like a 200 or 300 or maybe up to a thousand error, and then you go to the next gesture, and boom, like 4000 or 5000. So, I think we have a pretty big signal here with this algorithm. Again, you guys, let me know whether you were able to do it, or maybe somebody came up with a better way to do it than I did, but leave a comment down below. Post the work you guys did on YouTube. Share it in the comments down below. Okay, guys, I hope you are having as much fun taking these lessons as I am making the lessons. I am really, really excited about this lesson that we did today. I'm really excited about next week coming up with a generic way to detect gestures. If you like this, give us a thumbs up. Leave a comment down below. That always helps us with the old YouTube juice to get this video in front of more people. Uh, you know, and, and share this with other people because really, the world needs more people who understand engineering and are doing coding, and fewer people sitting around watching silly cat videos. Paul McQuarter from TopTechBoy.com, coming to you from the shores of the mighty river. Now, you guys have a great day. [Music]