Transcription
Welcome! In this video, where we are going to implement a new managed walk from scratch. So, we're not going to use any automatic differentiation library such as PyTorch, TensorFlow, Keras, or Jax. We'll really start from scratch, just using pure Python. But that means that if you are using another programming language such as Java or C++, you will be able to just adapt the code that we will be doing in this video on implement it in another programming language because we'll not use any, uh, animation learning library dedicated to Python.
These are the results we'll get by the end of this video. So, we'll reach a 98% test accuracy on the MNIST classification problem. Training will be about 20 seconds on the CPU. So, this is a very efficient implementation. We tried to implement it in only 100 lines of code, but there we'll use all the tricks that are needed when you implement your networks, such as the Jacobian-vector product, the log-sum-exp trick. So, we'll use all those tricks that are much needed, and we will have a very efficient implementation.
Okay, so let's move in directly into the implementation. I see that we're going to implement it from scratch, but we are still using a few libraries. Um, so for example, you see that I'm using NumPy. So, I'm using NumPy or need to have access to a few functions such as, for example, the matrix multiplication. But if I really wanted, I could get rid of NumPy very easily. So, all the functions that I will use NumPy for, I could implement them myself. For example, implementing matrix multiplication is very easy to do. I could do it on get rid of NumPy. So, let's say that was using Java, you can just replace, uh, with everywhere you see NumPy, you can implement the function yourself. That's very easy to do. If people are requested, I could do it in another video very, uh, without any library. But these are very, uh, those libraries are, yeah, again, what we'll use them for is very easy to do ourselves.
Tqdm is just to see, uh, the progress of training. So, we can totally get rid of it. While you're also using SciPy to have access to the log-sum-exp function. So, this is a trick that makes the, uh, the computation more stable numerically. But again, the log-sum-exp trick is very easy to implement. We could have implemented ourselves. And finally, we will be using Keras, but not for, uh, for implementing neural networks, just for loading the MNIST data. So, that you don't need to download it yourself on your computer. Uh, just calling this function will do everything for you.
Okay, so we're using four packages, but we could very easily get rid of them, and maybe that would, that would require maybe 50 more lines of code to implement everything that those packages are doing that we'll use. But anyway, let's move in into the implementation. So, first, we can start with the MLP. So, it's taking D in, the number of input dimensions, the out is the number of output dimensions. An MLP have two main components: the weights and the bias. So, the weight matrix of size D in comma D out, or we can also, uh, model the transpose of this matrix, so D out comma D in. So, basically, we're just using NumPy to create this matrix randomly. But again, we could just use a for loop, a nested for loop to create this. So, we could very easily get rid of NumPy.
And what you are doing is, uh, it's called Xavier initialization. So, we need to find the proper distribution to draw our weights from, otherwise the network can get stuck in local minima. So, it's very important to find the right distribution to initialize your weights. And in our case, we're using the Xavier initialization. Other basically, right, we're putting our weights between, uh, between minus, uh, square root of 6 divided by the square root of D in comma D out, D in plus D out. So, this is the lower bound. And the higher bound is the same value, but with a plus rather than the minus. So, we have these two bounds, and we're drawing values randomly between those two bounds.
So, once we have the weights and the bias that have been drawn randomly, we can implement the forward function of the MLP. So, was the MLP, it's a very simple function. It's W times X plus B. So, I'm just implementing to matrix multiplication. X, I'm multiplying X with the transpose of the weights and I'm adding the bias. So, again, I'm using NumPy to do this matrix multiplication, but I could implement it myself. That will be maybe five lines of code. On line, I'm storing X for the backward path. So, we'll see that in a moment.
So, once I've implemented the forward pass, that is very easy, it's just too simple. I'm just taking the equation of the MLP, implementing them, and now we can move on to the backward pass. So, for the backward pass, I need to compute Delta W and Delta B, so the gradient of the, um, gradient of the loss with respect to the weights, so that I can use those to do gradient descent. So, the first thing I need to do is to compute the Jacobian of the forward pass. So, that will give me, for example, the derivative of the error of the output of the output with respect to the weights. But what I want is not the gradient of the output with respect to the weight. I want the gradient of the loss with respect to the weights. So, what I need to do, I need to compute, uh, to take the Jacobian and to multiply it with the output, grad out. And basically, this is the chain rule. The upcoming layers will fit me, I will receive the gradient from the upcoming layers, so the gradient of the loss with respect to the output of this MLP. And then I need to do matrix multiplication with the Jacobian. So, that's I have the gradient of the loss with respect to my weights.
If you want more detail about that, I have a Fiverr course on Udemy where I go, where I go deeper into all those, all those equations. So, do note that we've looked at it, if you want more details. But you can see that I do not, in my implementation, you do not see the Jacobian. It's because I'm not storing directly. I'm not storing the Jacobians at all. I'm directly computing the results of the, of taking the Jacobian and multiplying it with grad out. Basically, I can compute the results analytically. And the analytical results will be grad out transposed multiplied with self.x. Basically, this is called the Jacobian-vector product, and this is a trick that is needed in neural networks. If you do not implement it, the memory of your implementation will explode. So, you won't be able to train your MLP even on very simple problems such as MNIST classification. So, it's very important. And sometimes people just plug this equation and do not really understand what's going on. So, first, we compute the Jacobian, we take the product with grad out, and then when you rewrite the results of this, this is this equation. So, this equation is called the Jacobian-vector product, and it's very important when you implement neural networks, otherwise, again, your implementation won't work because it will require too much conditional resources.
We can do the same for the bias. Compute the Jacobian-vector product, which is grad out dot sum over the first dimension. And then we also need to return the gradient of the input with respect to the output, no, the gradient of the output with respect to the input. So, that the previous layers can also use this value on the chain rule to compute the derivative.
Okay, so this is the MLP in short. Again, if you are more detailed, our IV course, and I will put the link in the description of the video. So, now that we have the MLP, we can compute a sequential network. There are similar functions in PyTorch, in TensorFlow, where we can combine multiple blocks between them. Because by the end, a neural network is just like a Lego, where you put building blocks between them, and then you have something that is nice. So, basically, we will combine multiple MLPs, and we will also use activation functions. So, a sequential network will take a list of modules. So, for example, the MLP, in a moment, we'll implement activation functions. So, we take a list of blocks of modules, and we can put them together.
So, the constructor is very easy. We just store the blocks. And for the forward pass, it's very easy. We just propagate the input to the blocks. So, basically, we start from the layer that is from the first layer to the last layer, and we propagate the inputs. So, this is the forward pass, very easy. And the backward pass is the opposite. So, we start from the output, from the last layer, from the loss, and then we backpropagate the loss from the last layer to the first layer. So, we iterate, we reverse the blocks because we want to start from the last block, and we backwards. So, we take the gradient of the loss, we feed it to the last layer, we'll receive a new grad out, and then we do this on backpropagate the gradient from the last layer to the first layer.
Okay, so this is the sequential channel. Pretty easy, just combining the blocks together. Now we can move on to ReLU. It's really similar to MLP. So, basically, ReLU is just the maximum between X and zero. So, but for any values that are lower than zero, we set them to zero. Again, we are using NumPy for this, but you could do that very easily in Python, just with a for loop, for example. So, you could get rid of NumPy.
All through the backward pass, again, you can compute analytically the Jacobian-vector product. And once it's been computed, these are the results you get. Um, so the ReLU is used in the hidden dimensions. And if we want to classify, classification, we need an activation function for doing classification for the last layer. Usually, we use the softmax or the log-softmax. In this video, we'll only do classification. But again, if you are interested in regression, I refer you to my course.
So, for the log-softmax, so what we want to do is to, um, what you could do is just compute the softmax. So, the softmax equation is, is, uh, you take the exponential of your inputs, and then you divide by the sum of the exponentials for the other classes. So, what you can do is just compute the softmax and then take the logarithm of it. Because in this case, we're going to log-softmax under the softmax. But if you do that, you can have numerical instabilities, and you will see that during training, your loss will oscillate, or you will have NaN values. So, if you want to avoid that, you can, there is what is called the log-sum-exp trick. Maybe I will not go into the detail for the log-sum-exp trick. Maybe you can read on Wikipedia to have more detail about it. But basically, yeah, this is a trick so that the result will be more numerically stable. So, basically, if we rewrite analytically the log-softmax, we can rewrite it as a function of the log-sum-exp trick, and we will have results that are much more stable.
So, this is the forward pass. You can either just take the log of the softmax, but very likely you will have instabilities during training, or you write it analytically, and you get this equation. Then there was the backward pass. Again, I will not go into the detail about those equations. They are detailed in my course. But if you, these are just the Jacobian-vector products that I just compute analytically on paper, and then I plug them in, like in my implementation. Maybe that can be a great exercise. You can try to compute them by yourself and then make sure that you get the same results as I do.
Okay, so we have our activation functions, we have our layers. The last thing we need is the loss. So, with the log-softmax, we usually use the negative log-likelihood loss. So, basically, we want to maximize the likelihood, the total log marginal likelihood of our observed data. So, for a given, for predicted, when predicts probabilities or log probabilities with the log-softmax, and with the label of the true classes. What we need to do is to take the predicted probabilities at the given index. Let's say that your true class is cats, let's say it's index 0, and that we have predicted a probability of 0.8 for the cat. You need to fetch this probability, and you want to maximize it. You want to maximize the probability that the neural network assigns to the true class. But because when you are doing gradient descent, we are doing a minimization, we are doing a minus there. We want to minimize the opposite of this value. So, in short, we want to maximize this value on the rightmost side. But because later on we'll do a minimization, we are doing a minus there so that we minimize the opposite of this value, so we will maximize it.
So, this is the forward pass. You've seen that the forward pass now we have two values instead of one, for example, if we compare to ReLU. Because now we have the predicted value and the true value. For the backward pass, we don't have an input as opposed to ReLU or MLP. Because the loss is at the very end of the pipeline, and the forward loss does not receive a gradient as input. The loss, we will start from the loss during the backward pass. We will start from the loss and go back to the initial layer of the neural network. So, the loss does not take an input. The loss computes the gradient and feeds the gradient to the other layers. Yeah, again, we are just computing. This time there is no Jacobian-vector product because the loss does not receive a gradient as input. We are just computing the Jacobian of the loss. Maybe again, you can try to compute it by yourself and to compare with this implementation. But this is the Jacobian of the negative log-likelihood loss. So, once you've computed the Jacobian, you can just return it.
And either I've, I'm just, just a Python. This is just a Python thing. I'm using a call function so that I can later on, I can do an annual loss parenthesis or on give some inputs and it will call by default this forward function. Yeah, we'll see that in a moment. But if you are familiar with Python, you see what I mean.
And finally, we can create an optimizer. So, the optimizer will take, will update the weights of the MLP because for now, we've just computed the gradients, and now we need to use those gradients to update the weights W and B. So, we're creating an optimizer with a learning rate, and it's also taking as input a sequential network, a sequence of modules. And we are creating a step function that will update the weights of the neural network. So, we iterate over all the blocks in the sequential network. And there are only for now in our codebase, there is only the MLP that has learned parameters. For example, the ReLU does not have any parameters that can be learned. But sometimes some activation functions do. So, in this case, for example, we need to check if the module is MLP. That means that it has learnable parameters, and we need to update weights, update them with gradient descent. And this is what we do. We take W, and we do W equals W minus learning rate times Delta W. So, basically, a basic gradient descent. If, for example, you want to add all the layers in your implementation, you can add them and then add if on updated weights of your older modules.
We can create a train function to train the neural network using the optimizer. Basically, this train loop will be pretty easy because we are just using, we are doing supervised learning with train X, which is our training data, train Y, which are our labels, with the loss function, the number of epochs, which we train on, the batch size. So, we iterate over all the epochs. For each epoch, we sample an ID. So, yeah, some data from our dataset. So, X and target, we sample them randomly because we're doing mini-batch gradient descent. Then we are making predictions with the neural network. So, we predict probabilities or log probabilities. We compute the loss thanks to the targets. And then, yeah, we can log the loss. We can compute the gradients of the loss by calling the backward function on the loss. And then we can backward all the older modules of the neural network by calling model.backward. And the model.backward will call sequentially the backward function of all the modules.
Okay, um, okay, that's great. Oh, okay. You see, loss function, I do not need to do dot forward. I can just put the parenthesis because I've added this, if I did this call function, but this is just Python. Okay, so once I've done that, I can update the weights using the optimizer.step. So, all the gradients that will be computed on this line, line 112, will be updated with the optimizer.step function.
Now we can put all those pieces together. So, first, I'm just loading the data. So, I'm using the load data function from Keras. If you are using, for example, C++ or Python, you cannot, you don't have access to this library. You can just download the data on your computer and load them by yourself. And then doing this processing. So, we want to feed to the neural network values that are centered at zero and that are not too large. So, in this case, we are processing the data so that they will be between minus one and one. And we also reshape the data because the data are 2D, these are images, but MLP takes 1D data as input. So, we reshape them as 1D vectors.
So, we can create an MLP. We create a sequence using the modules we've created. So, first, we start with an MLP that is some input dimension, some output dimensions. Then we put our ReLU. Then we put a second MLP. Then again, we do ReLU. And finally, an MLP that takes 64 hidden layer median units and that predicts 10 classes because we are using, we are doing MNIST classification. There are 10 digits, and for predictive probability for each of those digits. And then we're using the log-softmax because we want the predicted values to represent something. We want them to represent probabilities. So, we want them to be greater than zero and sum to one. So, we're using the softmax for this, and then we take the logarithm of it. So, we use the log-softmax. We create an optimizer. So, we feed our MLP and we give a learning rate of 1 times 10 to the minus 3. And then we can feed everything in the train function. So, we feed the MLP, the optimizer, and the dataset. And then we have a trained MLP, and we can check the accuracy of the trained network.
So, now we are doing prediction, but on the test dataset. So, this dataset is not being seen during training. That's very important. So, I'm making a prediction. So, I have 10 values that are outputted by the MLP, 10, one probability for each class. And we check which one is the greatest with the argmax function. Let's say that I assign more qualities to the last digits, 10 or 9, that means that I think that the digit I've seen is the digit 9. So, I'm making a prediction, and I'm checking if my prediction is equal to the true class, to test Y. Then I do accuracy plus equals to one. I, I that means that I've just made a right prediction. And then if we divide the accuracy by the number of digits that we've used for testing, we will get our test accuracy in percent. And if you want this code, again, it will take maybe only 20 seconds. Only will have run accuracy of about 98%.
So, I really hope this implementation, this video was helpful to you. That will help you to understand neural networks in more detail. I think it's very important to understand how everything is going, what is the log, the Jacobian-vector product. Yeah, I think it's very important to understand the mechanism of your networks. Please let me know if you want more videos like that. Again, I have, I went really quickly into all those modules. I have a course that goes much more in depth with more slides with more details. If you are interested, please consider leaving a thumbs up if the video was helpful to you. Want to subscribe for more videos like that. Thank you.