Learn Python for Data Science – Full Course for Beginners

Get started with data science using Python. This course covers essential tools like Pandas and NumPy, plus data visualization, cleaning, and machine learning techniques. Perfect for beginners. You'll gain the skills to analyze and interpret data effectively. Frank Andrade created this course. This is my Python for data science course for beginners.

In this course, you'll learn everything you need: Pandas, NumPy, visualizations, data cleaning, machine learning, and more. In this course, I not only teach you how to use Python for data science, but we also solve many exercises and work on four projects to put into practice everything we learn. I leave the link to the code and my Python cheat sheet I created for this course in the description of this video. In my cheat sheet, you'll find all the concepts we learn in this course, as well as code snippets that you can use for solving exercises and projects.

All right, now let's start with this course. Anaconda is all about data science. It brings all the tools used in data science like Python, Jupyter Notebook, and Pandas with just one install. And in this video, I'll show you how to easily set up Anaconda on your computer.

So to download Anaconda, we go to anaconda.com and click on get started. Then we choose the last option, download Anaconda installers. And then we have here the different Anaconda installers. So there are Windows, Mac, and Linux. So in my case, I'm going to choose Mac and I'm going to choose the 64-bit graphical installer. So now I'm downloaded Anaconda, and once it's downloaded, I'm going to click on it, and a message will pop up. You just have to click on allow, as I'm going to do right now. So just click on allow and then click on continue until the installation starts. So I just click continue and then agree and then continue, and it's going to start installing Anaconda. In case you're on Windows and you're installing Python or Anaconda for the first time, make sure to check the first box you see now on screen. So I'm going to speed up the video now.

Okay, the installation is almost done, and now it's telling me that an account that works with PyCharm, and now I'm just going to click on continue to finish the installation. So I click on continue, and then we'll see just a summary of what was installed. And now I'm going to close this window and I'm going to open Anaconda. So I'm going to locate the icon. It's a green icon, uh, this one that you see here, and I'm going to open Anaconda. I'm going to wait a couple of seconds, and let's see what was installed. So here we have the Jupyter Lab and Jupyter Notebook, which are widely used in data science. So I'm going to launch Jupyter Notebook.

So here it's opening Jupyter Notebook. Let's give it a second. And now we open a new notebook with Python 3. So, Python 3 was installed too. And here I'm going to import pandas. So, I write import pandas as pd. And if the code runs fine, then the installation was successful. So, let's wait a couple of seconds. And now, as you can see, it's working. So, I can create a data frame without any problem. So, now let's go back to Anaconda to see if Jupyter Lab is working fine too. So here is Jupyter Lab, and I press on launch and just wait a couple of seconds, and now Jupyter Lab is open. So here is the file I was using before, and it has the data frame, and everything is fine.

Finally, let's see how to install a new library. So we go to environments, and on the right side, there is a search box where you can write any library you want to install. So here I'm going to check if pandas is installed. So you can see pandas, and then there is also NumPy and scikit-learn. And if you want to install a new library, you just have to click on the dropdown and choose the option non-installed. By the way, you can install a new environment using the Anaconda Navigator. You just click on this create button, and then you will see this window, and here you just have to write the name of the environment and choose the Python version you want to install in this environment, and that's it. Now you can start working in your data science project.

In this video, I will introduce you to the Jupyter Notebook interface. Jupyter Notebook is an open-source web application that allows us to create and share documents that contain live code, equations, visualizations, and text. This is the perfect text editor for doing data cleaning and transformation, data visualization, and data analysis. This is why Jupyter Notebook is widely used in data science and also machine learning. As you might remember, we installed Jupyter Notebook and Python with the Anaconda Navigator. And this means that we already have installed some popular libraries used in Python for data analysis. By the way, one of the alternatives of Jupyter Notebook is Jupyter Lab. Both are similar, but we're going to use Jupyter Notebook in this course because of its simplicity.

So, let's open Jupyter Notebook. And to do that, we have to click here on the launch button. So I click here. And now we wait a couple of seconds. And now we have here the interface of Jupyter Notebook. So I'm going to maximize this. And by default, Jupyter Notebook opens the root directory of your computer. It's a good idea to create a folder where all your Python scripts will be located. In my case, this folder is called Anaconda scripts. So, I click here, and now I can navigate through the folders. And the folder I'm going to use for this example is this one that says my course. And here we're going to create our first Python script. To do that, we click here on the new button. So, click here. And we have to click on the first option that says Python 3. There are other options like text file, folder, or the terminal, but we're not going to use these options in this course. So click on Python 3, and now we have a Python script powered by Jupyter Notebook.

So here on the right, you can see that it says Python 3, and also there is the Python logo, and on the left, you can see here the Jupyter Notebook logo and also the name of this Jupyter Notebook file. We can change the name of the file by clicking here on Untitled. So I click here, and I can change it to, let's say, example. So I write example and I click on rename. And now we rename this Jupyter Notebook file.

All right. Now let's navigate through this menu bar that we have here in this Jupyter Notebook file. So the first option is the file. In here, we can create a new notebook with Python 3. So if we click here, we're going to open a new Jupyter Notebook file from scratch as we did before. Then we have that open. And in this case, we can open a Jupyter Notebook we created before. We can also make a copy of a Jupyter Notebook and then change the name. We can save a Jupyter Notebook file and rename the file as we did before. We only click here and rename the file. Then we can save all the progress we make in Jupyter Notebook. For example, after writing many lines of code, you can save all the progress you make by pressing Ctrl+S or Command+S on Mac, and you're going to create a checkpoint, and later you can revert to a previous checkpoint by using this option here. So here you will see many checkpoints, and you can revert to a previous checkpoint. By the way, by default, Jupyter Notebook makes saves every 30 seconds or maybe 1 minute. So there is no need to press Ctrl+S every time. So keep that in mind. Then we have other options that I don't use so much, like print this Jupyter Notebook or export that Jupyter Notebook file to HTML or PDF and so on.

Okay. Now let's see the second option that says edit. And here we can edit all the cells we have here in this Jupyter Notebook. By the way, here what you see here on the screen is a cell. So we can edit with this edit option. For example, we can cut cells, we can copy cells, paste cells above, and delete cells. On the right, you can see the shortcuts that we're going to see in the next video in detail. And well, you can check all the edit options that you can perform on Jupyter Notebook here.

Then in the view option, we can toggle the header, the toolbar, and also line numbers. So here if I click on toggle header, the header is going to disappear. And if I click on toggle toolbar, this toolbar disappears too. Also here in toggle line numbers, we can show here line numbers. So if I write anything, we can see that it says 1, 2, 3, and so on. And I'm not going to use this for this course. I'm going to leave it with the default options. So here I'm going to revert to the original option. So without line numbers, and I want to show the header and also the toolbar, but you can personalize it as you want.

Next, in the insert options, we can insert cells above or below. We only click here. And well, we're going to see the shortcuts later in the next video. Then we have the cell options. We can run cells or run all the cells in this Jupyter Notebook file. And then we have the kernel option. And a kernel is a computational engine that executes the code contained in a notebook document. When we open Jupyter Notebook, a kernel is automatically launched. And we can interrupt this kernel by clicking here. So by interrupting, we can pause the execution of a code. We can also restart everything and do more things here. Sometimes, for example, I interrupt the kernel when a line of code or a cell takes too much time to execute. And well, you can do the same here with restart or interrupt.

Then we have the navigate option that doesn't actually have anything here; widgets that I don't use so much; and well, help that I think it will send you to the documentation of Jupyter Notebook, and you can read it if you want.

All right, here then we have the toolbar, and here you will find some shortcuts of the menu bar that we've seen before. For example, here you can save and make a checkpoint. So here I click here, and as you can see here, it says checkpoint created or something like that. Yeah, checkpoint created and the time that it was created. Then you can here with this plus button insert a cell below. So I click here, and as you can see, we can insert a cell below, and also you can use shortcuts, but that I'm going to show you in the next video. Then we can cut selected cells with this button. We can copy a cell with this button. And also we can paste cells below. Also, we can move a cell above or below. For example, I'm going to write anything here. And this cell I can move it above with this button or below, as you can see here. Then we can run this code. For example, I can write the number one and then run the code. And as you can see here, the code could run, and it shows the number one. And well, those are some of the frequently used buttons in the toolbar. And that's everything you need to know about this Jupyter Notebook file.

Okay. Now, before finishing this video, I'm going to show you some other options that you can find here in the Jupyter Notebook interface. And here you can see that there are some other options. So, right now we are in the files tab. And we can change to the running tab here. And here you can see all the currently running Jupyter Notebook processes. For example, we can see here the Jupyter Notebook file we created and that we opened. So you can recognize that a Jupyter Notebook file is open or that is running because here the icon will be in green. So here if we go back to the files tab, we can see that this Jupyter Notebook file, which by the way has the .ipynb extension, is in green. So the icon is in green. So this indicates that the file is running and well, it was opened. So here we can see that it is open, and we can shut down this file. And this is different from closing this file. For example, here I have the file, and if I close this file, here we can see that the file is still running. Here we see running, and it's in green, and in the running tab, it still shows up. So if we want to shut down this file, we click here, and it says that there are no notebooks running, and we can see here that the notebook has a gray icon.

All right. Then we have the clusters tab. And this tab I don't use so much, and actually, it doesn't show anything here. And then we have the NB extensions tab. Here you can install any extension to personalize Jupyter Notebook even more. And we're going to see some cool Jupyter Notebook extensions in the next videos. And by the way, this NB extensions tab doesn't show up in some versions of Jupyter Notebook, but we can easily install it. And we'll also see how to install this NB extensions tab in the next videos.

Finally, we have this box that shows our directory. So here this folder indicates the root directory. So if I click here, we are not in the root. And if I click on the folders Anaconda scripts and then my course, I go to the folder where I was before. And that's it. These are all the things you need to know about the Jupyter Notebook interface.

Okay, in this video, we're going to see some cell types and cell modes in Jupyter Notebook. So first, we're going to open the Jupyter Notebook file that we created in the previous video, which is this one example.ipynb. So we click on it, and here we have the Jupyter Notebook file opened, and here by default we have this first cell in command mode, and we can say that this is command mode because here this blue color indicates that the cell is in command mode, and when we are in command mode, we can do things outside the scope of any individual cell. So basically, all the tools we see here in the toolbar, we can apply it in command mode. Also in command mode, we can apply some shortcuts that I'm going to show you later. And for example, if we want to see the shortcut window, we press the letter H in command mode, and we can see the keyboard shortcuts here. So here you can see all the shortcuts, and all the shortcuts that you can apply in command mode. Now I'm going to close this one, and also you can apply different shortcuts, like for example, if you press B in the command mode, you will see that there is a new cell because B is the shortcut that introduces a new cell below. Now if we press Enter, you're going to see that the color is going to change to green. So here we have green color, and this green color indicates that we are in edit mode, and the edit mode is for all the actions you will usually perform in the context of the cell. For example, introducing text or writing code. So here I can write, say, 1, 2, 3. So if I write 1, 2, 3 and then I click on this run button, I'm going to run this cell. And as you can see here, I run this first cell. And also after running the cell, you can see that we are again in command mode. So to go to edit mode, we press Enter again. And now we can edit the numbers we introduced. So for example, I can write 4, 5, 6, and then run again. And here you can see that the output shows 1, 2, 3, 4, 5, and 6. By the way, if you try to use the shortcut in edit mode, it won't work. Here, I press Enter. And now I'm on edit mode. And if I press the letter H, you can see that nothing happens. We don't have the shortcut window. And if I press the letter B, you can see that we don't insert any cell below. This happens because those shortcuts work only on command mode. So to escape this edit mode, we have to press the Escape button. So I press Escape, and now I'm again in command mode. So if I press H, we have here the keyboard shortcut. And if I press B, you can see that we inserted a new cell. And that's it for the command and the edit mode.

Now we'll see the cell types in Jupyter Notebook. In Jupyter Notebook, there are three main cell types, and we can see all of them in this dropdown here. Right now the type of this cell is code. So here it says code, but if we press here, you can see other cell types like markdown and Raw NBConvert. So we're going to see first the code cell, and it already has the check. So this one is a code cell. So now I press here, and now well, it's in code cell. If I press Enter, I'm in edit mode. And here I can introduce any code I want. So here I can write uh any number 999. And if I press Ctrl+Enter, we can see that here this is the input. And here we got the output of this code. We're going to see how the code cell works throughout this course. But now it's time to see how the markdown cell works in Jupyter Notebook. So here I'm going to this cell. And now I'm going to change the cell type. So I press here in the dropdown, and now I select markdown. In the markdown cell, we can introduce any type of text we want. For example, we can introduce titles. So if I uh delete this and press the hash sign, we can get title. So one hash it means title. So here I press space, and now I write title. Now I press Ctrl+Enter or this run button to run this cell. And here we got the title. By the way, you shouldn't get this wrong number because I just modify the default behavior of Jupyter Notebook. So mine enumerates the titles and subtitles. But in your case, you will see only the word title. And if you want, you can introduce also subtitles here. So for example, I'm going to insert a new cell with this button, this plus button. And now I'm going to move this cell up with this button here. So I press this. And now I'm going to change the cell type from code cell to markdown cell. So I go to the dropdown and select markdown. And by the way, you can change the cell type also with shortcuts. So if you're in command mode, you can press the Y button to change to code cell. So I press the Y button, and as you can see here, it says In, and this In with the square brackets indicates that this is a code cell. So here I can press Enter and introduce any code. Here I introduce numbers and press the run button, and here you can see that we have an input and an output. So this is a code cell. But now we can press the M button to make this cell a markdown cell. So now we press M. And here we are in command mode. So now we can get this markdown cell. And here you don't see the In word with square brackets anymore. So now I'm going to edit mode. So I just press here, or well, you can press Enter to go to edit mode. And now to introduce a subtitle, I'm going to write double hash sign. So I press hash sign twice. Now a space. And now I'm going to write the subtitle. So I write subtitle. I press Ctrl+Enter or the run button to run this cell. And we got here the subtitle. And we can also introduce text. So I'm going to introduce a new cell with a plus button. And you can also do it with a B shortcut. So I'm going to do it with a B shortcut. Right now I press B. And here I got this new cell. And we can move this with this button here. And now we have this cell in the position we want it. So here I can introduce text by converting this cell to markdown. So here I choose markdown. Now I press Enter to go to edit mode. And here I can introduce any text. For example, I can write hello. I press Ctrl+Enter. And now we can see that we have here this text. And finally, the last type of cell is the Raw NBConvert. And this type of cell is not evaluated by the notebook kernel. So if we convert this code cell to a raw cell, this cell won't be evaluated by the notebook kernel. So let's try here. I press Raw NBConvert. Now we can see that this looks like a plain cell. Well, this type of cell is not used that often. Actually, we're going to use only the code cell and the markdown cell in this course. And that's it. In this video, you'll learn the cell types and cell

Modes in Jupyter Notebook. Okay.

In this video, we're going to see some common shortcuts used in Jupyter Notebook. And we're going to start with the F shortcut. And by the way, to use these shortcuts, you have to make sure you're in the command mode. And to verify you're in the command mode, make sure that the cell has this blue color. Okay.

Now that you're in the command mode, you can press the letter F, and you're going to see this find and replace. So this first shortcut allows us to find a word in a cell and then replace it with another word. For example, I can write here the word hello. And here it found the word hello inside this hello world sentence. And now I can replace this word with the word say hi, for example. So here I write hi. And now in red we can see the match. And in green we can see the word that we're going to insert. So here let's click on replace all. And now you can see that it doesn't say hello world anymore but now it says hi world.

So now I press ctrl enter, which is another shortcut to run the cell. So you can press here on run or only press ctrl enter to run this cell. So I press Ctrl enter and now we run this cell. And another way to run cells is to press shift enter. But in this case, we're going to run and insert a new cell below. So now let's see. I press shift enter. And out here it ran this cell because now it says in and three inside square brackets. And here we can see that we have a new cell. Okay.

Now, another shortcut that is often used is the Y and M shortcut. So, now this cell is a code cell. And if we want to make this a markdown cell, we only have to press the M letter. So, we press M, and this is going to be converted to a markdown cell. And if we press the letter Y, this is going to be converted to a code cell. And also, you can change the heading here. You can make the heading bigger or smaller. So here I'm going to locate this cell. And now to make this one smaller, we can press the numbers. So if we press the number two, we can see that this one gets smaller. And if I press number three, the title gets smaller, for smaller and so on. So as you can see, the more hash signs, the smaller the text. So here I'm going to delete this hash signs, and one hash sign represents the biggest font size, which is the title. So now I press Ctrl enter, and now we have this in heading one. But if I press number five and then press Ctrl enter, we can see that now this cell has heading five and it's smaller. So now I'm going to revert to heading one. So I press one and then Ctrl enter. Okay.

Now we can navigate through the cells by pressing on the up or down keys on our keyboard. And as you can see here, we can navigate through all the cells here. Or we can also press with the mouse. We can press on the cells we want. Okay.

Now we can insert a new cell above by pressing the A key. So if I press A, we get here a new cell above. And if I press now B, we get a new cell below. Now if I press X, we're going to cut this cell. So I press X, and you can see that the cell was cut. And now if we press V, we paste that cell below. So I press V, and now we got this cell. And if I press shift + V, we get the cell pasted above. So I press shift and V, and we get this new cell above this cell I have here. Okay. Okay, now I can delete cells by pressing D twice. So I press D two times, and as you can see here the title disappeared. So now I try it again, and we don't have the title anymore. But now if we press the letter Z, we can undo those changes. So let's undo what we did before. I press Z, and we get here the title back. Okay.

Another useful shortcut is that Ctrl S that allows us to save the changes we made in this Jupyter notebook file. So I press Ctrl S, and you can see here that says checkpoint created. So I'm going to press again Ctrl S. And here it says checkpoint created, and here also says the time. And that's it. These are some of the most common shortcuts used in Jupyter Notebook. But you can see other shortcuts by pressing the letter H. So I press H, and here you can see more keyboard shortcuts, or you can also go here to help and then go to keyboard shortcuts here, and you get the same window. So here you can see a list of shortcuts for command mode and also for the edit mode. You can see the description of a shortcut and also how to do it in your operating system.

One of the typical ways to get started with a programming language like Python is printing a simple message. You can write any message you want, but it's traditional among coders to start with a hello world. So let's try it. Let's print our first message using the print function. The print function prints a message to the screen. So I'm going to write here print, and then I'm going to open parenthesis. Every time we use a function in Python, we have to open parenthesis. Well, in this case, for the print function. And as you can see here, the functions get a green color in Jupyter notebook. So that's how you can identify them. So inside this parenthesis, I'm going to write the message. So in this case, it's going to be hello world. So this is our first message. Now to execute this first line of code, we have to press control and enter or command and enter if you're on Mac. So I'm going to press this. And as you can see here, we have our first hello world. Another way to run this first cell is pressing here on the run button. It's going to have the same effect. So I pressed, and it ran. So as you can see here it says in, which it represents a code cell, and this is a markdown cell as we've seen before.

One of the advantages that Jupyter notebook has is that it allows us to print the last object in a code cell without specifying the print function. So for example, here I can print this hello world without uh writing this print function. So I'm going to copy this hello world message that it's uh inside quotes and I'm going to run this code. So just control enter, and as you can see here we have this message printed. So this is one of the advantages that has Jupyter notebook. If you do this in another Python IDE, it won't work. So here you can try yourself. You can write any message you want apart from the first hello world. You can try with your name. So we write print, then parenthesis, and we open quotes because uh we need to define a string. I'm going to tell you about strings a little bit later, but just so you know right now, and here for example, I can write my name. So my name is Frank, and I can print my name. Then I can print also numbers. So I print my age 26, and it's going to work too.

And besides writing code, you can also add comments. Comments are a useful way to describe what we're doing in our code. So here we can use comments. We just have to write the hash sign, which is this one. So you write hash sign and then you write the comment. In this case, I'm going to write my name and I'm going to say printing my name. So we know what our code is doing here in the first message we wrote. We can also add a comment. So we write hash sign and then we can say printing my first message. As you can see here the comments also have a different color. So so far we have three colors. This color for the comments, uh green color for the functions, and red color for the string. This is just a useful functionality most text editor have that allows us easily read code it now. And now I'm going to write any message. So I'm going to write uh for example again uh hello world and again to verify the type we can use the type function, parenthesis, run this code and we got the str that represents a string.

And one cool thing a string has is methods. We can apply different functions to strings as we will do in Microsoft Excel, for example. However, in Python, we use methods. A method is a function that belongs to an object. To call a method, we use the dot sign after the object. Let's see some string methods to change the case of the text. So here I'm going to write again hello world, but now I'm going to use some string method. So I write hello world, and in this case I'm going to use the upper method to make this uppercase. So I'm going to use the print function. But actually, we don't need to use the print function because as I told you before, uh in Jupyter notebook we don't need to use the print because it automatically prints the last line of code. So since this is the only uh line of code in this cell block, it's going to print it automatically. So we just run this uh cell, and we have hello world in upper case. So as you might expect, now we can also change the case of the text. In this case, it can be on lower case or title case. So I'm going to just copy and paste this twice. And here I'm going to write instead of upper I'm going to use lower and then title. So you can see how it's going to change the case. So here I'm going to run and let's see what happens. So as you can see here it only printed the last one uh because as I told you before it only prints the last one. And if we want to print the three of them, we have two options. So we can maybe here cut and paste on each cell or what we can do is to print each of them. So here for example I can do print here and I can do the same for them. So instead of using more cells we can print all of them. And here we can print this one too. Actually, we don't need them. We don't need it because it's going to print the last line. But just for the sake of this video, I'm going to print the three of them. So here, I'm going to run this code. And as you can see here, the first it has an uppercase, the second has lower case, and the third has a title case. So that's how you do it on Python.

Other string method that you can find Python is the count method. So I'm going to delete this and actually this one too. And we're going to see this now. So first I copy this and now I paste it here. And here I'm going to use the count. So the count method. So I write count. And then here I open uh single quotes and I write the letter that we want to count. So here for example I'm going to write the L letter. And what this uh string method is going to do it's going to count how many times this L letter is included in this string. So as we can see there are two L's. So it should set to two. So I run this code and actually is three because there are two in hello and one in world. So I was wrong.

And here uh another string method that you can use is the replace method. So we can replace one letter for another. So here let me copy this and I'm going to paste it here. And instead of writing count I can write replace. So here the first letter that we're going to see here is the letter that we want to replace. So in this case I'm going to change the L with O. And the second letter is the letter that you want to put in that string. So I'm going to use the U. So I'm going to change every time that an O appears here in the string. We're going to replace it for a U vowel. So let's try. So I run this code and now it says hello world but with U. And these are some of the most common string methods in Python. Okay.

Now it's time to learn something that you're going to see often in Python, which are variables. Variables help us store data values. In Python we often work with data. So variables are useful to manage this data properly. A variable contains a value, which is the information associated with a variable. To assign a value to a variable, we use the equal sign. So let's create a message that says I'm learning Python and store it in a variable called message_1. So here I write message_1 and we set it to the string I'm learning Python. So I open uh double quotes and here I write I'm learning Python. So this is string. We've seen this before. And this is the variable. And we assign this value to the variable using the equal sign. Now I'm going to run this. And as you can see, nothing happens. But actually, we just assigned that string to the variable message_1. Now, if we want to obtain the message, I'm learning Python, we only have to type the variable name and then execute that code. So, I'm going to copy and paste it here. And then we run this code. And as you can see, by running this cell, we obtain the content inside the variable message_1.

We can create as many variables as we want. Just make sure to assign different names to new variables. So let's create a new message that says and it's fun and store it in a variable called message_2. So first I write message. So message and underscore 2. And then we set this equal to uh open double quotes and write and it's fun. This is my second variable. And I'm going to run this cell. So as we can see the string was assigned to this second variable. And if I copy and paste this variable here and run this code, we can see that the message it's there.

By the way, if you're using single quotes instead of double quotes as I'm using in this video, probably you had the following uh issue. So here I'm going to copy this one and paste it here so you can see what I'm talking about. So let's say you're let's say you're using single quotes instead of double quotes. So you get this. This is a problem that you will have when using single quotes because in the English language we use these apostrophes often. So an simple way to deal with this is using double quotes. So as you can see here if I use double quotes everything is okay. Everything remains as a a string but with single quotes uh it doesn't happen. So only the I gets this string but the rest it doesn't get a string value or the string data type. So just make sure you use double quotes every time you have these apostrophes, and that's it. Okay.

Now let's put these two messages together. So message one with message two, I want to put them together. So this is called a string concatenation. If we want to put message one and message two together, we can use the plus operator. And we can just do this. So I'm going to copy message one or the variable message one. And now I'm going to copy the variable message_2. And I use the plus in the middle to concatenate this first message with this second message. So I run and let's see what happens. So here we can see that the two messages were uh concatenated, but here there isn't a space between these two messages. So this is the first message and this is the second and there isn't uh any blank space in the middle. So what we can do here is to just uh add a blank space. So I'm going to copy this one and paste it here and show you how to do it. So here I add a new plus operator and in the middle we open a string. So with single quotes or double quotes, in this case I'm going to use single quotes here and to create this blank space I'm going to press space and here we have our blank space here and then we run this code and now let's see and here as we can see there is a space so between python and the and we have this blank space and if we want we can assign this new message to a new variable. So I'm going to assign this to a variable called message and I write message here and I include here below the code and here I can print this. So as you can see if I run this we can see that the message is there. Okay.

Now let me show you an alternative way to join two strings. So this is called the f string and it works like this. You write f and you open a string. So we write a single quotes here. So one and two and here as you can see the whole uh the whole thing is red. So it's like everything is a string. In here inside we can write the message. So let's see uh let's say we write I don't know a simple hello world. So hello world and we run this and as you can see here this is a string. It just has this f uh in front of that string and here uh one of the advantages that this f string has is that it can have variables inside the string. So here for example we can write a variable opening this curly braces. So this curly braces can have variables inside it. So here I can write message uh_1 and we can print it. So if we print we have this string I'm learning python. And now if we want to concatenate this first message with our second message, we just have to include curly braces again. I put it here. And now I write message two. And between message one and message two, I just have to press a space. And we have this. So I'm learning Python and it's fun. So here we just press space and the space also appears here. So for example if we add some random text let's say ABC we get this ABC between Python in between and so this is how F string works. You just have to write the f then open single quotes and inside you can write any message and to include any variable just you have to open this curly braces write the variable name and that's how you join strings. Okay.

Now it's time to see a data type that is used often in data analysis. I'm talking about lists in Python. Lists are used to store multiple items in a single variable. Lists are ordered and mutable containers. In Python, we call mutable to objects that can change their values. That is, elements within a list can change their values. To create a list, we have to introduce the element inside square brackets separated by commas. So, let's create our first list. First we have to set the name of the list. In this case, I'm going to name it countries. And now to create the list we have to open square brackets as I said before. So here we open square brackets. And here we have to write the elements. So I'm going to include in this countries list just strings and they're going to be uh names of countries. So the first one I'm going to write uh United States. So this is the first element in my list and to write the second we have to use the comma. So here comma and now the second. So let's write India, uh two more. So now China and finally Brazil. So these are the four countries as you can see here. uh this is a list. So we have the square brackets that represent the list and we have four strings and this is how you define or how you create a list. So now I'm going to run this one and to see the content I'm going to paste the name of this list and now I run here I include only strings but keep in mind that lists can have elements of different types. So for example, one string and the other an integer and then a float and so on. And also lists can have duplicated elements. So for example, I can have here uh United States uh written twice. So here for example, I can write United States twice. And that's okay because lists can have duplicated elements. But I don't want it that way. So I'm going to delete it and leave it as it is. Okay.

Now, if we want to get an element inside this list, we have to use something called indexing. By indexing, we can obtain an element by its position. So, each item in a list has an index, which is the

position in the list. Python uses zero-based indexing; that is, the first element. So, United States has an index zero. The second, so India has an index one, and so on. To access an element by its index, we need to use the square brackets again. So, let's see some examples.

Let's start by getting the first element. So, United States. So, what we have to do is to write the name of the list, in this case, countries, and then open square brackets. And inside square brackets, we have to write the position of this element. So it starts with zero. So we write zero to get the first element, and then we run this code, and as you can see, we got the first element. So if we write here country square brackets one, we get India, and if we write countries square brackets two, we get China. And if we do this with the number three, we get Brazil.

To verify this, I'm going to print each of them. So let's see what happens. So here, print and finally print this one. And now I'm going to run, and we should get uh each element of the list from United States to Brazil. So let's try out. So here we have each of them: United States, the first one, then India, then China, and then Brazil. So it's correct.

So this is the most common way to use indexing, but there is also negative indexing. This helps us get elements starting at the last position of the list. So instead of using indexes from zero and above, we use indexes from minus one and below. So let's get the last element of the list, but now using a negative index. So we want to get uh the last element, which is Brazil, and we did it before with uh countries square brackets three, but now we're going to do it with negative indexing. So here I'm going to write countries. I copy and paste it here. And now I open square brackets. And instead of writing three, we're going to write minus one. And this minus one represents the first element starting from the last position. So Brazil will be minus one, China is minus two, India minus three, United States minus four. And that's how it works. So I'm going to run this one: Country square brackets minus one. And we should get Brazil. And we got it. So let's do this one more time. And in this case, I want to get United States, which is minus one, two, three, and four. So it's countries minus four. So we run this, and we got United States, but now using a negative index. Okay.

Now let's see something called slicing. Slicing means accessing parts of a list. A slice is a subset of list elements. A slice notation takes the form of list. So the list name and then a square brackets and the start. Then this colon and stop. This start represents the index of the first element, and stop represents the element to stop at without including it in that slice. So let's see some examples. So I'm going to use this countries list again. I just I'm going to copy this one and I'm going to paste it here. So this is the name of my list. And now I open square brackets, and we're going to get uh let's say we're going to start at position number zero and then column and let's get from zero to the position number two. So we have to write three because it stops at three without including this element in the position number three. So let's run this one. And as you can see here, we have index zero, index one, and index two. So it didn't include index number three. And now let's say we want just the first element. So we write from zero to one. So it's only zero and one. No, because it doesn't include one and it stops at one. So here I run, and we got only United States.

So now let's try something different. Let's say we want to get uh elements from index one to the last one. So let's say let me see here we want to get from India to Brazil. So it's one, two, and three. So we have to write four because it stops at four, and we got three. So let's write here one, two, four, and we should get yeah, India, China, and Brazil. So this is one way to do it. But another way to do it is just delete this and leave it as it is and then run the code. And as we can see, we got the same result. So every time you want to get from one position to the last one, you can omit the stop element and just leave it without that element. So just as we did here, and the same goes for the start. So let's say we want to get from the first position. So index zero to two. So we don't include the start element and we write only colon and two. So we run this, and we get United States and then we get India because this is the first and this is the second. So every time we want to get from the first element or until the last element, we can omit the start and stop elements as we did in these two examples. Okay.

Now let's see how we can add elements to a list. There are different methods that help us add a new element to a list. So let's have a look. The first one is called append, and we're going to use the counters list as an example. So I'm going to write countries just so you can remember. And here it's countries, and as you can see, it has four elements. And let's say we want to add a new country to this countries list. So what we can do is just write here or paste here countries and now add append or that append, and here as you can see, is this is a method. So inside parenthesis, we can write the new country we want to add to this list. So let's say we want to add the country Canada. So we write Canada, and now we run this code. As you can see, nothing is printed. But if we print the counters list again, we see here a new element. So as you can see here, the append method adds a new element at the end of the list. So this is by default at the end. But what happens if you want to add an element in a different position? So here you can use another method, which is called the insert method. So let me show you here. I'm going to copy countries and now I'm going to use the insert method. So I write insert then parentheses, and this one uh accepts two arguments. The first one is the index. So the position of the element you want to insert. So let's say we want this at the first position. And the second argument that it takes is the new element you want to add. So in this case, let's say we want to add the element Spain. So this is uh another country, and it's going to be uh in the first position. So index zero. So let's try; I run this one, and again nothing happens. Apparently nothing happens. And here if I uh run this countries list again, we can see that there is a new element, and this element is Spain, and it's located in the first position unlike Canada that was placed in the last position. This is one of the differences between the append method and the insert method. So with insert, we can specify the position we want to insert this new element, but with append, the element is added at the last position.

Another thing you can do is to join two lists using the plus operator. We use the plus operator to concatenate strings before, but you can also join two lists. So let me show you here. I'm going to create a new list just to show you how it works. So my new list is going to be called uh countries_2. So I'm going to include different countries. So in this case, it's going to be uh the UK, then Germany, and let's write Austria. So we have three countries in this new list. And now I'm going to run this one. And if we want to concatenate this uh first list countries with this second list countries too, we can use the plus operator. So here I write plus and then I run this one. And as you can see, I got the five elements from the first list and the three elements from the second list.

And another cool thing you can do in Python is putting these two lists inside another list, which is called a nested list. So let's try out. So here I'm going to create a new list, and it's going to be called nested list, and here I'm going to open square brackets to create a new list, and as elements, I'm going to write countries, which is my my first list, and then comma and then countries_2, and this is my second list. So as you can see here, this uh elements inside this list, the first is a list, and the second is a list. So we have lists inside another list, which is called a nested list. So I run this one and then I paste nested_list, and we run, and we get here the first list as the first element and the second list as the second element. You won't see this nested list so often, but you will encounter this a couple of times. So it's good for you to know.

So now we're going to see the opposite of adding an element to a list, which is removing an element. So here I just pasted the countries list we had before. And what we're going to do is to remove some of the elements of this list. So there are different methods that help us remove an element from a list. One of them is the remove method. So to remove an element using this, we have to first write the name of the list and then use that that sign and then write remove and write parentheses, and inside here we have to write the element we want to get rid of. So first, it's United States. So I write United States, and let's run this one. And as you can see, apparently nothing happens. But if we paste countries here, we have uh all the elements, but United States is not there. So as you can see, the first matching value was removed, but you can also remove an element by its index. So this is accomplished with the pop method. So I'm going to copy all of this and now I'm going to paste it here. So instead of writing that remove, I'm going to write that pop, and here I'm not going to uh use the name of the element, but it's index. So I write the index in this case. Let's remove the last one. So it's going to be index minus one. And what pop is going to do is to remove the element with index minus one and then returns this element. So this element is Canada. I didn't run this code here. So you can ignore it. So I'm going to comment this one. And our reference is going to be this this list. And to verify, we just write countries and then run. And here as you can see, there isn't Canada anymore. And that's how you remove an element using the pop method. But there's still another way to remove an item using a specific index, and it's the del. So I'm going to show you here del. It's uh the function del function, and here we have to write the countries list and then again open square brackets and here write the index. I write here the index, and unlike the pop method, we're not going to get the name of the element we're getting rid of, but just deleting the element. So I run this one, and here we didn't get anything. And I'm going to print this. So countries, and the element at index zero was removed. So Spain because that's the first element. So we deleted or we remove the first element. So we only got India, China, and Brazil. And there you have it: three different ways to remove an element from a list. Okay.

Now let's see how to sort a list. We can easily sort a list using the sort method. Let's create a new list called numbers and then sort it from the smallest to the largest number. So here first I write numbers and then open square brackets. So I'm going to write uh some random numbers. So first uh four, then three, then 10, then seven, one, and then two. So this is my list. So I run this code, and now to sort it from the smallest to the largest number, we write numbers then sort then open parenthesis, and by default, this is going to be sorted from the smallest to the largest number. So I run numbers again, and here it starts with one and it ends with 10. And as you can see, it's from the smallest to the largest number. So that's the default behavior of the sort method. But we can control how this works. So we can add the reverse argument to the sort method to control the order. So if we want it to be descendant, we set reverse to true. So here again, I'm going to create again the numbers list. And then write numbers.sort, and inside parenthesis, I write the reverse argument, and I'm going to set it to true here, and then I'm going to print numbers. So here I got an error because here it I wrote number, and it's numbers. So here I'm going to add the s and here s2. So run again, and here we have uh from the and here we see that the list is sorted from the largest number to the smallest number. So as you can see, the default behavior of this sort method is reverse equal to false. So you can control it here by writing reverse equal to true as we did here. Okay.

Now let's see how we can update values in a list. To update a value in a list, we use indexing to locate the element we want to update, and then we set it to a new value using the equal sign. So let's say we want to update the first element of this numbers list. So now it's four, but we want it to be uh let's say 1,000. So we write here numbers, and we use indexing. So we write numbers; the first element has index zero. So we write numbers square brackets then zero. Then we set it equal to the new value we want to include. So in this case, I'm going to write 1,000. And now I'm going to print the numbers list to see the results. So run this one. And as you can see here, the numbers list we got is from the last change we made. So the one that starts with 10. So it's not this one, but this one because it's the last one we ran. So instead of 10, we replace this one with 1,000 because this is the first element with index zero. So we did numbers square bracket zero, and we update that first element with 1,000. Okay.

Finally, we can make copies of the lists we created. So there are different options to create a copy of a list. One of them is the slicing technique. So as you might remember, to do a slicing, we have first to write the name of the list, which this case is countries, and then we open square brackets. Then we're supposed to write the start and stop. So in this case, we're not going to write start and stop but only colon. So if we don't write start and we don't write stop, it means we want the whole list. So let's try this out. I'm going to run this one. And as you can see here, we got the whole list. So the contours list doesn't have the original values because of the changes we made when we added and removed elements. So I'm going to paste the original countries list with the four original values that are United States, India, China, and Brazil. And here, let's see the changes. And now we test it out. And as you can see, we got the whole list. So from the first element, United States, to the last element, Brazil, because we're slicing the whole list. So if we write here new_list and we set this equal to countries with this slicing, what it's going to happen is this new list is going to have the same values as the country list. So I write here new list, and as you can see here, it has the same values. So we created a copy of the counters list. So this is one way how you can create a copy. And the second way is more straightforward or is more explicit. So is using the copy method. So we write again countries, the name of the list, and then we use the copy method. So we write copy and then parenthesis. So with this, we create a copy of this list. So let's run this code. And as you can see here, it returns the list. But if we assign this to a new list, we're going to create a copy. So here I'm going to write new_list_2. So here we assign this copy to this new list. So I'm going to copy this new list and paste it here. And as you can see here, we have the values of this list, which are the same as the original countries list that is here. And that's it. That's how you make a copy of a list.

So now let's see how dictionaries work in Python. In Python, a dictionary is an unordered collection of items used to store data values. And a dictionary contains a key and a value. So this is what you will often see in a dictionary. So here, for example, the name of my dictionary is my_dict. And to create this dictionary, we have to use these curly braces. So we open curly braces, and inside we write our first item, and the first item consists of a key here on the left and then a value here, and it's separated with the colon. So here we have the key then colon and then the value, and then we have here the second item. So the second key and the second value. So now let's create a dictionary that has some basic information about me. So I'm going to name this dictionary my_data. And now to create this dictionary, I'm going to open curly braces, and the first key is going to be name. So I write name, and it has uh a value that is my name. So I'm going to write Frank. So I open single quotes and then write Frank. And then I'm going to add a new item. So I write comma, and then the second key is going to be age, and the second value is going to be my age. So in this case, I'm going to write my age, which is 26. So as you can see here, the first is a string; the first value is a string, and the second is an integer. So we can mix different data types. So now I press ctrl enter to run this code, and we created this dictionary. So now I write my_data, and here you have the dictionary we created. So here we can get the keys of this dictionary. We only have to write my_data.keys(). So this is the keys method. So we run this, and we get this dict_keys, and the values are name and age, which are the keys of this dictionary we created. So name, the first key, and age, the second key. Now we can get also the values. So my name and my age. So we just have to use the values method. So I'm going to paste this one here, and instead of writing that keys, I'm going to write that values. And now run this, and we get my name and then my age. So next I'm going to get the items. So as I said before, an item is this. So this is the first item, and this is the second item. So we can say that the item is a pair of key and value. So we can get this by using the items method. So instead of writing that values, I'm going to write here items and then run this one. So here we got the first item. So the first pair key and value, which is my name and well the key name and then my name Frank and then the second item. So the key name age and the age which is 26. Now we can add a new pair of key value in this dictionary we created. So let's say we want to add my height. So I write my_data, and let's say we want to add the key name height. So I write height. So we use square brackets here. And then we set this to the value. So let's say uh it's 1.7. So I write my_data and uh then square brackets then height inside it and then equal to 1.7. So if I run this and then I run the dictionary, we can see that there is a new item, and it's the height. So height uh colon and then 1.7. This is how you add a new item to the dictionary. And now we can update this height. So let's say uh I'm not 1.7, but

I'm 1.8 m. So what we can do is to use the update method to update this uh value. So I write my data and here I can use the update method. So I write update and then inside parenthesis we have to open curly braces to update this new item. So I'm going to write the key which is height and then I'm going to set the new height which is 1.8. So let's try this out. I run this and then let's see the values. So let's see if it was updated. So I run this and we got the height 1.8. So it's perfect.

So now let's see how we can make a copy of a dictionary the same way we did before for the lists. So to make a copy, we just have to write the name of the dictionary. In this case, it's my_data and then just as we did for the list, we can use the copy method. So we write copy with parentheses and then we create a new copy. So here you can see the copy and now I can assign this to a new dictionary. So I'm going to write new_dict. And now I'm going to copy this one. I'm going to run and then I write new_dict and run this. And as you can see it has the value of the my_data dictionary.

And something I didn't tell you when I make a copy of the list is that if you change the data inside the my_data dictionary, so the old dictionary, the effect is not going to be seen in the new dictionary. So for example, if we write 1.9 and here I update this in the old dictionary. So here you can see height 1.9 and if we run this new_dict we can see that after running this height remains with the same value 1.8 and it doesn't change to 1.9. This doesn't happen if you make one of these copies. Most people do. So let me show you what I'm talking about. So most people just make a copy doing new_dict = my_data. So this is the old dictionary and this is my new dictionary. So what happens if I run this and then I going to show you the values of this new dictionary. So this is 1.9 and if I update this to let's say 1.95. So update here, update here, here is 1.95. And if I run this new_dict_2, we can see that the value was updated too. And this shouldn't happen. So if you want to create a new dictionary that works independently from the old dictionary, you should use the copy method. And this is the same if you're making a copy of a list.

Finally, let's see how to remove elements from a dictionary. So, just like we did with the lists, we can remove an item in a dictionary. So, there are different options. First, we have the pop method. So, we write my_data. I'm using the old dictionary we've been using so far. So, my_data and I'm going to write that pop. So this is the pop method. So here I can write the key. So in this case I'm going to write the key. Let me see here uh my_data the key name. So I write uh pop then parenthesis then name. So as you might remember the pop method returns this value of the key. uh before we did with the list and it returned the list element. In this case, it returns the uh value of the key. So this is the key name and it returns the value. So if we print this my_data dictionary, we see that this pair um key value isn't here. So we successfully remove this item.

Another way to remove an element or an item from a dictionary is using the del function. So we write del and then we write the name of the dictionary. So my_data and then we have to specify again the name of the key. So we open square brackets and uh open quotes. And here let's say we want to delete the or remove the H key with its value. So we write H and we run this. And then if we print this dictionary again, we get the dictionary and we see that the H key was removed and also its value.

And finally, you can remove all the items in a dictionary with a clear method. So we write my_data and use that clear with parenthesis. And now if we print this dictionary, you can see that this is an empty dictionary because we remove all the elements from this dictionary.

Now let's see one of the most common statements used in Python. This is the if statement. The if statement is a conditional statement used to decide whether a certain statement or block of statements will be executed or not. Here you can see the syntax of this if statement and as you can see it starts with the if keyword followed by the condition. So if the condition is true, this code here is going to be executed. And if the condition is not true, so it's false, the code here in the elif it's going to be tested. So here in this elif block, this new condition will be tested. And if this is true, this code below will be executed. But if it's not true, then the else block will be tested. And here this is the last block and automatically this code will be executed. So here one little detail that most beginners forget to write is the colon. So it's sometimes easy to forget it's there but you have to include it. And one other things some people miss is this indentation. So here there is an indentation you have to include after the colon. So every time you write here colon you press enter and you automatically in most test editors you're going to get this indentation. But if for some reason you don't get that indentation and you get something like this, you can indent this uh line by using the tab key in your keyboard. So just press tab and it's going to indent this line. So make sure you write the colon and you include an indentation for each code that will be executed. So here, here, and here.

So now let's have a look at some examples to see much better how the if statement works. So first I'm going to create a new variable. And as you might remember, to create an variable, you have to write a name of this variable. In this case, I'm going to name it H. And then you have to set it a value. So in this case, this is going to be a number. So I'm going to set this h to the number 18. And now I'm going to write this if condition or if statement. So I write if h is greater than or equal to 18 then colon and then this code is going to be executed. So if this is true, I'm going to write print and then a message. So if this person or if the age is equal or greater than 18, I'm going to write the message you're an adult. And as you can see here, I'm using single quotes and I wrote the apostrophe. So I'm going to use double quotes. and everything is fine now. So here print uh then the message you're an adult. So if this isn't true I write else and then colon and print uh here a new message which is you are a kid. So let's see this again. So if the age is equal or greater than 18 then we print you're an adult but if it's uh less than 18 we print you're a kid. So here we run this code and we should get this because 18 is equal to 18. So let's run and as you can see we got the message you are an adult.

So now we can play with this. We can change the age value. So here I'm going to set it to 15. So I run and as you can see here 15 is less than 18. So this is false and this code is executed. So this block here it's going to be executed. So we got you are a kid. So we can try this one more time. So in this case I'm going to write another age. So 30 and again 30 is greater than 18. So this is executed. So you're an adult.

So now let's add a new block. And I'm going to use the elif. So I write elif and then h and then greater than let's say 13 and then colon press enter and we got this indentation and then we print another message. So if the age is equal to or greater than 13 we write the message you are a teenager. So teenager. So if it's between 13 and 17 or well less than 18, it's going to be you're a teenager. But if it's less than 13, it's going to be you're a kid. So let's try this out. So I write first 10 and then we get you're a kid because it's less than 13. Then we'll change this to 14. And then we get you're a teenager because 14 is greater than 13. And finally, we write 20 and we get you're an adult because 20 is greater than 18. And that's it. That's how the if statement works.

Now it's time to see one of the most common loops in Python. This is the for loop. Python for loops are used to loop through an iterable object and performs the same action for each entry. One example of an iterable object is a list. So we can loop through each element of a list and perform the same action on each element of that list. Here you can see the syntax of the for loop. And as you can see here is the for keyword and then we have to use a variable. Then we have to write the in keyword and then the iterable. In this case, as I told you before, the most common is the list. So you have for variable in list. I'm going to write here list so you can see much better. And then we have to write the colon. And then after a colon, it goes an indentation. So here we have the indentation and the code that will be executed for each iteration here that we make with a for loop. So to see this much better, I'm going to use the countries list we created before. So this is the countries list and I'm going to loop through this list. So I write for and then we have to set a variable that is going to be just just temporarily. So this variable is going to be called country. So this variable doesn't exist. We just create it temporarily. So for country in and then we have to write the name of the iterable which is in this case a list. So countries. So for country in countries and then colon and then enter and we get this indentation. Then we say print country. So for this variable in this iterable which is a list print each element. This is what we're saying in this um for loop. So we run this and as you can see each element of the list country is printed. So we're looping through the count's list and printing each element. So the first is United States, then India, then China and Brazil. And this is how the for loop works.

Now let me show you a new function that you can implement along with a for loop and it's called enumerate. So I'm going to write here enumerate and here I'm going to put this countries list inside this new function. So what this enumerate function does is to enumerate each element of the countries list as we loop through the list. So I'm going to add here a new uh variable and it's going to be i then comma and then country. So this enumerate will return two elements. The first one is going to be the number of the loop and the second one is going to be the element itself. So here I have to print uh apart from the country the i variable that I just created here or it's just temporarily here. So I write print i and then print country. So here we're going to print here the number of the iteration and the element. So I run control enter and here we got it. So first is United States in the iteration uh the first iteration which which is zero. Then we got India in the second iteration which has uh one and so on. So as you can see here uh the i starts with the zero. So this is how enumerate works. It starts with the number zero and it returns the number of the loop and the element.

And finally, let's loop through elements in a dictionary. So, let's use the dictionary we created before that was my_data. Well, this is empty. So, I'm going to use the original dictionary. So, here I have the original dictionary and it's here. So, I'm just going to print it. So, this is the dictionary and now we're going to loop through this dictionary. So let me show you here. First we have to write for and then we write key and value because one item as you might remember is made of a key and the value. So key and value. So we say for key, value in and then the name of the dictionary. So we write my_data. And now to get the items of this dictionary we have to use the items method. So we write .items and then parenthesis. Then we uh write colon and we press enter. So here we can print the key and we can also print the value. So key and value and then we run this code and as you can see here we get the key the first key and we get the value we get name and we get Frank and then the second key H and the H26. So this is how you loop through elements or items inside a dictionary. Okay.

Now let's see how functions work in Python. A function is a block of code which only runs when it is called. You can pass data known as parameters into a function. So here it's the syntax of a function. And as you can see here we have first to set the keyword def to create this function. And then we have to write the name of this function. And inside parenthesis we define the parameters of the function that we're creating. Then we write colon and below you have to write the code and every function should return something. So we have to use the return keyword and then return something like a variable for example. So now let's create a basic function. So first we write def and then we write the name of the function. So this function is going to do something really simple. It's going to sum the values we pass into it. So it's going to be named sum_values and as parameters we set a, b then colon and press enter. Then what this function is going to do is to add the a + b values and we're going to set this equal to x. So we write x = a + b. And as I told you before, you should return something after we finish the function. So we write return and here we're going to return the x variable. So we write x and that's it. That's how you create a function. I ran this uh code and as you can see apparently nothing happens but this function was created. So to use this function we have to call it. So to call this function we have to write the name of the function and then we pass some parameters in. In this case it's called arguments when you call the function. So I'm going to write argument 1 and argument 3. So once you call this function, it's going to go to the function here and it's going to set this 1 equal to a and this 3 equal to b. So you have 1 + 3 and this is 4. So x is going to be equal to 4 and then this function is going to return the value of x which is 4. So this is supposed to return the value of 4. So we run this and we got the value of 4. So this function is working properly.

Okay. Now let's see some built-in functions that Python has. Python has lots of built-in functions that can help us perform a specific task. Let's have a look at some of them. So let's start with the len function. We only have to write the word len and then we open parenthesis. And as you can see here, Jupyter notebook gives the green color to functions. And now let's calculate the length of the countries list. So I have here the countries list. And now I'm going to copy this one, paste it inside parenthesis. And what the len function is going to do is to calculate the length of any iterable object. In this case, the countries list is an iterable object. And now I'm going to run to calculate the length of this object. So I run this one. And as you can see here, the length is 4. And this is how the length function works.

Now let's see a different function. And in this case, I'm going to create a new list that contains only numbers. So I'm going to write random numbers here. 10, 63, 81, then uh 1, then 99. So this is my new list and I created this list with only numbers to try the max and min function. So the max function is this one. We write max and then parenthesis. And this one returns the item with the highest value in an iterable. So my iterable is this list and we're going to get the highest value of the elements inside this list. So we run this one and as you can see here the maximum value is 99 and we can do also the min function and it's going to have the opposite effect. In this case we're going to get the minimum value of this list. So we run and we get 1.

Okay. Another common function used in Python is the type function. And this function give us the type of the object. We only have to write type. And what this function does is to return the type of an object. So in this case, let's copy and paste the countries object. And if we run this, we can see that this object is a list. And that's correct because here we created a list with square brackets. So that's what the type function does.

And finally, the last function we're going to see is the range function. This one returns a sequence of numbers that start with a number and ends with another number. So let's see how it works here. So this one has three arguments. First the start number this one I'm going to write 1, then the number where the sequence stops in this case I'm going to write let's say 10 and then the last argument is the increment so how the this sequence is going to grow by how much so in this case I'm going to say that this sequence is going to grow by 2 so I write 2. Now I run and as you can See nothing happens. We only get the same text here. But if we make a loop here. So I write for i in range. Now print this i. So this is a for loop. We saw this before. And here we run. And as you can see here, we're iterating over this range. And we're getting the elements inside this range. So the first element is 1. The second is incremented by 2. So 1 + 2 is 3. Then 3 + 2 5. Then 7 and then 9. And then we should get 11. But the last element here it's 10. So this sequence stops at 10. So we only get until number 9. And that's how the range function works in Python. And that's it. Now you know the most common built-in functions in Python.

Okay. In this video, we're going to see what are modules in Python. In Python, modules are files that contain Python code. A module can have classes, functions, and variables, and even runnable code. And to get access to a module, we have to use the import keyword, this one. And to see a module in action, we're going to see the OS module. And this one comes with Python. So you don't need to install it. So to get access to this OS module, we have to write import os. And that's it. We only write this. And now let's see some functionalities of this module. So the first one that we're going to see is the getcwd method. So to get access to that method, we write os.getcwd and then parenthesis. So this getcwd stands for current working directory. So we're going to get the directory where our Jupyter notebook file is located. So this file I'm working with right now. So let's run and let's see what happens. So as you can see here I have the path where the Jupyter notebook is located. So this is the complete path and you can see it by using the getcwd method. So now let's see another method and in this case we're going to list all the elements in the folder where this Jupyter notebook file is located. So here to do that we're going to use the method listdir. So this

means list directory, and I'm going to run it.

And as you can see here, I have this Jupyter notebook file that is named untitled. As you can see here, the name of my file is untitle. And these other elements, you can ignore them. They are not files; they are just some hidden elements in my folder, but they don't matter. So right now, the only file I have in this folder is this untitle file. So this is what the lister does. So it lists all the elements in the folder where this Jupyter notebook file is located.

And now let's see the last method which helps us create a new folder. So this method is called make dirs, and we have to write os.make dirs and then parentheses, and inside parentheses we have to write the name of the folder we want to create. So in this case, I'm going to name it new folder, simple as that. And now if we run it, we're going to see that nothing happens, but now if we use this list dir method to list all the elements in my folder, we can see that there is a new folder. So here, if we compare this result we got before with this new result, we can see that there is one new element, and this element is the new folder element, which is the folder we created using the make dirs method.

And that's it. Those are some basic things you can do with the OS module. In the following videos, we're going to install different libraries, packages, and modules so we can do even more things in Python.

In this first introduction to pandas, we're going to learn what pandas is. We're going to compare pandas with Excel, and then we're going to learn what pandas data frames are.

So first, pandas is probably the best tool to do real-world data analysis in Python. It allows us to clean data, wrangle data, make visualizations, and more. You can think of pandas as a supercharged Microsoft Excel because most of the tasks you can do in Excel, you can also do them in pandas, and vice versa. That said, there are many areas where pandas outperforms Excel. So before you learn pandas, let me show you why you should learn pandas, especially if you already know Excel.

So there are some benefits that pandas has over Excel, or Python has over Excel. So before dedicating time to learning pandas and also Python, let's see what these benefits are. So first, limitation by size. Excel can handle around 1 million rows, while Python can handle millions and millions of rows. Another benefit that Python and pandas have over Excel is complex data transformation. So in Excel, memory-intensive computations can crash a workbook, while in Python, when you work with pandas, you can handle complex computations without any major problem. Also, Python is good for automation, while Excel was not designed to automate tasks. You can create a macro or use VBA to simplify some tasks, but that's the limit. However, Python can go beyond that with its hundreds of free libraries available. And finally, Python has cross-platform capabilities. This means that Python code remains the same regardless of the operating system or language set on your computer.

Okay, before we start writing code, let me explain to you the core concepts of pandas. So we're going to start seeing the concepts of arrays. So arrays in Python are a data structure like lists. So you can find like one-dimensional arrays or two-dimensional arrays, also known as 2D arrays, and the two main data structures in pandas are series and data frames. So the first is a one-dimensional array, while the second, a data frame, is a two-dimensional array.

In pandas, we mainly work with data frames. But if you didn't understand so much the definition of a data frame with arrays, let me show you another definition. This one using Excel. So a pandas data frame is the equivalent of an Excel spreadsheet. Pandas data frames, just like Excel spreadsheets, have two dimensions or axes. So there are two axes, and one is the row and the other is the column. So the column is also known as a series. So what we've seen before, this one-dimensional array, series, is a column. This is another name to call the columns in a pandas data frame. On top of the data frame, you will see the name of the columns, and on the left side there is the index. By default, the index in pandas starts with zero. The intersection of a row with a column is called a data value or simply a data. We can store different types of data, such as integers, strings, booleans, and so on.

Right now, you see on the screen a data frame that shows the US states ranked by population. I'm going to show you the code to create a data frame like this later. But now, let's analyze this data frame. So the column names are also known as features. So our features here are states, population, and postal. While each row value is known as an observation, we can say that there are three features and four observations because there are three columns and four rows. Keep in mind that a single column should have the same type of data. In our example, the states and postal columns only contain strings, while the population column only contains integers. We might get errors when trying to insert different data types into a column. So avoid mixing different types of data.

So now let's see that terminology translation between Excel and pandas. So as I mentioned before, in Excel we work with worksheets, and in pandas we work with data frames. So the columns in Excel are also known as series in pandas, but we also mention or we also say often the word columns, and in pandas we work with indexes. So the indexes are those numbers that are on the left, and in pandas we also say rows. We have many rows, well observations too, but rows are fine. And finally, in pandas we work often with this NaN, that stands for not a number, and this is the equivalent of an empty cell that you might find in Excel.

So that's it for now. In the next video, we're going to learn how to create a pandas data frame from scratch.

Welcome back. In this video, we're going to learn different ways to create a pandas data frame. So as you might remember, a data frame looks like this. It has columns and rows, and the columns are series. So series are a 1D array, and arrays are how we create a data frame. So this is the first way to create a data frame with arrays. So these are arrays. We have 1D arrays, 2D arrays, and 1D arrays are basically columns, while 2D arrays are data frames. So usually to use arrays, we use a library named NumPy, and NumPy is what is under the hood of pandas. So to use NumPy, we have first to import NumPy. We're going to do that a bit later when we write code. But just to give you an idea what a NumPy array looks like, here I wrote a basic array. We have to use np.array to create this data frame that you see on the right, and well, this is one way to do it. You can also use lists as I'm showing you right now, and as you can see here in the second option when you create a data frame with lists, you don't need to use NumPy arrays because you're using some kind of list arrays. So we're going to write the code to create a data frame with arrays. But let's see the second option to create a data frame.

So the second option is dictionaries. You can create a data frame with dictionaries. And as you might remember, a dictionary has a key and a value. So we can use the key as a column name and the value as the data. So the value can be a list. So this data will be many elements inside a list. So a pair of key and value is known as an item in a dictionary, and in this case, it's going to be a series because it's one column what we have here. So this is the second way to create a data frame with dictionaries, and we're going to see that with code a little bit later. But now let's see the third way, which is with CSV files. So CSV files are files that can be opened in spreadsheets like Excel. And this is the easiest way to create a data frame because we only need to read the CSV file, and then the data frame is created, and that's it.

So now let's go to Jupyter notebook to create a data frame writing some code.

Okay, now we are on Jupyter notebook, and here we're going to write the code to create a data frame, and we're going to use the three ways I showed you before. So the first thing we're going to do is to import the libraries we're going to use to create a data frame. So that's the first line of code, and I already wrote that. So it's here. So first, we import pandas, and then we import NumPy. So import pandas as pd. pd is just a convention to name pandas, and np is a way to name NumPy. So to run this code, just press control enter. And now just wait, and we imported pandas and NumPy. So let's see the first way to create a data frame. So the first is with arrays, and to create an array we have to use NumPy. This is the first option. So we write np, which is the short name for NumPy, and then we use the array method. So we write array, open parenthesis, and inside we write the array we want to create. So I'm going to create, I'm going to write random numbers just for the sake of this example. So I open double square brackets, and then let's write, let's say one and four, and then let's say two and five, and the last one it's going to be three and six. So each pair of, let's call it lists actually, they are lists, each list represents a row. So this is the first row, or this is going to be the first row. This is going to be the second row in our data frame, and this is going to be the third row. So here we can name this array, and I'm going to name it as data. So data is equal to this NumPy array. So I'm going to execute this code, and now we have this data. So we created the array using NumPy. Now let's create a data frame with pandas. So to create a data frame with pandas, we have to write pandas. In this case, I can write pd because I named it like this here in my first line of code. So I write pd. And then to create a data frame, we use the data frame method. So we write .data frame and then we open parenthesis, and here we have to fill some arguments. So the first one, and that's what something that you always have to include in this data frame method is the data because you cannot create a data frame without data. So first we include the data. So first copy here our array and then you paste it here. That's the first argument. So you can create this data frame as it is. I'm going to show you here, just control and enter. So as you can see here, here is my data frame. But as you can see, it's full of numbers, and the column names also have numbers, and the row names also have numbers. So to make it more understandable, we can rename these column names and row names, or index. Actually, the name of the row names are index. So first we can name this index as rows, for example. We only need to add the index argument as I'm writing right now. And then you have to specify the names you want to set. So you have to open a list. So this second argument has a form of a list. So the first element is going to be the first index. So here zero. So in case you don't want it to be zero, you can set here another name. So in my case, I'm going to set it as row one. Then comma to set the second index as row two and the third as row three. So now we can add or we can modify also the column names. We have to use the columns argument, and here we write it columns, and then we open square brackets because it's a list here that we're going to edit, and in this case we have to modify only two elements. So the first going to be I want to name it col one and the second col two. So I'm going to write this one, and actually I'm going to name this data frame. So I'm going to set it to a variable, and this is going to be equal to DF. DF is the common way to name a data frame. So DF stands for data frame. So I'm going to run this code now. And as you can see here, it ran, and now to show the data frame I can write here DF, to DF. And now we have here the data frame. And as you can see here, the first row 1, 4, it's my first, my first list. And the second is the second row. And the first column, well, that's a series as we discussed before. So we have also the column names that we modified and the row names. So now let's quickly see how to create a data frame with arrays, but in this case without NumPy. So I'm going to copy this line of code and I'm going to paste it here. Option two. So here I'm going to paste this because this is the base of these arrays with list shape. And I'm going to just delete this. So I don't want NumPy array anymore. Just this double square brackets. So I run this now to create the data frame is the same way we did before. So just copy this and paste it here. So run this, and now I can run the I can write the DF and now execute this code. So as you can see, we have the same result. I'm just showing you the second way so you don't have to worry about learning right now NumPy.

Okay. Now let's create a data frame from a dictionary, and we're going to use lists in this example, and we're going to create a data frame using more meaningful data. So in this case, to create a dictionary, I'm going to use two lists. The first is going to be a list named states, and the second is going to be the population, and it will contain the population of each state. So the first list is states, and I'm going to write it here, and I open square brackets because this is a list, and now I write some states in the US. So the first is California, the second is going to be Texas. Let me write it here. The third's going to be Florida. And the last one, New York. So I quickly write it here. And now I'm going to create the population list. So in this case, I'm going to paste this data. So I pasted the population of each state. And now I'm going to create a dictionary from these two lists. So I'm going to write the name of the dictionary. So the name is going to be dict_states. Then this is a dictionary. So I should use curly braces, and now I'm going to set the name of the key. So the first key is states, then colon, and now the element or the value. So this is states, the first value, and the second key and value is population. I'm just going to set it to with a capital letter, and the second is the list population that we have here. So with this, we create our dictionary. So I'm going to run these two, and now we have lists and the dictionary. So now we can easily create a data frame using the data frame method that we used before for the first option when we create a data frame with an array. So to do it, just write pd, then data frame, and now we have to write inside parentheses the name of the dictionary. So I'm going to copy dict_states and I'm going to set this to a new variable. So I'm going to name this df_population. So a data frame about population. So now I run this, and here I got an error because I didn't write data frame correctly. Here is in capital letter. So I run again, and now everything is okay. So now to show the data frame, I just paste this one here, and now I run. So here we have this data frame, and as you can see here, my first key, states, is the name of my first column, and the data inside the states list is here. So here is my first column or my first series, and the same goes for population with its data. So here we created a data frame using a dictionary.

Okay. Finally, let's create a data frame from a CSV file. To create a data frame from a CSV file, we have to use the read_csv method. So first we write as usual pd, that stands for pandas, and then we use the method. So we write read_csv, open parenthesis, and then we have to write the name of this CSV file. Here I'm going to paste the name. So it's named students_performance.csv. And to download this data, you can check the notes of this video. And actually, we can have a look at this data before importing it into pandas. So it's here; I have it in Google Sheets. And as you can see here, we have the scores of some exams, math, reading, and writing. And we have some other data. So we can import all of this data. All of the 10,000 rows in pandas. So all of this is going to be here. So here we only have to define the name of this data frame. So here I'm going to name it df_exams. So now I run it, and to show now the first five rows of this data frame, we can use a method named head that we're going to see later. But just to give you an idea of this, we can write that head, and we get the first five rows. So as you can see here, we have the first five rows of this Excel, or actually CSV file. And you can see here, for example, the first row it says female, group B, and math score 72. So let's check if that data is the same here. So we have female, group B, and math score 72. So we have all this data here in this data frame. So if we want to see all of them, all of the rows here, we can forget about the head, and now we have all the rows. Well, here we cannot see part of the rows. I'm going to show you how to see that part later in this course. But now, as you can see, if we run this df_exams, we can see like the summary of this data set, or well, data frame in this case. By the way, in pandas, or when we work actually in Python, we usually call this type of CSV files, we call it datasets. And when we read a dataset using pandas, the result is a data frame, what we have here. So the CSV file is a dataset, and this when we read it with pandas is a data frame, and that's it. These are the three ways to create a pandas data frame.

Okay. Now it's time to see how to display a data frame in pandas. So here I have the CSV file we used before to create a data frame. And a little detail I forgot to mention before is that this CSV file should be located in the same directory where your Jupyter notebook script is located. So what I mean by the Jupyter notebook script is what we're seeing right now. I mean, what we're working on right now is a Jupyter notebook script. This file that we're working on right now. So what you have to do is to download this CSV file and place it in the same folder where your Python or your Jupyter notebook script is located, in the same folder. And this is how you're going to read this CSV file using the read_csv method. So just make sure both the CSV file and the Jupyter notebook script are in the same place, in the same folder.

Okay. Now I'm going to run these first two lines of codes that we've seen before. So the first, import pandas, and the second reads this CSV file. So I run this, and now we have this CSV file stored into this df_exams. This is my data frame. So now let's see how we can see this data frame. So the easiest way to see this data frame is just copying this name, this variable, and now pasting it here. Now I execute this, and now we have the data frame. Actually, this is a summary of the data frame because not all the rows are seen here. So here

If we scroll down a little bit, we can see here that there are 1,000 rows and eight eight columns. So here we can see all these rows, uh, and the columns, but as you can see here in the middle, we cannot see the the rows. So it's until four, and then it continues with 995.

So usually when we work with pandas, we don't need to see the data one by one. So row by row, that's not how we do it with pandas. But if for some reason you need to see all the data in pandas as you will do it here in Excel or in Google Sheets, I'm going to show you a way to do it a bit later. But first, I'm going to show you uh different ways how we usually uh display a data frame in pandas.

So the first way to do it is using the head method. So here to use the head method, we only have to write the name of the data frame, in this case df_exams, and then write head, then parenthesis, then we run this, and this is how we get the first five rows in a data frame. So as you can see here, we have uh from row zero to row four, and this is how we got these first five rows. So this is the head method.

In the same way, we can get the last five rows of this data frame by using the tails method. So here we only have to write again the name of the data frame, in this case, well the same dfc_exams, and then write tails, then parentheses, run this, and actually I think it's tail. Yeah, it's tail in singular. And now we get this uh we got the last five rows. So it's from 995 to 999. So these are the five rows, the last five rows.

And now in case you want to get more rows. So not only the first five or the last five rows, you can add an argument to the either the head or the tails method. So I'm going to use here the head method as an example. So here I copied this. And now I'm going to paste here. So let's say now we want to get the first 10 rows. So we write here inside parenthesis 10. And now we run this. And I scroll down here, and we can see that the first 10 rows are here. And we can do the same with tail. So here I write tail. And as we can see, the last 10 rows are displayed here. So you can specify the number of rows that you want to display. And that's how you do it.

So now I'm going to show you how to display all the rows of this data frame as you will do it in Excel or in Google Sheets. To do so, first we have to know how many columns this data frame has. So an easy way to get the number of columns is using the shape attribute. To get the shape attribute, first we write the name of the data frame. So in this case DF_exams, and then to get to this attribute, to get access to this attribute, we use the dot and then the name of the attribute, in this case shape. So now we run this, and we get 1,08. The first is the number of rows, and the second is the number of columns. So we have 1,000 rows.

So now to display all the rows, we have to use the set_option method. So we write pd.set_option, and inside parenthesis our first argument is going to be the following: display.max_rows. So here we have to specify one more argument, and this is going to be the number of rows we want it to to have. So here it's 1,000 because we have 1,000 rows. And we run this. And as you can see here, nothing happened because we only modify the default behavior of pandas. So if we want to get the the data frame, we just press enter and execute this data frame. I'm going to scroll down, and here as you can see here uh there are all the rows of this data frame. So I'm going to scroll all the way down here, and as you can see, it says 999. So all the rows are here displayed. And that's it for this video.

In the next video, I'm going to show you the different attributes, methods, and functions a data frame has in pandas.

Welcome back. In this video, we're going to see some basic attributes, methods, and functions that we can use in pandas. But first, let's learn what are each of them.

So first, attributes are values associated with an object, and they are referenced by name using that expression. So to get to an attribute, we have to use the dot dot sign. So for example, below you can see that we have a data frame named df, and to get to the columns, we have to use the dot columns. So columns it's an attribute, and that's how we get this attribute of this data frame.

So now we have a function. A function is a group of related statements that performs a specific task. So we've seen functions before in Python. We've seen some Python built-in functions like the max that gets the maximum value of a list or min that gets the minimum value or length that gets the length of a list. So those are some Python built-in functions that we can use in pandas 2.

And finally, methods are functions which are defined inside a class body. So we haven't talked anything about classes because it's not the main topic in this course. So just keep in mind that functions are inside a class. So when the creators of pandas built pandas, they use many classes, and those functions inside some classes are known as methods. So for example, below you can see the head method, and we've seen also the tail method and some other methods so far.

As a rule of thumb, when we use methods, we have to write the parenthesis, but when we want to get access to attributes, we only write dot and the name of the attribute. So the method it's with dot and parenthesis, and the attribute is with only dot and the name of the attribute.

So enough talk. Now let's write some code in Jupyter notebooks. So here we're going to use the same CSV file we used in the previous video, and we imported pandas as we did before. Then we read this CSV file with the read_csv method. And now we show the data frame simply by writing the name of the data frame. So we've seen this before. I'm just uh reminding you, and now we'll see some basic attributes, methods, and functions that we can use in pandas.

So first let's check some attributes of this data frame. So first I'm going to copy the name of the data frame. And now let's check. So the first attribute it's going to be the shape. So we've seen this before I believe, and to get to the attribute, we write the dot and then we write the name of the attribute. So it's shape. So df_exams.shape, and we get the name of the attribute. The first is the number of rows, and the second is the number of columns.

So let's go to the next attribute. The next attribute it's going to be the index attribute. And as you might expect, we have to write only the name of the data frame. Then the dot and now index. And this is how we get the index of this data frame. So as you can see, this has uh some form of a range. A range as you might know has three arguments, and actually two are uh necessary. The first is the start; in this case, starts in zero, and the second is the stop. So the last element and stops at 1,000. So this is uh true because here my data frame starts with zero and and finishes with 999. Well, it's 1,000 because stopped one before 1,000. And here it increases by one. So zero, one, two, and three, and so on. So the step is one. So this is my my index attribute.

So now let's continue, and now let's get access to the column attribute. So to do so, we write the name of the data frame and then we write the name of the attribute. So in this case, column, and it has to be written with s, so in plural. So we run this, and we get the name of the columns. So as you can see here, we have eight columns: the gender, race, ethnicity, and so on. And we can use this attribute even to modify the name of the columns. But we'll see that later.

And now let's see how we can obtain the data types of each column. To do so, we have to use the dtypes attribute. So we write, well the name of the data frame again, and then the dtypes, and this is going to give us the type of each column. So the gender is object, and uh actually from the gender to the test preparation course are objects, while the math score, reading score, and writing score are integers. So numbers; by default, anything that says object is some kind of string. So I'm going to print this so you can see much better. So here is the data frame again. And as we've seen before, from gender to test preparation has the type object, and as we can see here, all of them are strings. So we can say that objects are the same as strings here. And also anything that says a score here represents uh some kind of number. So that's why we get here integers. So in 64. So these are the most common attributes in a pandas data frame.

Now let's review some methods. So first let's see the first five columns. And as you might know, it's with the head method. So we only write the name of the attribute, sorry, the name of the data frame, and then we write the head method. So head and parenthesis. So we run this, and we obtain the first five rows.

So we can also obtain some summary info of the data frame by using the info method. So here we write the name of the data frame, info, parenthesis, and execute this. So here we have some information about this data frame, and here we have again the data type here and also how many rows are nonnull. So as you can see here, all the data that we have in this data frame are nonnull. So there isn't any empty data here in this data frame.

Okay. Now if we want to get some basic statistics of a data frame, we have to use the describe method. So we write the name of the data frame and write describe, parenthesis, execute this. So we run this code, and we have some basic statistics. So first the count. So this indicates how many rows each column has. So uh each of them has 1,000 rows. Then we have the mean. So it's basically uh they assume each of the the data here, the numeric data, and then divide it by 1,000 because there are 1,000 rows. Then the standard deviation, the minimum value. uh for example, in math score, the minimum value was zero, then 25% represents the percentiles, so this is Q1, 25%, Q2 is 50%, and Q3 is 75%. Then we have the maximum value on each score on each exam, and we see that the maximum score is 100 on each of them. So the describe method is a useful method whenever we want to get some basic statistics of the data frame, especially of the numerical data that we have in our data frame.

Okay. Now let's see some functions that we can use in pandas. We can use some built-in functions uh that Python has in pandas. For example, if we want to get the length of a data frame, we only have to write len and then inside parenthesis the name of the data frame. So, we run this, and we obtain that the length of this data frame is 1,000. Actually, the length of a data frame indicates only the number of rows. So, here I made a mistake; it's rows, and this is how we obtain the number of rows of a data frame.

So also we can use other built-in functions that Python has like the max function. So we write max, then the name of the data frame, we run, and in this case we didn't get anything anything meaningful because we get like a string. But if we write here the index and we write max, as you might remember, if we use this um this attribute, we're going to get the list of index. So if we use the max function, we're going to get the maximum or the highest index here. So run, and it is 999. So we can also get the lowest index of a data frame. We only have to copy this and instead of writing the max function, we write min. So in this case, we get the minimum index; and it is zero.

So now we can obtain the data type of the data frame. Well, the data frame has a data frame type, but we can verify that using the type function. So we write type, then, sorry, we write only the name of the data frame, and we run. So here as you can see, the type of this object is a data frame.

And finally, we can use a common function that is the round function. So we write only round, and this has two arguments. So first the object that we want to round, and in this case is our data frame, and the second argument is the number of decimal points that we want to have. So in this case, I want two decimal points. So we run this, and we're not going to get this number of decimal points in this particular example because the the numerical data we have here it's uh integers. So they are not floats. So this doesn't have any effect. But if you have a data frame with float numbers, you can round those numbers using the round function. And that's it. These are the most basic attributes, methods, and functions that we will see often in pandas.

All right. Now it's time to learn how to select a column from a data frame. So here I have the same CSV file we've been using in the previous videos, and well let's import pandas and let's read this CSV file. So I have this in the same data frame, and I'm just showing the first five rows. So now to select one of the columns of this data frame, we have two options. So let's see the first option. The first option is using the square brackets. This is the preferred way to select a column in pandas. And let's see how to select the gender column. So the first one here. So the first thing we have to do is to write the name of the data frame, in this case df_exams, and then open square brackets. So I open square brackets. And now we have to write the name of the column. So we open quotes. And now here I'm going to copy the name of this column, and I'm going to paste it here. So we have here the name of the data frame and then the name of the column we want to select. So now we press Ctrl+Enter to run this code. And as we can see, we have the first column of this data frame. So here we have this, and as you might expect, this is an array. So this is a 1D array, and as we discussed before in previous videos, 1D arrays are series. So we can verify if this is true. So we can do this with the type function. So I'm going to copy this, uh, column, this selection. And now what we're going to do is to use the type function. So we write type, then open parenthesis, and then inside parenthesis we write the object we want to evaluate. So in this case, it's this. And now we run this. And as you can see here, we get a series, and series, just like pandas data frames, have attributes and methods. So we can access those attributes and methods, and actually the attributes and methods between a series and a data frames are very similar. So for example, if we want to get the index attribute of this series, we only have to write the name of the series and then write and the name of the attribute. So index. So we run this, and we get this index in form of a range that starts with zero and ends with 1,000. So another method that share pandas and series is the head method. So we can also get the first five rows by writing dot head and parenthesis. So as you can see here, we get the first five rows of this series.

All right, that's it for the first syntax. This is my favorite syntax, and actually most people use it because it's the most practical. And now it's time to see the second syntax to select a column from a data frame. So this syntax involves writing the dot sign, which is here. So let's say we want to get the same gender column. So we write the name of the data frame followed by dot and the name of the column. So gender; in this case, we don't need to open quotes, and we don't need the square brackets. So we run this code, and we get the same series. So it's here, and probably now you might be thinking that this is more practical than the first syntax. But this syntax has some pitfalls. So now let me show you here. So what if you want to get uh one column that has two words? For example, what if you want to get, let me show you here, this column that has as name math score. So now let's try to get access to this column. I'm going to copy this column name and now scroll down and and now let's try. So I'm going to write first the name of the data frame and now the dot. So to get access to this or to select this column, we have to write the column name. So this is the column name. But as you can see, uh if I run this, we get an error because Python doesn't work like that. In Python, when we have two words, we usually add an underscore. So that's how Python understands this that this is a variable. But if it's like this, Python will not understand what you're trying to do. However, if you use the first syntax, so the square brackets, you won't have this problem. So, let me show you here. Now, I'm going to write this. I'm going to copy. And now I'm going to paste it here. And instead of having this only notation, I'm going to open the square brackets, open square brackets, and then add the quotes. So as you can see here, the column names has a string type, and now Python knows that this is a string, and now if you delete this dot sign and you execute this, you get this column without any error. So this is one of the advantages that the square brackets has over the dot dot sign. And that's it. In this video, we learn how to select one column from a data frame. And in the next one, we're going to learn how to select two or more columns from a data frame.

Okay. In this video, we're going to learn how to select two or more columns from a data frame. So, as usual, we're going to start by importing pandas and reading the CSV file we've been using so far. So we execute these two lines of code, and we get here the data frame. So what we're going to do in this video is to select two random columns from this data frame. So first let's pick some columns. So I'm going to select the gender column and also the math score column. So to select these two columns, we have to use the square brackets again. So here in this case, we have to use two square brackets to select two or more columns. So to do this, we have to write first the name of the data frame. So it's df_exams, and now we open square brackets. So we write one and two twice. So we have two pairs of square brackets, and inside we have to write the name of the columns we want to select. So we said that uh we wanted the gender column. So we write gender, and the second column that we chose was the math score. So I open these quotes, and now I write math score. So here I have these two columns, and by the way, the order that we write these columns is the same order that we're going to get that data frame. I mean, we can define the order of the columns inside this square bracket. So here we're saying that first is the gender column, and second it should be the math score column. So now let's run this, and as you can see here, we obtain first the gender column and second the math score column. So here we can see that it's a data frame, and there are 1,000 rows. So now we can verify that this is actually a data frame by using the type function. So let's check if this selection is a data frame. So now I'm going to copy this in here. Let's check out the data type of this selection. So here I paste

It and now we use the type function. We open, uh, this parenthesis, and now we execute this code. And as you can see here, we got that this is a data frame.

So here, one little detail I want to tell you is that when we use these two square brackets, or two pairs of square brackets, we're always going to get a data frame. But when we use only a single pair of square brackets, as we did in the previous video, we get a series. So one pair of square brackets is for a series, and two pairs of square brackets, it's for a data frame. Okay.

Now, to continue with the video, I'm going to select two or more columns using these two pairs of square brackets. So now let's choose the columns that we're going to get. So, in this case, I'm going to get the gender column and all the scores that we have here: the math score, reading score, and writing score.

So to do so, first I'm going to copy this first selection with it to have it as a reference. And now I'm going to paste it here. So here, so far we have two columns. So let's add the two remaining columns. So here, an easy way to to write these columns is just by copying this in the data frame, and here we can paste it. So instead of writing those names, we can just paste it here. Now I delete and I put it inside quotes. So here, inside quotes, and here we have it.

So here, as I said before, we can change the order of the columns. We just have to, for example, here I cut this, and let's say we want to have the writing score in the beginning. So here I paste writing score, and now what we're going to get is first the gender column, then the writing score column, and then the math score and reading score columns. So now let's run this code, and as you can see here, we have this data frame in the order that we defined here. Okay.

Now you might be thinking if there is a way to select two or more columns using the dot sign. So let's check if that's possible here. For example, let's say we want to get the gender and the math score column using the dot notation. So here I have it. And as you can see here, this doesn't look right because you have two strings separated by a comma, but you don't have a list; you don't have square brackets. This is probably going to fail. So let's check. I'm going to run this code. And as you can see here, we got an invalid syntax. So it's a syntax error. So, as you can see, we cannot select two or more columns with the dot sign. And this is one of the disadvantages that the dot sign has over the square brackets. This is why most people prefer to use the square brackets instead of the dot notation. And that's it for this video. In this video, we learn how to select two or more columns from a data frame.

Okay. In this video, we'll see different ways to add a new column to a data frame. So, here is the same students performance data frame. And as you can see, we have three columns with scores: math score, written score, and writing score. So, let's say we want to add a new score. So in this case, let's add a language score.

So to add a new column in spreadsheets like Google Sheet or Microsoft Excel, we'll simply insert a new column, and that's it. But in pandas, we have to use different methods or different ways to allow us to insert a new column. So let's see how to do it here. So first, let's add a new column with a scalar value. So a scalar value is simply a single value, and in this case, the column is going to have one single value. So all the rows are going to have the same value.

So to do so, we're going to have to select this imaginary column because this column doesn't exist so far. So what we're going to do is to select a column as we will do with any other column. So first we write the name of the data frame, in this case df_exams, and then we open square brackets and open quotes as we will do in any, uh, column. So here, instead of, for example, writing math score, I'm going to copy this; instead of selecting math score, we have to write the name of the column we want to create. So in this case, let's write language score. So this is a new column we want to create. And now we have to assign to this new column; we have to give it a new value or a new scalar value. In this case, I'm going to add a value of 70.

So now if we run this code, we're going to see that nothing happens. But if we now show the data frame, we're going to see that we have a new column. And this column is named language score. And the value that this column has is, uh, the same value. So it's 70 in all its rows. So we have 70 in row zero. And if we scroll down, we're going to see that it's 70 in all the rows. So even in row 999. But it's a bit weird that in an exam you will have, uh, all the students with the same score. So what you will usually do is to add some different values to this column.

So to do this, we have to use arrays, and to create arrays, we have to use NumPy. So here, in the second way to add a new column, we're going to use arrays. So in this case, we have first to see how many rows this data frame has. So in this case, it has 1,000 rows. And this is important because the number of rows has to match with the number of the array we're going to create. So let's create this array. And first, let's import NumPy. So we write import numpy as np. So we run this code. And now we imported NumPy. So now we have to create an array of 1,000 elements. And to do so, we're going to use a method called range. So it's written like this, arrange. And this gives us a range of numbers that start with the first argument that I'm going to write zero. And the last argument, that in this case it's going to be 1,000. So these are the limits of my range. So I execute this, and as you can see here, it starts with zero until 1,000. So to verify the length of this, uh, range, we have to use the length function. So as you can see here, the length is 1,000. So the array has 1,000 elements. So now I'm going to assign this to a new variable, and I'm going to name this variable language score. So language_score. So we execute this, and here I was planning to see the length of this array. So I quickly do it here as we did it before. So length, and now we have the length of the array.

So now we have to add a new column to a data frame with this array. And to do that, we have only to use the same way we did before. So first we, uh, write the name of the data frame, and then we make the selection. So this, uh, selection is going to be with the new column. Well, in this case, it's not new because we already created it. But let's imagine it's a new column. So it's language score. And now we have to set the array to this column. So we write language score here, and we set it to this new column. So now to see the results, we only show this data frame. And as we can see here, we have a new column, and this new column starts with zero and it ends with 999. So it doesn't have a single value anymore. But now it has a range of values.

And now there is a little detail we have to take care of. So scores are supposed to be between 0 to 100, and we have here from 0 to 999, and also here we have a sequence of numbers. So it's from zero and then one, and it increases by one, and usually in scores you will see that students have random scores. So we have to create here an array with random numbers, and to do that we have to use NumPy again, but here we have to use a different method; in this case, the method is named random.randint. So let's write it here: np.random.randint. So the first argument is the lowest value of these random numbers. And by the way, these are random integer numbers because scores are usually integer numbers. And in this case, I'm going to set this this to one. And the second score is the highest number or value in these random numbers. And I'm going to set it to 100. And the third argument is the size. In this case, we want an array of 1,000 elements. So, we set the size to 1,000. Now, we execute this. We run this. And I'm not going to see this, uh, array again. I'm just going to check that it has the length we want by using the length function. And here we have 1,000 elements. So, now let's create a new variable and store this in the variable. So here this is going to be int, then language_score, and this is going to be our new variable.

So here, one little detail you should know is that the first argument is inclusive and the last one is exclusive. So this means that if we, here, let's say we get the minimum value of this new, uh, array, we're going to get that the minimum value is one because this first argument is inclusive, which means that it can be included in this new array. However, if we print now the maximum value of this, uh, array, we're going to get that 100 is not there because it's exclusive, which means that this second argument shouldn't be included in this array. Okay.

Finally, let's insert this random integer numbers in the new column that we created. So, we have to just use the same way we did it before. So here I copy, and now I paste it. So here, instead of assigning this language_score, I'm going to use this int language_score. So here I'm going to run this code, and as you can see here, we have this, the same column, and we have now this data that is random, random integer numbers from the row zero to the row 999. So now this new data looks more like scores, like real scores, because these are random numbers, and these are between zero and 99. And that's it.

Now, one more little detail I want to share with you is how to create random float numbers because before we created random integer numbers, but if for some reason you want to create random float numbers, there is a way how to do it with NumPy. So we only write np, then random, then rand, and the arguments are the same. So, the minimum value, then the maximum value, then the size, which is 1,000. Then you run this, and well, it's similar to the one we got before, but now we have float numbers. And that's it. In this video, we learn different ways to add a new column to a data frame.

In this video, I'm going to show you two more ways to add a new column. In this case, we're going to use the assign and insert methods. So first I'm going to import pandas and read the same CSV file we've been working with in this course, and here it is. So it's the students performance.csv file. So it's here, and now let's add a new column. So here we're going to start with the assign method. And first let me explain to you when to use the assign method. So, it's a good idea to use the assign method when you want to add multiple columns and you only want to use a single line of code. And also, the assign method is preferred when you need to override the values of an existing column. This is kind of a best practice.

So, now I'm going to add a new column using the assign method. And first I'm going to create a series. And to create a series, first I'm going to create some random numbers. So here I have the code so we don't waste time doing this again. I'm just going to quickly explain what this does. And first we import NumPy, and then we create random numbers using the randint method. So we create 1,000 numbers from 1 to 100 because those are the scores, and the scores are between one and 100. So we create two variables because we're going to add two columns, and then we create a series. We create a series because we need an index. So apart from the random numbers, we need to add an index. So we create a series, and the index are from 0 to 1,000 because 1,000 is the number of rows that this data frame has. And that's it. So I'm going to run this code now that I explain it to you. And I'm going to show you how this series one looks like. And it's this one. So basically this is one column, and series 2 is the second column. So we're going to add this to the data frame. So to do that, first we're going to write the name of the data frame. So we write df_exams, and then we use the assign method. So we write dot, open parenthesis. And inside here we're going to write the name of the columns we want to create. Unlike the previous methods, we don't need to open quotes to create a new column. So it's only the name itself. So, for example, I want to create a column named score one. So we only write score one. And then that's it. This is my new column. And now I'm going to assign this to the series one. So I copy series one. And here I paste it. Then I create my second column. So it's going to be score two. And I'm going to assign this to series two. So what we're doing here is creating two columns, score one and score two, and we're assigning these new columns to the series one and series two. So we run this to see the results, and as we can see here, right now here we obtain the data frame. So we have the math score, reading score, writing score, and we have two additional columns. So the first one is score one, and the second one is score two. So we successfully assign these two new columns to the data frame. And as you can see here, this is only a copy. So if we run the df_exams data frame, you can see that the values weren't updated. So the data frame is the original data frame. So to update the values, we have to overwrite this data frame. So we write df_exams equal to this. So the assign method only creates a copy, and we have to overwrite the values of the data frame to update the data frame. So here I run this. Now I copy the df_exams. I paste it here. And now I show you the results. And as you can see here, the two new columns were added. Okay. Okay.

Now let's see the second method to add columns to a data frame. And this method is the insert method. The insert method allows us to add a new column at a specific position or index. And to use the insert method, we only have to write the name of the data frame. So df_exams.insert, open parenthesis. And now let's check the arguments this one has. So first we have to introduce the index of this new column. In this case, I'm going to add it in the index number one. Next we have to write the name of the column we want to create. In this case, I'm going to name this column as test. And then we have to write the value. So in this case, I'm going to use the series one, which is here, as the value of this test column. Now to insert this new column, we only have to run this, and as you can see here, apparently nothing happened, but actually the values were updated. In this case, the insert method updates the data frame, or the values of the data frame, and it doesn't create a copy. So unlike the assign method, we don't get a copy with the insert method. So the assign method returns a new object, or a copy, with all the original columns in addition to the new ones. And that's why here we had to overwrite the values because we only got a copy. So to update it, we have to override this data frame. But this isn't the case for the insert method because we ran this, and after that only the values were updated, and we didn't get any copy. So now let's verify this. We run this. So as you can see here, we got a new column named test, and is in index number one. So this is index number one. And by the way, index zero is where gender is located right now. And here in my new test column, we have the values inside the series one variable. So here are all the random numbers we created before. And that's it. That's how you add a new column with the assign and insert methods.

All right. Now it's time to see some operations we can perform on data frames. So here we have the same data frame, df_exams, and here we can apply some common operations to the numerical columns like math score, reading score, and writing score. So let's see how to do this in pandas. So first we're going to see how to make operations in columns. So our first task is to calculate the total sum of a column. So let's pick first the math score and let's calculate the sum of this column. So to do that, we have first to select a column, and as you might remember, to select a column, first we have to write the name of the data frame, in this case df_exams, then we open square brackets, and then write either single or double quotes, then we have to write the name of the column, and in this case it's this one, math score; this is the column we want to select. And now, instead of selecting, we're going to perform an operation. So in this case, I want to calculate the total sum of this column, and we have to use the sum method. So we write sum and parenthesis, and this is how you calculate the total sum of this column. So to verify this, we run this code, and here we got 66,000, and this is the total sum of this math column. Great.

Now we can make some other operations you will do in Excel. For example, we can calculate the number of rows using the count method. So here we can easily do that. I'm just going to copy this one. And now, instead of writing the sum method, we write count. So here, count. And now let's see. So we see 1,000 rows. And yeah, this is correct because this data frame has 1,000 rows. So now we can calculate the mean of this math score column. We have to copy this one. Now paste it. And instead of writing count, we have to write mean. And here we got the average value of this math score column. So to get the average, we have to sum all the rows in this math score column and then divide by the total number of rows, in this case 1,000. And this is how you get this mean value. Then we can get other, uh, other operations using the method. So here, for example, we can get the standard deviation by writing std. So we execute this, and the standard deviation of this math score column is 15, and we can get also the maximum and minimum value. Let's do it quickly here. So first the max and then the min value. You can actually do it with the Python built-in function, but we can also do it with the methods. So here I run, and as you can see here, the minimum value of the math score is zero, and the maximum is 100. Okay.

Now I'm going to show you a quick way to make the same calculations using the describe method. I think we saw the describe method in previous videos. But in case you don't remember it, I'm going to write here, uh, the name of; actually we only need the name of the data frame. We don't, we don't need the name of a specific column. We only need the name of the data frame. And now we can use the describe method. So we write .describe with parenthesis. And now we got like a summary table with some important statistical values. And here we have the count, the mean, the standard deviation, the minimum and maximum value. And as you can see here, we got all of these with one method. Okay, so far so good.

Now, instead of making operations in columns, we're going to learn how to make operations in rows. So now let's calculate, uh, let's say the sum of the math score, reading score, and

writing score. To do so, we have to make some selections, and in this case, we have to make some independent selections. So, to show you, I'm going to copy the name of these three columns. I copied it, and now I paste it here. And now we have a math score, reading score, and writing score. So now let me delete that sign. And now we have to make some independent selections. So first, we write the name of that data frame. So, DF exams. And now, to make the selection, we open square brackets in quotes. So now let me do this quickly in the others. Now here, so I open square brackets, and now let me do it here too. And now it's ready. So here we made some independent selections.

And now, to make uh to calculate the sum in a row, we have to use the plus sign. So here, the plus operator, we have to write it here and here. So basically, here we're making uh sum in each row. So, to verify this, we run this code, and as you can see here, we got the sum of the scores column. So here, let's verify fast the sum of the first row, and it's 72 + 72 + 74. So 72 with 72 is 144 and with 74 is 218. So here we have it, and it's correct. So now let's do something else.

So now, instead of just summing these three rows or actually these three columns, what we're going to do is to calculate the average to get like an average score. So here, let me copy this. And now here we're going to calculate the average by summing this and then dividing this by three. So this is how we calculate the score. And now let's assign this result to a new column. To do so, we only write equal in them. As you might remember from previous lessons, we have to add a new column by writing the name of this column. So we do that writing the name of the data frame and then making like a selection. So we open square brackets, then open quotes, and here we write the name of the column that we want to create. So this is the same as we did in previous lessons where we added a new column. So in this case, I'm going to name this new column as average, and I'm going to execute this. And now, to verify that this new column was created, I'm going to show this data frame here below, and here is our data frame. So now in the last column, you can see that there is an column named average, and it has the average value of this math score, written score, and writing score. And now here we can control the number of decimals. We can just use the round function and write the number of decimals we want to get. So in this case, I want only two decimals. So I run this, and as you can see here, our data frame looks much better because we only have two decimals. And that's it. In this video, we learn different ways to make operations in columns and rows on data frames. All right.

Now let's have a look at the value counts method. So so far we have seen how to count the number of rows in a data frame. So, for example, if we want to count the number of rows in the gender column, we either use the length function. So we write length, then the number or the name of the data frame, and we only have to write the name of the column. So, as you might remember, this give us the number of rows, and we can also use the count method. So here we write count, and we get the number of rows. But what if we want to count the gender elements by category? So female or male. What if we want to know how many female and how many male elements are in this gender column? So this is when the value counts comes in handy. So we can use this method to count each category of the column. So, to use this method, we only have to write the name of the data frame followed by the column that we want to count. So in this case, is the gender column, and then we have to use the value counts method, as you can see here. So now we execute this, and as you can see here, we have not only the total rows in this gender column, but now it's divided by category. So we have that there is 518 females and 482 males. So this is how the data is spread in the gender column.

So now we can do more with the value counts method. So we can get the percentage that each category represents in the whole column. So here I'm going to copy this. And now, to calculate the percentages, also known as relative frequency, we have to add an argument named normalize. So we write normalize equal to true. And then we execute this. And as we can see here, female represent 51% of the total observations in the gender column, while male only represents 48% of the total observations. So, as you can see here, the value count method is useful when you want to have a look at the data by category. Okay. Now let's see another example. And in this case, let's pick a different column. So here I'm going to choose this parental level of education column. I copy this. And now let's calculate uh let's count the elements by category. So here I'm going to write the name of the data frame DF exams. And now I open square brackets quotes. And here I paste this column. Now, to count the elements by category in this column, we use the value counts method. So we run this code. And here you can see how the data is divided in this column. So most people have some college level of education, while just a few people have a master degree. And now if we want to get the percentages that represent each category, we again use the normalize argument. So we write normalize equal to true. And now we're going to get the percentages. So we can see the percentages. And if we want to round this to two decimals, we use the round method. So we write round parenthesis. And now two decimals. And as you can see here, we round it to two decimals. And that's it. Now you know how to use the value counts method.

Okay. In this video, we're going to see how to sort a data frame using the sort_values method. First, let's import and read the CSV file that we've been working with in this tutorial. And now let's sort the data frame. So here we have the the data frame. And as you might remember, it's it has these three numerical columns. And now I'm going to sort it using one of these columns. So let's use the sort_values method. And first I'm going to write the name of the data frame, which is df_exams, and then write sort_values. Now I open parenthesis. And now I can use this help here. And as you can see, the only mandatory argument is by. So we can use this one by. And this one we have to specify the name of the column we want to sort by. So in this case, I want to sort by the math score. So I'm choosing this numerical column to start with. So I'm going to write math score. Actually, I'm going to copy this one and paste it here. So by math score, and sorting this data frame is as simple as that. Now we can run this code. And as you can see here, the data frame was sort ascending by default. So it starts with zero and it ends with 100 in the math score. So this is how the sort_values behave by default. And here one little detail, you don't need to specify the by word. We can omit it, and we run this. And as you can see here, it still works.

So here we can modify the default behavior of the sort_values method. We only have to add a new argument, and is that ascending argument. So let me show you here. I'm going to copy this one first and show you here. So in this case, we're going to sort descending by the same column. So we only write comma and then we specify the ascending argument. So we write ascending equal to, and here I want to show you something in this little help here. Here the ascending is set to true by default. This means that is ascending by default. But we can change this default behavior by setting ascending equal to false. And that's what we're going to do here. Ascending equal to false. So it means descending. And now I'm going to run this one. And as you can see here, is sort descending by the math score column. So here it starts with 100 and it ends with zero. But that's not all. We can do much more with the sort_values method. So first I'm going to show you here how to sort by two different columns. So here let's copy and paste this one. So in this case, we're going to sort descending by multiple columns. So instead of writing only math score, we're going to add here one more column, and it's going to be the reading score column. So here I copy this one. I'm going to copy and paste it here. But first we have to add the square brackets because as you might remember, when we write two or more columns, we need the square brackets. Now I write comma and I paste this written score. Now I add quotes. And that's it. That's everything you have to do to sort by multiple columns. Now I'm going to run this one. And as you can see here, it was sort descending first by the math score column and then by the written score column. So the priorities are set here in the list that we included here. So first is the math score column, first priority, and the second priority is the written score column. And that's what you can see here.

Now I'm going to show you a little detail here. Let me copy the df_exams. And if I print this one, you can see that the changes we made weren't updated. So this here, the math score column has the original values. This happens because the sort_values method, like many other pandas method, only creates a copy of the data frame. So here we obtain a copy. This one is a copy, but it doesn't update the values of the data frame unless we add a new argument, which is the in_place argument. So, I'm going to show you here, but first I'm going to delete this uh df_examp. And now I'm going to copy this one and show you how to update the values of this data frame. So here I'm going to copy those are the same values, but now I'm going to add a new argument, which is the in_place argument. So here write in_place equal to, and now I'm going to show you the default value. So here the default value of in_place is false. This means don't update the data frame but only create a copy. But if we set it to true, it means update this data frame. So here I'm going to set it to true to update the data frame. So here I write true. And now I run this, and apparently nothing happens. But if now we print the df_exams data frame, we're going to see that we have the data frame sorted. In case you don't want to add the in_place argument and you want to update the values of the data frame, you have another option that we used before, which is overwriting the values of this data frame. So, for example, you can only delete that in_place argument and write df_exams equal to this. So this is overwriting the values. But in this case, we're not going to do that. We're going to add the in_place argument as you can see here.

Finally, we're going to see how to sort, but now not with numerical data, but with text. So, as you can see here, we before sort this data frame by the math score column. And this one has this numerical data, but in this case, we're going to sort it by the race/ethnicity, which has this text. So we're going to sort this one. So first we're supposed to get group one and then group B, C, D, E, and so on. So let's do this here. I'm going to scroll down. And first we have to write the name of the data frame. Uh followed by the sort_values method. And I'll specify the name of the column. So here I'm going to copy race/ethnicity. here. Let me copy here, and it's done. Now I have the name of the column. I'm going to set to ascending true. And now the new argument we have to add to sort this is the key. So I add key then equal to, and in this case, we're going to use the lambda function. I'm not sure if you're familiar with the lambda function, but it works similar to an average function we've seen before in the Python crash course, but in this case, it's going to behave a little bit different. So, let me show you here. First, you have to use the lambda keyword. So, we write only lambda. And now, we should write the object that is supposed to return. In this case, I'm going to write col that stand for column. And then we have to write col and specify the operation we have to make over this variable. So in this case, I want to write col or call and then access the string attribute. So I write str and then use the lower method. So what we're saying here is get the string values of the column and then transform it to lowercase. So here we get the textual data in lowercase. And with these three arguments, we're saying sort the values inside the race/ethnicity column and sort it ascending and then sort the textual data of this column in lowercase. So here we have this A B CDE E in uppercase, but we're going to get it in lowercase and sort it by this text data. So now let's run this one and let's see the results. So as you can see here, we have this race/ethnicity column and its order ascending. So here we got the A and B and C and D and so on, and that's it. These are the different ways to sort a data frame using the sort_values method.

Okay. In this video, we're going to learn the set_index and sort_index methods. The first one is going to help us set a column as a new index, and the second one is going to help us sort the index. So let's get started. So as usual, we're going to import pandas and read the same CSV we've been working with. So here we have the student performance CSV. And now I'm going to create an index. So we can set this as a new column and then set it as an index. And then we're going to sort those indexes with the sort_index method. So first we're going to import numpy and random. So here I'm going to import this. And now I'm going to create a non-repetitive values for the index. So it has to be nonrepetitive values because index in a data frame doesn't allow duplicates. So the values have to be different from each other. So here, to create different values, we have to use the arrange method from the numpy library. So we write np then range then open parenthesis and specify the first and last element. So zero and 1,000. As you can see here, we get an array from 0 to 999. So now I'm going to set this to a new variable called new_index. So it's here. And now I'm going to set it here. And we have our new index variable. Now I'm going to shuffle this index so we can sort it later. So to shuffle this, we have to use the random library. To do so, we only write random here and use the shuffle method. So we write shuffle then open parenthesis. And here we have to include the object we want to shuffle. in this case is this new_index variable. So we write it here, and now we run this, and apparently nothing happened, but this new_index was shuffled. So now let's verify here. I'm going to write new_index. And as you can see here, it doesn't start with zero one and two anymore. So it's shuffled. Now I'm going to delete this one. And now I'm going to set this new_index variable to a new column of my original data frame. So here I'm going to copy the name of the data frame. And now I'm going to create a new column as we did in the previous videos. So we're going to use the simplest way. I'm going to write the name of the data frame. Now square brackets in here the name of the column we want to create. So in this case, it's going to be new index, and I'm going to set it equal to this new_index variable. So this is the variable that contains my shuffled numbers from zero to 1,000. So I'm going to run this one, and before I do that, I'm going to show the new data frame. So here I run, and as you can see here, the last column is new index, and it has these shuffle numbers. So great. Now I'm going to show you how the set_index method works. So here what we're going to do is to set this new_index column as the index of this data frame. So instead of being a column like right now, this new_index is going to be the index of this data frame. So it's not going to say here 0 1 2 and so on anymore, but the values of this column like 342 are going to be the index here. So let's do it here. So I scroll down, and now we only have to write the name of the data frame here df_exams and then we use the set_index method open parenthesis and write the name of the column we want to set as index. So in this case is new index. So this one here. And then we run this. So that's it. We run. And as you can see here, my new index is the new_index column. So it's here 342. And all the random numbers we created. So this is my new index. And as you can see here, the set_index method creates a copy. And if we want to update the values of these df_exams, we have to use the in_place argument as we've seen in previous videos. So I write in_place equal to true. And now I'm going to show this. So now df_exams data frame is updated. Now I'm going to show you how to sort the index of this data frame with the sort_index method. And this is going to work similar to the sort_values but in this case for index. And now I'm going to show you here. So first we write the name df_exams. This is the name of the data frame. Then sort to sort the index. And that's it. So we run this, and as you can see here, now it's sort ascending. So from 0 to 999. And if for some reason you want to sort descending, you only have to add the ascending argument as we did for the sort_values method. So we only write ascending equal to. And as you can see here, I'm going to show you here the default value is true. So we're going to set it to false. So I write false, and now is sort descending. So it starts with 999 and ends with zero. And we can even save the changes of this sort_index by adding the ascending sorry the in_place argument. So we write in_place equal to true. Now we run this, and let me show you how this looks like right now. And all the changes were saved, and the index is sort descending. And that's it. That's how the set_index and sort_index methods works in pandas.

Okay. In this video, we're going to learn how to rename columns and indexes with the rename method. So first we import pandas and this CSV file students performance. And now I'm going to show you how to rename a column. And the first column we're going to rename is the gender column. So in this case, I want the G in capital letter. And we can do that with the rename method. So we have to write DF_exams, which is my data frame. And now I write rename parenthesis. And now we have to specify the columns argument. So we write columns and then write equal. And this has the shape of a dictionary. So we have to open curly braces. So here I have the curly braces, and now the first element is a key because this is a dictionary, and the key is going to be the actual or current column name. So in this case is gender, and now we write gender, and the value is going to be the new column name. So in this case, it's going to be Gender but with G in capital letter. And that's it.

That's all you have to do to change the name of a column.

So now I'm going to run this one. And as you can see here, the name of the column is gender now, with G as a capital letter. But this is only a copy. So we can override the values of this data frame by writing DF_exam equal to this. I run this one. And here you can see that the value of the data frame was updated. So here gender is with G in capital letters.

So now I'm going to show you how to update two or more columns with the same method, but now I'm going to use the in-place argument. So for this example, I'm going to change the name of the math score, reading score, and writing score. So in this case, I think these are long names. So I'm going to use only the first letter of each word. So for example, for math score it's going to be MS, and then for reading score it's going to be RS. So those are the new column names. So here, to change the column names we only write the same method. So here I copy and paste it. And now I'm going to write one by one the name of the column we want to change. So first math score—here I made a mistake; I'm going to change it fast—so math score. Now the new name is going to be MS. And now the second element is going to be—I'm going to copy and paste it twice—so the second is reading and the third one is writing. So here reading and here writing. Now I'm going to delete those blank spaces, and it's ready. So now I'm going to add the comma here and write RS and here WS, and I think it's ready now. Now, to save the changes we're making here, I'm going to add the in-place argument equal to true. And now everything is ready. So I run this. Now let's see if the data frame was updated. So here I have the data frame, and as you can see the three last columns have the values MS, RS, and WS. And actually, it looks much, much better because the numbers fit much better in this narrow column. And that's it for the columns argument.

Now I'm going to show you how to change the index using the same rename method, but in this case using the index argument. So here I'm going to change the name of the index here. For example, instead of zero, I want it to be a, instead of one b, and then z, and so on. So here I'm going to write the same, uh, the same syntaxes: So df_exams, then rename, then parenthesis, and here I'm going to use the index argument. So it's here, and by the way, you can see all the arguments here by pressing shift-tab, and here it's the index and columns that we've been using so far. So here I'm going to add that dictionary shape, and then I'm going to set the key and the value. So the key is going to be, um, here zero, and the value, as I told you before, is going to be a. So zero, the current index, and a, the new index; so the new name of the index. Then one to B, and then two to C. So that's how it's going to be. And now to update the values, I'm going to set in-place equal to true, as I'm writing right now. And now I'm going to run this one. And let's check the result. So here DF_exams. And as you can see here, the first three rows have different indexes: So A, B, and C. I'm going to use the head method so you can see much better: Head three. And here are the three first rows with index A, B, and C. So we successfully renamed the first three indexes. And that's it. That's how you rename columns and indexes using the rename method in pandas.

Welcome to our first pandas project. In this project, we're going to learn how to make web scraping with pandas. Doing web scraping in Python usually involves learning libraries such as beautiful soup, selenium, or scrapey, that you can perform basic web scraping using pandas, and in this project I'm going to show you how to do it, and also we're going to see some methods we learned in this course so far. So let's start by importing pandas as pd. So I import pandas as pd, and here I'm going to read a CSV from a URL using pandas. And now let's perform some basic web scraping with pandas. Web scraping consists in extracting data from websites. So instead of doing it manually, we can automate it with some web scraping techniques. And in this video, we're going to extract CSV files from a URL using only pandas. So here is the target website we're going to scrape, and it's this one. So this website contains data about football matches of different leagues. So here you can see a lot of leagues. And now I'm going to choose the first one that says England football results. And here we're going to see some data about the Premier League and other leagues that England has. And if I want to download one of these files, I will have to click on any of those. And as you can see here, I downloaded the CSV file of the first listed here. So this one corresponds to the season 2122 and is from the Premier League. So instead of manually downloading each file, we can use a specific pandas method to read these files from the internet. And also, by using the for loop we can automate this and download all the files that you can see here. So instead of clicking one by one, we can download all the files listed here. So there are a lot of them, and we can download it just with pandas and a for loop in Python. So let's do it here. And now I'm going to show you how to extract data from a single CSV file from this website. So to do that, we have to use the read_csv method. So we write pd.read_csv, open parenthesis. And we've used this method before; that when we use it, we read some data that was in the folder where we were working; so in the folder where this Jupyter notebook file was located. But in this case, we're not going to read anything inside our computer, but we're going to read data that is in a website. So instead of writing the path of the file in your computer, in this case, we're going to write that link of the file. So here I'm going to show you this file has a link. So if we want to download, we have to make a request to that link to get that file. So I'm going to show you here. I'm going to right-click. And now I'm going to copy the link address. So I copy. And now I'm going to paste it here. And now I press enter. And let's see what's going to happen. So I press enter. And as you can see here, instead of going to the website, it downloaded the file. So this means that this link contains the data we want to extract. So we're going to use this link. So I'm going to copy again here. Copy. I'm going to make sure this is the address. So copy link address. And now go back here. Open quotes. Paste that link. And this is the link we want to extract because it contains here the that CSV. So this means that this is a CSV file. And as you might remember, we read here a CSV because we're using the read_csv method. So that's everything you have to do to read this CSV file that is stored in this website. So now I'm going to run this and let's see the results. So here, as you can see, all the data was read here with a read_csv, and it was successfully loaded here. So now I'm going to set here a new variable, and it's going to be called df_premiere_21. And as you can see here, this belongs to the 2122 season because here in the date it says 2021. And also this is Premier League because the teams belong to the Premier League. You may know if you're familiar with this competition, but if you're not, it doesn't matter. So now let's continue. So here I'm going to set this data frame to this variable. So I press Ctrl-enter. And now I'm going to show here this data frame that we saw already. And now I'm going to rename some columns because some column names aren't so obvious. So maybe you will struggle to understand what this column means. For example, and I'm going to rename some of them. So let's do it here fast. And we also practice that rename method. So here I'm going to copy here. Now I write the name of the data frame, rename, open parenthesis, and now we want to change the columns. So I write columns equal to, then open the dictionary, and here the key is going to be—let's say we want to change only these two columns—so I'm going to tell you what they mean. So here I can write the name of the column we want to change, then column, then the value, and now comma, and now the second key or the second key-value pair. So here is the second element, and now I'm going to paste this one. So here it's the second—you know this first FTHG stands for final time home goals. So it means all the goals scored by the home team. So I'm going to replace this name with the home goals name, and here is final time away goals. So I'm going to write only away goals. And that's it. Those are my new names. Now, to update the column names, I'm going to write in-place equal to true. Now I run this. And now I'm going to show this data frame updated. So now this column is named home goals, and this one is away goals. And that's it. In this video, we extracted a single CSV file from a URL with pandas. In the next video, we're going to learn how to extract many CSV files from multiple URLs using pandas.

Okay, in this video, we're going to see how to extract CSV files from multiple URLs with pandas. So, in the previous video, we extracted this CSV file that is listed here. We extracted the Premier League CSV file, and to do that we got the link address and then we used the pd.read_csv method. So we use this link that we have here. And what we're going to do here now is to use this link again, but now we have to concatenate strings. So let me show you here what we're going to do. So this is the link that we used before. So now instead of using the same link, we're going to insert variables. So for example, here I have the link structure. So this is the same as this to show you. I'm going to run this. And as you can see here, I have the link. So this is the same, but I divide it by some elements because I want to show you something. I want to show you how this link is structured. So first we have this chunk that is the root of the link. So this is what all these links here have in common. So if I right-click the second one and click on copy link address, you're going to see here that it has a pattern. So I'm going to show you here. So it's here. So this is the first one, and this is the second one. So first and second. So the two links are almost identical, but the only difference is the name of the link. So in this case is is zero, and this one is E1. And we can also notice some important data like this 2122 that represents the season. So here I have season 2122, and this is represented by these two numbers. So 2122 is the season. So here I divided by some important parts, and this one is the root as I told you before, and this one is the season, this 2122, and then the slash that is here is only, uh, what separates the season from the name of the league. So here is zero is the name of the league, and then you have the that CSV. So this is the whole link, and now instead of writing is zero, we're going to create a variable so we can dynamically extract all this data. So instead of only extracting one link, we're going to extract all the links in a for loop. So here I have all the links that belong to all these five CSV files. And instead of extracting one by one, we're going to extract it in a for loop. So now let's do this. And to do that, first I'm going to create a root variable. And the root is going to be this chunk. So I'm going to write root equal to. And now I copy this and I paste it here. So this is my root because this is what all these links have in common. And actually this is kind of static. This is not going to change no matter what. So here I ran this, and now my root is equal to this chunk of link. So now to make the for loop, we're going to scroll down here, and now first I'm going to create the list. So this is going to be a leagues list. So I write leagues and then equal to open square brackets. And now here we're going to write the name of the leagues that these files have. So here instead of saying Premier League, Championship, League One, the names are E0, E1, E2, and so on. So I'm going to write E0, then E1, then E2, then E3, and then EC. So we have the five elements, and they are in this leagues list. So now I'm going to loop through this list. So as you might remember from the Python crash course, we loop by writing for league in leagues, as it's here. So now I'm going to press enter, and here we have this indent, and here we have to write the code that will be performed on every iteration. So here we have the code already, and it's the same code we used in the previous video. It's this pd.read_csv with a link. So I'm going to copy it. And now I'm going to paste it here inside the for loop. And now I'm going to replace some parts of the link for the variables we created. So here instead of this, I'm going to write the root variable. So here I press the plus and now I write root, and now I replace the root with this part of the link. So now we have the root. Now I'm going to delete this because the root includes the last slash. As you can see here, it has the last slash. And now I'm going to separate this season from the slash. So here plus now quotes now plus and now quotes. So now I have the season here; I have the slash; and I forgot here is the league. So here I have the quotes, then plus, and then quotes. So now I have the season. I have the CSV extension. So now I'm going to delete this name of the season because now we have this league variable. And now I'm going to copy it and replace it here. So now league. And what we're doing right now is to extract the CSV files that belong to these elements in the leagues list. So for example, we're going to start with E0 because it's the first element, and now we're going to get here the root 2021, and here league E0, and we're going to extract this one. In that way we're going to get the CSV files from E0 to EC. Now we have to assign a name to this data frame. So I write equal to and df. So df is going to be the name of this data frame. And now we have to create a list, an empty list, to store all this data. So I write frames. This is the name of my list equal to square brackets. So this square brackets represent an empty list. So I'm creating here an empty list because I want to store all these data frames inside this list. And to do that we write frames.append. We use the append method to store all these data frames inside the list. So here append and then inside we write df. So what we're doing here is to store each data frame on the frames list. And we do this because in each iteration we're going to lose the values inside the df. For example, here on E0, we're going to get that CSV file, and this is going to be stored in df. But then in the next iteration when we get the CSV file of E1, this is going to be stored again in df. So this means that we're going to lose the data that we extracted for E0, and this is going to be replaced with the data of E1 and so on. So we're going to lose the data of each iteration. And to keep the data, we have to append these data frames on each iteration into a list. And that's what we're doing here with this line of code. Okay. Now I'm going to run this code, and let's wait a couple of seconds until this is done. If you get this error, you have to change the encoding of the pd.read_csv here. So I'm going to do it here because I got the error. So here I have to set the encoding to unicode_escape for this to work properly. So now I'm going to run again. And now everything was extracted. So just add this encoding argument in case you got the error that I showed before. So now let's see these data frames. And as you might remember, the data frames were stored in this frames list. So now I'm going to copy the list that was initially empty, and now it's not empty anymore because we store the CSV files of these five leagues. And now let's check the length. So we use the len function, and we're supposed to get five elements because we're supposed to get five CSV files because of these five links. And now I'm going to run this. And yeah, we got five. And now to check each data frame, we have to index this list. So as you might remember, to index this list we have to write the name of a list followed by square brackets and then the index. So if we want the first element, we have to write the number zero, and it's here. So now I press Ctrl-enter, and this is the first element. So as you can see here is E0, the name of the league, and yeah, it's supposed to be E0 because that's the first element here. Now let's get the last element, that it should be index, uh, 0, 1, 2, 3, and 4. So we want this one with index four. So we write four here. Now Ctrl-enter. And here we got this column that says E. So this is the element we're supposed to get.

Great. So far we extracted the CSV files of these five leagues. But now, as you can see in the website, we only extracted the five leagues from this season 2122. But if we want to get the CSV files of the other seasons, for example 2021 and 1920, we have to make another for loop, but in this case for the seasons. So I'm going to show you how to do it. And to do that, I'm going to use the same code here. I'm going to copy it. And now let's make the for loop to extract multiple seasons. So now I'm going to paste the code here. And now I'm going to organize this one. So now it's ready. Okay. And now let's write the for loop that helps us extract the CSV files from different seasons. So first we have to write the for loop here, and I'm going to write this for loop inside the original for loop we got here. So I'm going to press enter. And now I'm going to write for season in, and here to get the seasons I'm going to use a range. So I'm going to write range, parenthesis. This is a range function. And here we have to write the minimum and maximum value of this range. And in this case, I want to extract the seasons from 15 to 20. So I'm going to write 15, comma. And here instead of writing 20, we have to write 21 because range excludes the last element. So it's like 21 minus one. And here now I write the colon, and now I'm going to introduce this block of code inside this new for loop. So I press tab, and now is inside. And now I'm going to show you what I'm doing here. So I'm going to copy this so you can see in detail this. So now I'm going to print and write season. Now I run this, and here I didn't write correctly. So now season, and here as you can see all the seasons are printed. So here we got 15, 16 until 20, and this is great because we want to build something like this. This is the format of the season in the link.

that we seen before. So here we can build this using this season variable because we already have the first year. So in this case, for example, it's 2021 until 2022 or only 2122. So we already have the first one. So we only have to create the second one. And we can get this structure of year one and year two together by concatenating the seasons. So here I'm going to show you how to do it.

So first we have to make this an string because right now it's a number, and if we want to concatenate this, this object has to be an string, not a number. So now to concatenate, we use this plus operator and then write str again. Now parenthesis and now season plus one because the first one is the first year and the second one should be the year plus one. For example, here 21 22. So now I run this and let's see the result. So here, as you can see, we got 15 16, 1617, and so on.

So now, instead of writing 2122 that only allows us to extract the CSV files from this season, we're going to copy here this and we're going to replace it. So here we delete this and paste that. So here we have the format we want it, but now the values are going to increase with this for loop. So first it's going to be 1516, then 1617, and so on, and we're going to get the CSV files from the past six seasons. So that's what we wanted.

So now we have to add one more little detail here. And this is a new column because here the data frames don't have a column that helps us recognize what is the season that this data frame belongs to. So, for example, here in this data frame, we don't have a column that says season or year. We only have the date. So to easily recognize which season corresponds to a data frame, we're going to add a season or year column. So here I'm going to do that with the insert method.

So we've seen different ways to add a new column, and we're going to use here the insert method because I want this new column to be in the index one. And the insert method allows us to put a column in a specific index. So now I'm going to show you: here I write df.insert, and here we write that index. We want to put this new column. So here I want it to be in index one. Then we have to write the name of the new column, and I'm going to name it season. And then we have to introduce the values. So here I only want this season column to have the value of the first year, or well the first season, let's call it. So I only want this. I'm going to show you here. I only want the first one. So the first year that in this case is 2015 or only 15. I want it to be in this season column. So not the second one as it was here. So only the first. So here I'm going to write season, and I'm going to get this because I'm doing here a for loop through this range. So this is going to be in my season column.

And finally, here, to avoid making the video so long, I'm going to delete some of the elements so we get fewer CSV files. So here I'm going to get rid of some of them. So, for example, here E1 and then EC because before there were five, and here there are six seasons. So 6 * 5, we're going to get like 30 CSV files, and it will take some time. So now with three, it's going to be faster. And now you can also remove this encoding argument because it's going to work fine with these three seasons. So it's only for the sake of this video. Now I'm going to run this code, and let's wait a couple of seconds until this is done.

All right, the execution was finished, and now we can see the content inside the frames list here, and we're supposed to get 18 data frames because there are three elements here and six seasons here. So 18 in total. So let's verify this by using the len function. I use length, and here we got 18. So it's correct. We have 18 data frames. And now I'm going to show you here the first data frame. So here I write frames square brackets zero, and we're supposed to get the data frame of leak e from season 1516. So let's check here. I run, and as you can see here, we have the season 15. So it's correct, and the leak is zero. So yeah, it's working fine. Now let's check fast the last element. So there are 18 elements in this frames list. So the last element should have the index 17. So here I write frames square bracket 17, and we get the season E3, sorry, the season 20 and the league E3. And let's check here. So E3 is the last element, and the last season is 20. So here 20 E3. So yeah, it's correct. And that's it. In this video, we learn how to extract multiple CSV files using pandas.

Okay. In this video, we're going to store all the data frame extracted with pandas into a dictionary. Dictionaries help us properly manage data. And in this case, I created a dictionary called dict_countries, and I have as key the name of the league, the actual name of the league, and the value is the code that the website, this website, uses for each league. So, for example, Premier League has the code E, and Championship I think it has E1. So here we have the original name of the league, and here we have the code. So if we want to get access to one code, for example, we want to get SP1, we can write only the name of the key. And since it's a dictionary, we're going to get the name of the value first. Here I'm going to run this dictionary. And now I'm going to run this one. And here, as you can see, I got the SP1 because this is the value of the Spanish La Liga key.

In the same way I created this dictionary, I'm going to create another dictionary, in this case called dict_historical_data, to store all the data frames that we extracted here in this previous video where we extracted multiple seasons and CSV files of multiple leaks. So all of that is going to be stored into a dictionary. So we don't have to use the list indexes as we did in the previous video, but only use the key that represents the actual name of the league. So the first thing we're going to do is to copy the code here that we use to extract CSV files from multiple seasons and leaks. So I'm copying this that we created in the previous video and I'll paste this here. So I paste it here. And now I'm going to modify this one. And we're going to loop through key elements. So here before we looped through a list, which is this list, but now we're going to loop through a dictionary, and it's going to be this dictionary I created before. So here I'm going to write for league in dict_countries. So this dictionary contains some leagues like Spanish La Liga, uh, Bundesliga, and English Premier League, and also the codes of each league. So, for example, English Premier League has the code E0, and I got these codes manually, and you can get these codes yourself by going to the website and copying the link address. And then you're going to get, for example, SP1, D1, and any code of any leak you want.

Now I'm going to show you what we get when we loop through a dictionary because here we're looping through a dictionary. So just a recap here. So for league in dict_countries, now I'm going to write print league. So let's check what we get here. If we print this, we get the keys. So Spanish La Liga, German Bundesliga, and so on. So these are the keys. But if we want to get the values, we have to use this syntax to get the values of a dictionary, which is this. And now Ctrl enter, and we get the values. So, as you might remember, this is the syntax we use to get the values. So now we can use this syntax to replace the name of the leak or the code of the leak. So here I'm going to show you this league variable represented the code of the leak here. Before we use this leaks list that I'm going to uncomment. So we used this links list that contains these codes, and we loop through this list. So this represented E0 and E2 and E3. But in this case, we don't have the leaks list anymore. And now we're using a dictionary. So to get the same codes, we have to use this syntax. So we're going to replace this league variable. So I double click here and I delete. And now I paste this new syntax of the dictionary. So I copy now. And now I paste it here. So now this link here is ready.

And now the first thing we're going to do is to concatenate all the seasons that correspond to a same league. So, for example, if we have the Spanish La Liga, we have to concatenate all the seasons from 15 to 20. So to do that, we only have to use the concat method. We didn't use this method so far, but this is really simple. So I'm going to explain you how it works right now. So to use the concat method, we write pd.concat, open parenthesis. And now we have to write the object we want to concatenate. In this case, what we want to concatenate is this frames list. So I copy this and I paste it here. And as you might remember, this frames list contains the data frames of all the seasons that we got here in this range. So 15, 16 until 20. So by doing pd.concat, we can concatenate all the seasons from the season 15 to the season 20. So all of them will be in just one data frame. So I'm going to assign this to a new variable that is going to be df_concat. So this one represents to all the data frames we concatenated. So all the seasons from season 15 to season 20. So now is in just one data frame. And by the way, we put it here outside this for loop because if it's inside, I'm going to do it here. If it's inside this for loop, we're not letting this frames list to append the data frames from season 15 to season 20. So what we have to do is to put it outside this for loop that iterates over the seasons.

And now that we have this df_concat that represents all the seasons of a league, we can assign this data frame to a dictionary. In this case, I created this dictionary as I told you before, dict_historical_data. And this is going to be my dictionary that contains all the data frames I want. So here I'm going to assign this by writing the name of the dictionary. And now I open square brackets. And now I write the name of the league. So in this case, the name of the league is stored in this league variable. And then I set this equal to the df_concat. So what we're doing here is assigning a key that has the name of a leak, and this key is going to have a value, and this value is going to be the df_concat. So the league is going to be the key, and the df_concat is going to be the value, and the df_concat contains all the data frames from season 15 to season 20.

Great. This is almost done. But now we have to make one little change here. We have to put this frames empty list inside the loop. So here, and now I'm going to explain you why we have to do this. So if this empty frames list is created outside the loop as it was before, what is going to happen is that here when we append a new data frame, we're going to accumulate all the data of the previous league. So, for example, we start with Spanish La Liga, right? Then we iterate for season 15 to season 20, and we get all this data and we concatenated it. So that's fine. But then when we go to the next key, for example, German Bundesliga, what it's going to happen is that the previous data of Spanish La Liga is going to be here. And when we concatenate this with the concat method, we're going to get a data frame that contains the data of the Spanish La Liga and also German Bundesliga. And we don't want that. We want the data frames to have independent leaks. So what we have to do is to delete the data of the previous league. And we do that by introducing this empty frames list inside the loop. So here, every time we iterate over this dictionary, we're going to create a new frames empty list. And now when we go over the second iteration, for example, here German Bundesliga here, when we get to this line of code, the frames object is going to be an empty list. So this means that all the data that was stored for the Spanish La Liga is going to be lost, and that's what we want. But all the data is actually stored in this dictionary. So we are not actually losing the data. We're just removing the data from the frames list. But all the data is stored in this dictionary.

Great. Now everything is ready. And now I'm going to run this code. So now I press Ctrl enter, and now let's wait a couple of seconds or maybe 1 minute. Okay, now the execution is finished, and now let's check the data that was stored inside this dictionary I created here. So now I'm going to show this, and we see the data inside this dictionary. However, it's not readable. So here first I'm going to show the keys of this dictionary. So I write keys. This is the keys method. And as you can see here, I got the keys. So the keys are represented by the actual names of each competition. So I have Spanish La Liga and English Premier League. So to get access to one of those data frames, we only have to use the syntax to get a value from a dictionary. So, for example, let's say we want to get the English Premier League data frame. So we only write the name of the key, in this case, English Premier League. And that's it. Now I run, and we got this data frame that contains the data of season 15, 16, 17, 18, 19, and 20. And also we can see that this is English Premier League because here we got the E0 that is the code for this competition. And as you can see here, this is an easier syntax than writing the index because the index actually doesn't tell you much about the data. So in most cases, it's more convenient to use a dictionary to store data frames because we can use keys, and it's easier to memorize a key name. And it's not so easy to memorize an index when we store data frames inside a list. And that's it. In this video, we learn how to manage all the CSV structed with dictionaries.

Okay. In this video, we're going to learn how to filter a data frame based on one condition. In pandas, we can filter a data frame based on conditions, just like on Microsoft Excel, but here we use methods. So let's start by importing pandas as usual. Import pandas as pd. And then we're going to use a different data frame. And in this case, is a laptop price data frame that I have here. And you can download this data frame in the notes in the description. You can find it there. And we're going to read it with the pd.read_csv. So I'm using this method, and I'm assigning this to a variable named df_laptops. So that's going to be my variable. So now I run this code, and now let's see the first three rows. So here we have the first three rows of this data frame. And now let's check the columns that this data frame has. So first we have the ID. Usually IDs are unique. So you won't find duplicate values. And then we have the companies. Here is the name of the company that this product belongs to. Then we have the name of the product. So here we have MacBook Pro and other types of laptops. Then we have the type of the laptop. So ultrabook, notebook, and so on. Then we have the the screen size. So this one is in inches. And then we have some other values that we're not going to see so much like the screen resolution, CPU, RAM, memory, uh GPU, and some others. Uh but this one, price in euros, we're going to use it. So here are some of the columns we're going to use to filter the data frame in this course are the price in euros here and also the company and the product and some others like the screen size in inches. So this was just a little review of this data frame, and now let's filter the data frame based on one condition.

So to see how to filter a data in pandas, let's solve this task that we have here that says find which rows have the word apple in the company column. So we have to make like a comparison. We have to compare two objects. So here we have to compare the company column with this string that is apple. So in Python, as you might know, we compare with the equal sign. So we write the equal sign twice, and this means comparison. So this is like we're comparing two values. For example, if I write here one is equal to two, and we run this, we get the value of false because these two values are different. But now if I write one is equal to one. Now we get true because these two values are the same, and the same way we compare numbers, we can compare other type of objects like series. So here we can write the name of a series. So, for example, I can write the series of the company column. So here I write df, and now I'm going to copy the name of the data frame. So I write df_laptops here, and then to create a series also known as a column, we open here the quotes, and then we write the name of the column. So this is a series, the company series or the company column, and we have to compare this to the apple string. So we write Apple, and we're comparing these two values. So here we want to know if the values inside the company column are equal to Apple. So, for example, let's go to the data frame, and here in the first row, it's going to compare. For example, Apple is equal to apple, then true because those are the same. So we get true. Then the second apple equal to apple, true. And then the third one, apple equal to HP. And then we get false. And that's how we're going to build this uh new series because we're going to get from this a new series of values that are true or false. So now let's verify this. So I'm going to run this one. And as you can see, we got the values of true and false. So the first three values are true, true, and false because apple is equal to apple, true, then apple equal to apple, true, and then the last one is different to false. And that's how all the rows were filled.

So now that we got this series, we can filter the data frame based on these results. So to do that, we have to first copy this code, and now I'm going to paste it here. And now to filter the data frame, we have to follow the following syntax. So first we write the name of the data frame, and then we put this code inside square brackets. So write square brackets, and this is the syntax. So inside the square brackets, we write the condition, and outside we write the name of the data frame. This is the syntax. So now let's filter this data frame based on this condition. So I press Ctrl enter, and now let's see if the data frame was filtered. So here I got this result, and let's verify if all the values inside the company column are Apple. So here it says Apple, and let's scroll down. All of them are Apple, and yeah, all of them are Apple. So we successfully filter this data frame.

Now I want to give you one tip. So sometimes when we have larger data frames, it's not so easy to verify that all the values have this type of string. So, for example, here we could easily check that all the values were apple because it's a small data frame. But for bigger data frames, this is not going to be so easy. So in those cases, we can use the value_counts method. And I'm going to show you how to do it here. So we actually uh already learned what the value_counts method does, but in case you don't remember, the value_counts method counts all the values inside a column. So here we only have to write value_counts

open parenthesis and inside we write the name of the column. So I write company. So write company and then we run this and we got here that there are 21 rows with the word Apple in the company column. So we can easily see the categories inside the company column. And that's it. That's how you filter a data frame based on one condition.

All right. Now is your time to filter a data frame based on one condition. So here I made two exercises that will help you practice what we learned in this video. So the first exercise consists in finding which rows don't have the HP in the company column. So this is similar to the example we've seen in the video but in this case it's like not equal to but is the different symbol. So once you build the code of this condition you have to filter the data frame as we did in the video. And then in the second exercise you have to find the laptops with price over €2,000. And in this case you have to use a different column. So here you have to use the column price_euros for exercise number two and for exercise number one the company column. So now you can pause the video to solve these two exercises on your own and after you finish them you can continue watching this video to solve these exercises together.

Okay, now let's start solving these two exercises and let's start with exercise number one. So here we have to find which rows don't have the HP word in the company column. So this is similar to the example we've seen before. Before we use the equal sign to compare two objects, but in this case we have to use the sign that represents not equal to. So here I'm going to show you. So this one is equal to the sign equal to and this one is the sign not equal to or different. So we have to write this exclamation mark with the equal sign. So let's test it out. I write one not equal to two and it says true because these two values are different. But now one not equal to one. And now it's false because these two values are equal. And now we're going to compare the series with the string. So here I write the name of the series which is DF_laptops and then open square brackets. Then we write the name of the column in this case company. And then here on the right we have to write the name of the brand in this case HP. So here we're comparing these two values and I almost forget I need to write the exclamation mark. So now it says this series is not equal to HP. So we're comparing that. Now I press Ctrl enter to compare these two values. And now we got the uh the result that is a series. So now let's verify very quickly the original data frame. I'm going to show only the first three rows. And now let's compare quickly. So HP is not equal to Apple. Then true. So we got true. Then again HP not equal to Apple. True. and then HP not equal to HP false. So everything is correct and we got here the result and we have to introduce this inside square brackets to filter the data frame as we've seen before. So here we write the name of the data frame square brackets and inside square brackets this condition. Now we run to filter that data frame and we get this new data frame and here in the company column we get all the companies except for the HP company. So here we can see that there is Apple, Acer, Lenovo and other companies but not HP. So now to verify which companies are in this column, we can use again the value count method. So let me write it here parenthesis and inside the name of the column we want to count. So here I press Ctrl enter and now we can see how many rows have the value of Lenovo and Dell and so on. So we can see that there are a lot of Lenovo laptops but none of them are from HP because we used this condition. So we filter HP out.

Okay. Now let's solve the second exercise. And here we have to find laptops with price over €2,000. So in this case we have to compare with this this sign that is greater than and in this case it works like this. So we write one greater than two for example here we got false because one is not greater than two but if we write three greater than two it's true and that's how it works. So here on the left we write the series name in this case I'm just going to copy here uh this syntax and now I'm going to paste it here. So here this is the series also known as column and here instead of writing company I'm going to change it for the price column. So I'm going to copy this column which is price_euros. So I copy this and now I paste it here. So here I have my column price_euros. Now on the right we have to write the €2,000 here. So here I'm going to write the number 2000 and that's how we're comparing. We're comparing the price_euros column with this 2000 value. So we're going to see which values inside this series are greater than 2,000. So let's press ctrl enter and let's find out. So here we can see that uh the first three values are false. So they might be uh less than 2,00 and the fourth is true. So this one is greater than 2,000. So now let's filter the data frame with this syntax and let's run this and let's verify very quickly the values in the price in euros. And here, as you can see, all the values in this column are greater than 2,000. And that's it. I hope you successfully solve these exercises.

All right, in this video, we're going to learn how to create a column based on one condition using the where method. So here first we're going to start by importing pandas as usual and reading the same CSV file we worked with in the past video. So here we have the same CSV file. Now to create a column based on one condition we have to import numpy because this where method belongs to numpy. So here we import numpy as np and we run this one. So here to show you how the where method works. We're going to create price tiers based on the price_euros column. So here we're gonna use this column and we're going to compare these prices with another price to determine if this is either cheap or expensive. So let's use the number 2000. And let's say that if a laptop costs more than 2,000, this is considered expensive. And if it costs less than 2,000 is considered cheap. So here we can create this with a condition. So as you might remember from that previous video, we make conditions by comparing two objects. So in this case, I'm going to compare this uh price_euros series. So I write df_laptops and then I write the name of this column and I put it inside quotes. So now we have this series. I'm going to run it. And now we have to compare this with the value 2000 as I told you before. So if this is greater than 2000 then this is going to be true. And now we're going to compare these values one by one with the value of 2,000. So for example, the first one is less than 2,000. So the first one is supposed to be false. So here I run this and we got the first value of false. So now this is what we learned in the previous video. But now we're going to use the np.where to determine which value is going to replace the false and which value is going to replace the true. So I'm going to show you here. I write np.where then open parenthesis and inside the first argument is going to be this code. So the condition this is our first argument. Then the second argument is going to be the value we want to set when this condition is true. So when this is true I want to set the value of expansive. So it's going to be expensive when this is greater than 2,000. And the third argument is the value we want to set when this is false. So when the value is less than 2,000. So in this case I want to set it to cheap when is less than 2,000. So the second argument is the value when the condition is true and the third argument is the value when the condition is false. Now I'm going to run this one. And as you can see here, we don't have any more the true or false values. But now we have the values of cheap or expensive. Well, here we only have cheap. But this is only the summary. If we check this array in detail, we will find many expensive values. And now we can set this array to a new column as we learned in this course. So first to create a new column we write the name of the data frame in this case df_laptops. And now I uh write this square brackets. And now I write the name of the column I want to create. In this case I'm going to name it price_tier. And now I'm going to set this equal to this array. So I copy this one and paste it here. So now my new column will have the value of this array which is here. So now I'm going to run this uh cell and now I'm going to show the first five rows of the df_laptops data frame. So here I use the head method and now we can see the first five rows. Now if we go to the last column here we have a column named price_tier and this one has the values of cheap or expensive. So let's verify fast the condition. So here the first one is less than 2,000. So we got the value of cheap because this one is false. So it's correct. And this one is greater than 2,000. So we got the value of expensive. And this column was successfully created. Now we can count the values inside this new column by using the value_counts method. So to do that we only write the name of the data frame. Then we write value_counts then open parenthesis and now the column we want to count. So here the column is price_tier. So now I run this one and we can see that almost 1,100 laptops are considered cheap and only 137 laptops are expensive. And that's it. That's how you create a column based on one condition with the where method.

Okay, now it's your time to create a column based on one condition with a where method by solving this exercise. In this exercise, you have to create an array based on the screen size. So in this one, you have to use this inches column that is right here and you have to compare the values. So let's say that if the size of the screen is greater than 15 in we can say that the screen is big but if the screen size is less than 15 in the screen is small. So here I'm going to say if the screen size is greater than 15 that's going to be the condition. So here is the condition and you have to create an array based on this condition. So you have to do this as your first task. Then you have to set this array as a new column as we did in this video. And then you have to show the first five rows and count the values inside this new column that I'm going to name it screen_size. But you can name it as you want. And that's it. You can pause this video and try to solve this exercise on your own. And after you finish this exercise, you can continue watching this video to see my solution.

Okay. To solve this exercise, I'm going to create first a condition that says that the screen size is greater than 15. So first I write the name of the data frame df_laptops. Then I open square brackets and I write here the name of the column inches. So here I write inches. Then I compare this with 15. And this is going to be my condition. So here we got the values of false and true. And now if we use the where method we will be able to replace these values of false and true for the categories we want. So here I'm going to write as first argument this uh condition and then the value when this condition is true. So if the screen size is bigger than 15, the value that is going to be assigned is big. And if that's false, so the screen size is less than 15, the value is going to be small. So this is my uh array that I'm going to create here. And I have here an array that contains the values of small and big. Now I'm going to set this to a new column. So I write here the name of the column that I want to create. So it's going to be name screen_size. And then I'm going to set this to this array. So I copy this one and paste it here. So this array is going to be set to my new column screen_size. Now I'm going to show this uh this data frame, the first five rows of the data frame. But first I run this code. And here I got an error because I didn't write the S here. So I wrote the S. And now I run this one. And everything is okay now. So now I'm going to show the first five rows. So I write this and use the head method. And as you can see here, I have here the first five rows. And if I go to the last column here, we have the screen_size column. So let's quickly verify if it was successfully created. So here in the size the first value is less than 15. So this one is supposed to be false. So we should get the value small and it's correct. And now the third one is greater than 15. So this one is true and we should get the value of big. And yeah we got big. So this column screen_size was successfully created. And finally, let's count the values inside this screen_size column. So here I'm going to use the value_counts method. So I write value_counts open parenthesis. And now I write the name of the column. So I'm going to copy here and I paste it here. Now I run this one. And as we can see here, there are 835 laptops that are considered big and 468 are considered small because their screen are less than 15 in. And that's it. I hope you successfully solve this exercise.

All right, in this video, we're going to learn how to filter a data frame based on two or more conditions. So to start with this, let's import pandas and read that CSV file that we used in the previous videos. So here I have the file. And now it's stored in the df_laptops data frame. So here's the data frame. And now before filtering a data frame based on two or more conditions, let's make a recap of how to filter a data frame based on a single condition. So here in the previous video we learned how to find for example Apple laptops. So we only compare the company series with the Apple string. So here we use the double equal sign to compare these two. So we run this and we got this true or false values. And the same goes for laptops that cost more than €2,000. So we compare this one with the 2,000 number. And in this case, we use this sign and we compare this series with this number. So I run this one and here I got the results true or false in this array. So now what if we want to get Apple laptops that cost more than 2,000. So condition one and condition two together. So in this case we have to create a multiple condition and we have to use either the and or the or operators. So here this is the and this is the or operators. So here let's do this. Let's find Apple laptops that cost more than €2,000. So to do this first I'm going to copy the the single condition that finds Apple laptops. So I copy this one. Now I paste it here. And now let's write the second condition that finds laptops that cost more than €2,000. So I copy this and now I paste it here. So now we have condition number one and condition number two. So now to filter a data frame based on multiple conditions, first we have to separate this condition one and condition two with the parenthesis. So here I write parenthesis and inside parenthesis is my first condition and I do the same for the second condition. So I write parenthesis and inside this is my second condition. Now we have to use either the and or the or operators. And here we have to find Apple laptops that cost more than €2,000. So this has to be Apple laptops and the cost has to be more than €2,000. So we have to use the and operator. So I write the logical operator and this is the and now I execute this. So here before I execute I'm saying first condition and second condition. So here we have to write the parenthesis as we did and then the logical operator in the middle as we did. So now I run this one and here we can see the result. So here is an array with true or false values. And now to filter the data frame, we have to copy this condition this multiple conditions and we paste it here. So we're going to filter the data frame based on this multiple condition. So first we write the name of the data frame df_laptops followed by square brackets and inside square brackets we write the multiple condition. So I paste it here. And now we can run this to see the results. And here I got an error because I didn't write laptops correctly. I need to add that P. And now I run again. And now everything is fine. So here I have the data frame that is the result of this code. And let's verify if it's correct. So here in the company column, we're supposed to get Apple. So here let's verify company. It says Apple, Apple and only Apple. So here this first condition was successfully evaluated. So now the second condition says that the price should be greater than 2,000. So let's check this price column. And now we can see that all the values are greater than €2,000. So we successfully build this multiple condition and filtered this data frame. So now before we continue with the or operator, you should know that in Python there is this and and or keywords. So here those are logical operators but in pandas we have a special operators. So instead of writing and in pandas we use this sign that we used in this example and instead of writing or in pandas we use this sign here. So it's just the syntaxes we have to follow in pandas. So try to memorize them. And now I scroll down and here is the or operator. So now let's continue and let's find laptops that are Apple or Dell. So in this case we have Apple or Dell. So now as you might expect we have to use the or operator. So here I have the individual conditions. So first the first condition is this company array equal to Apple and the second one is this company rate equal to Dell. So now I'm going to create the multiple condition based on this uh simple condition. So here I'm going to write the first one then write the parenthesis. Now the second one. So now parenthesis. So here I have condition number one and condition number two grouped. And now I have to add the operator. So here is the or operator because we want either Apple laptops or Dell laptops. So here I run and we build this multiple condition. So now let's filter the data frame based on this multiple condition. So let's copy this code. And now I'm going to paste it here. And we have to follow the same syntax to filter a data frame. So we write laptops then open square brackets and inside we put the uh multiple condition. So now I run this one and here I got the result. So now let's verify quickly that company column should have either Apple or Dell. So let's check here is the company column and here we see Apple Apple and here Dell. Well, the data frame is a bit longer. So, it says Apple and then we cannot see what's in the middle. But then it says Dell and then Dell Dell and yeah, only Dell. So, apparently this column only has Apple and Dell. But let's verify this by using the value_counts method. So, I write value_counts. Now, I write the name of the column. And now I run this one. And as we can see there are 297 Dell laptops and 21 Apple laptops. So yeah, we successfully filter this uh data frame based on two or more conditions.

Okay. Finally, let's challenge ourselves with this last example. So in this case, we have to find laptops from Apple or

Dell that cost over €2,000. So I have here each condition separated, and the first one it says equal to Apple, then equal to Dell, and then the price greater than €2,000. So here we have all the simple conditions.

So now we have to create a multiple condition based on this one. So based on these three simple conditions. So first we're going to write the code that corresponds to Apple or Dell. So here first we select the first condition, then the second one, and we write the parenthesis to group the conditions. So here parenthesis, and now here between these two conditions I write the or operator. So here it's or because it's either Apple or Dell. And then it says that it should cost over €2,000. So here I copy the third condition and paste it here. And now I add parenthesis. And now since it should cost over €2,000, the operator we need is and because the laptops have to cost over €2,000. So now I wrote here this and operator, and now this is almost finished.

But if we leave it as it is, we could get some unexpected behavior because Python could group these conditions in an undesired way. So here we have to give clear instructions. So to do that we have to add parenthesis. So here we want this multiple condition I'm selecting right now to be evaluated first. So if we want it to be evaluated first, we have to add a parenthesis to group this first multiple condition. So with this parenthesis that I added here, we're telling Python to evaluate this first multiple condition. And in this way, we're giving Python clear instructions of what we want to accomplish.

So now a little recap. This first multiple condition gives laptops from Apple or Dell. And now we're grouping here with this parenthesis. So this one is going to be evaluated first. And then this third condition gives laptops that cost over €2,000. So after evaluating this first multiple condition, we're going to get laptops that cost over €2,000. So now hopefully that's clear to you.

Now I'm going to copy this, and I'm going to filter the data frame based on that condition. So here I write the name of the data frame. Then I open square brackets, and inside I paste the code we wrote before. So here is the code, and now I'm going to run this one. So here I press Ctrl enter, and now we can see the data frame.

So now let's quickly verify the results. So here the company column should have only Apple or Dell. So here in company says Apple and Apple, Apple, and then Dell, and then Apple, and only Apple and Dell. So that's correct. And now then we have to verify if the price column is over €2,000. And here in the price_euros column, we can see that all the prices are over €2,000. So here the first one is over 2,000, and the second one, and the third one, and all of them. So we successfully build this multiple condition and we filter this data frame based on two or more conditions. And that's it. That's how you filter a data frame based on two or more conditions in pandas.

Welcome back. In this video, we're going to create a conditional column from more than two choices using the select method. So, as usual, we're going to import pandas and read the CSV file laptop price that we've been using in this course. And now I'm going to create a conditional column for more than two choices. So this is similar to a previous lecture where we created a conditional column from only two choices using the word method. But in that lecture we had only two choices that were cheap and expensive. But in this case we're going to have more choices like affordable, expensive, and too expensive. So when you have more than two choices, you have to use the select method. And I'm going to teach you how to do it in this video.

So first we import numpy as np. So I run this one. And now to create an array based on multiple choices, we have to create first two variables. First the condition variable, which is this one. And this is a list. So here I open with square brackets, and inside we're going to create the condition. And the second one is the values. So here I write values, and this one is also list.

So let's start with the values. Here we introduce all the values that we're going to assign based on the condition that we're going to create. So in this case we have four values as I mentioned before. So in this case the first one is too expensive. The second one is expensive. Uh the third one is going to be affordable. And the last one cheap. So these are my four categories. And now I'm going to create a condition for each of them. So here I write the first condition, and I have to write the name of the data frame df_laptops. Now square brackets open quotes. And here we'll use the price column, which is this one, the last one, price euros. So we use price because we're going to decide whether a laptop is too expensive, expensive, affordable, or cheap based on the price. So let's say that if the laptop costs more than €3,000, this is going to be too expensive. So the first one will correspond to the first element in the values list.

So we continue with the second condition. And now I copy this so it's much easier. And here we're going to create the condition for the expensive value. And let's say that if a laptop costs more than 2,000 and is less than 3,000, uh less or equal to 3,000, then is expensive. So here I have two conditions. This one is the first, and this one is the second. So as you might remember we have to separate both of them with parenthesis. So here I write parenthesis, and now I write here parenthesis, and to create this range because the expensive value is a range between €2,000 and €3,000, we have to introduce here a operator. In this case we need the and operator because we want it to be greater than 2,000 and less than 3,000. So at the same time. So now I write and, so now we have this range. So this one corresponds to expansive.

Now let's create the third one. So I add comma, then press enter. And now I'm going to copy this one. So I can create the third range much faster. So here now let's say a laptop is affordable if it costs more than €800 but less or equal to €2,000. So that's the range for the affordable value. And the last one is going to be cheap, and this is uh when is the price equal or less than €800. So that's it. We have the conditions and we also have the values. So now I'm going to delete this, and I'm going to run this code. And here I got an error because I forgot to include here the comma. So I write comma, and now it's ready. So now I press Ctrl enter. And now it's done. So we have the conditions variable and the values variable.

So now let's create the new column. So first to create a conditional column with the select method we have to write np.select, then open parenthesis, and here we have to introduce two arguments. The first one is the conditions, in this case my conditions variable, and the second one the values you want to assign. So here I have the values in my values variable. So here I write values. So these are the two arguments we need to insert inside the select method. So now if we run this we see that we get an array. So this array should be our new column. And here in this array we see the values of affordable, affordable, cheap, and all the values that correspond to the price that are in this price euros column. So now we're going to create a new column. So I write equal to, and here I write this name of the new column. So here I write the name of the data frame first. So df_laptops open square brackets, and here inside quotes I'm going to write the name of my new column. So it's going to be price tiers, and here it's ready. So now I run this. So I have my new column created. So now I'm going to show it here so you can see it much better. I go to the last column, and it's right here. So it's my price tiers column. And as you can see here, we see the values affordable, cheap, expensive, and so on.

So let's check if the values were assigned correctly. So here, let's check the first one. And its price is 1,300. So this one is between €800 and €2,000. So the third range that we created here. And the third corresponds to the affordable. So let's verify here. And yeah, it says affordable. Let's do it one more time. So in this case with this one, it says 2500. So let's check here. 2500 belongs to the second range because it's greater than 2,000 and less than 3,000. So the second range corresponds to this second value expensive. So here it says expensive. Okay. So we successfully created a conditional column from more than two choices using the select method. So now let's count the elements inside this price tiers column. So I'm going to do it right here, and I write the name of the data frame. So df_laptops open square brackets, and here I paste the name of the column. And to count this series or this column, we have to use the value counts method. So I write value counts parenthesis. I run this one. And by the way, you can uh either put this inside parenthesis or put it here as I did before. It has the same effect. And now we can see that the majority of laptops are considered affordable.

And that's it. Now let's do an exercise so you can understand much better what we learned so far. Okay, this exercise is similar to the one you solve before in the wear method, but in this case we're going to use the select method to create four categories. So in this case, we're going to create four categories based on the screen size. So we're going to use the inches column that is right here. And we're going to get four categories. And here we have the four ranges that we have to follow to solve this exercise. And after this, you have to create a condition and values variables. So once you have all of these, you set it to a new column as we did in this video. Then you show the data frame and you count the values inside this new column.

Okay. Now you can pause the video to try to solve this exercise yourself, and then you can continue watching to see my solution. All right, let's start by creating the conditions variable. So I write conditions equal to square brackets here. And now here is going to be the conditions that we're going to create. Now also the values list. And here let's write the elements. So the first one is too big, and here I write too big. The second one is only big. The third one is small. So I copy and paste it. Then we have too small. So I paste this one and write too small. So here we have the four values. And now let's write the conditions. So here I write first df_laptops which is the name of the data frame and inside we write the name of the column we want to evaluate, in this case is here and is inches. So now I scroll down and paste it here. So I paste it, and now I'm going to compare this with 16. So if this is greater than 16 it's going to be considered too big. Now I'm going to write the second one. So here I copy and now sorry I paste it here. Now if this one is greater than 14 but less than 16 then it's going to be big. So I write this one here. So less or equal to 16. And I write it here. Now we write parenthesis to separate both conditions. So here now the and operator, and now we have it here. It's ready. So now I copy and paste this one. And now I delete this blank space. And now I press this tab. So now this one is greater than 12 and less or equal to 14. And the last one should be less than 12. So I write this now 12 less or equal to 12. So here I delete this one, and now it's ready. So I have the conditions and I have the values here. So now I run this, and it's ready. So now we create this new column. So we write np.select, and the first argument is the conditions variable here, and the second one is the values. And now I run this one, and we have this array. So now let's create our new column which is going to be named screen size. So df_laptops now open square brackets and I write screen size. So I assign this to this new column, and now I show the data frame. So here I paste it, and now I run this one, and let's check the last column, and it's here. So I have my new column screen size, and now let's count the number of elements inside this new column. So I I'm going to use here the value counts method. So I write df_laptops now square brackets and I write the name of this new column which is screen size and then I write the name of the method value counts open parenthesis controll enter, and now we have that the majority of laptops are considered big, and that's it. I hope you successfully solve this exercise.

All right. In this video, we're going to learn how the is in method works in pandas. We're going to start by importing pandas and reading the CSV file we've been working with in this course. So, we have here the laptop price CSV file. And now it's here in the df_laptops data frame. And here our first task is to select Apple or HP laptops. Before we did this task with another methods, but now we're going to do it with the is in method. So first we start writing the name of the data frame. So df_laptops and then we write the name of the column we want to apply this filter. In this case is the company column because this one has values like Apple, HP and so on. Then we have to use the is in method, and we only have to write dot is in then parenthesis, and here we have to introduce data in form of list. So here if we want to select apple and hp we have to open the square brackets to start a list. So here I open square brackets and then the first element should be apple. So I write Apple and the second one HP. So now what we're saying here is select the Apple and HP element from the company column or company series. So we're going to filter out all the elements that are not Apple or HP. And now let's run this code. So I press Ctrl enter. And as you can see here I got a series with true and false values. And here it made a comparison between this Apple and HP with this company column. So here for example the first three rows we have here the data Apple, Apple and HP. And here we're saying look for Apple and HP in the company column. So the first three rows will be true. So here we have true, true and true. And basically that's how the is in method works. And now let's filter the data frame based on this condition that we created. So here we have to write first the name of the data frame. So I write df_laptops and now I open square brackets and as usual here inside square brackets we have to introduce the filter or condition. So here I copy and now I paste the code we wrote before. So here I have the filter or condition. So now we're going to filter this data frame. So now I press Ctrl enter, and as you can see here the company column only has Apple or HP data. So here it says Apple Apple HP and so on. So now let's verify if this column only has apple and HP. So we're going to use the value counts method. I write value counts. Then inside parenthesis I write the name of the column. So I write company. Now I press Ctrl enter. And here we got that there are only HP and Apple in the company column. So great, we successfully filter our first column.

Now let's continue with multiple filtering. And now our task is to find notebooks or ultrabooks from Apple or HP. Here we have two conditions. The first one is notebooks or ultrabooks, and the second one is from Apple or HP. So here we have to use the operators as we did in previous videos to mix the condition one and condition two. Now we're going to write the code for each condition. So let's start with condition one. So I write filter one and equal to and now I write the code. So here it says find notebooks or ultrabooks. This type of data is inside the type name column. So here we have the typename column and it says ultrabook or notebook. So now we're going to filter that column. So we write the name of the data frame df_laptops. Now I write the name of that column. Let me copy it here. That's why I copy this one and I paste it here. And now we use the is in method. So we write is in parentheses square brackets and inside we write the uh the elements we want to get. So here notebooks actually I'm going to copy it from this one because it's written in a different way with capital letters here. So notebook and the second one is ultrabook. So here notebook and ultrabook and here we're selecting notebooks and ultrabooks from the type name column and that's the first filter. Now the second filter is from Apple or HP, and actually we already built this filter or condition. So this one is here. Here it says uh company and is in Apple or HP. So here we copy this one. And now I'm going to create a new filter. This is going to be filter 2 equal to and here the code that we created before. So now we have filter one and filter two. And what we have to do now is to link filter one with filter two with an operator, a logical operator. So we have to choose between the or operator and the and operator. So here in this example we should use the and operator because here we have to find notebooks or ultrabooks that are from Apple or HP. So here we have to satisfy both conditions. So we write here filter one and then filter two and we write the and operator in the middle. So we have filter one and filter two. And here I forgot to run this code first. So I run this code. And now I'm going to delete this one. And now I run. And we got this result here, this series. So now to filter the data frame first we write the name of the data frame and we open square brackets and now we run this and we get here a data frame that is supposed to have the values of Apple and HP in the company column and also the values of notebook and ultrabook in the type name column and you can verify this by using the value counts method as we did before and that's it. In this video, we learned how the is in method works in pandas.

All right. In this video, we're going to learn how to find duplicate rows with the duplicated method. So, here we start by importing pandas and reading the CSV file laptop price. So, here we have the data frame and the column that we're going to use for our first example is this laptop ID. So as you might know ids are unique. So we shouldn't have any duplicate ID in this data frame. So here let's see if this column has duplicated values. So we're going to use the duplicated method and first we're going to find duplicates in one column or series and the series is going to be this laptop ID. So here first I copy the name of the data frame which is df_laptops and then we write the method that is duplicated. So I write duplicated open parenthesis and now we only have to write the column. So here I copy and paste the name of the column and that's it. We only need this. So now we run this code and we got here a series with true or false values. So here we can filter the data frame based on this condition. So here let's do this. I write here first the name of the data frame which is df_laptops then open square brackets and as we did in previous videos we have to introduce this condition or this series inside this square brackets. So here we paste it and now we have this filter. So with this code, we're going to see if there is a duplicate element or duplicate value inside the laptop id column. So now let's find out. I run this one and now we got this data frame and we only see the names of the columns and there is

no data. So this means that this laptop id column doesn't have any duplicated elements, because if there was any duplicated element, we will see it here in this data frame, but there isn't any duplicated element. Great.

Now let's find duplicates in two or more columns. So here we only found duplicates in a single column, but now let's find duplicates in two or more columns. So now to do this, we're going to find duplicates inside the columns product, type, name, and inches. So to do that, we write the name of the data frame. So df_laptops, then we write duplicated, then we open parenthesis, and inside we write the columns that we want to analyze. So here are these three columns: product, type, name, and inches. So I'm going to copy this and now I paste it here. And now I'm going to write the quote. So here, comma, quotes, and now comma here, quotes, sorry, now comma, and that's it. So now I have these three columns. And here what we're doing is to find rows that have duplicated elements inside these three columns at the same time.

So now let's run to explain you this much better. So I run here, and we got this series with true and false values. So now let's make the filter. So we write the name of the data frame. So here is DF_laptops. Now we open square brackets and we introduce this uh series or condition or filter, call it as you want. So now I paste it here. Now we press Ctrl+Enter to run this one, and we got the data frame with the filter. And as you can see here, we got some duplicated elements. So here, for example, we got this uh first row that has this uh in product has MacBook Pro, then ultrabook in type name, and 13.3 in inches. And this one is exactly the same to this one. It has the same values.

So now to see this much much better, I'm going to store this data frame. But first, I'm going to put this inside a variable so you can see it much better. So here I'm going to create a variable name duplicated equal to this. I run this. Now I copy the new variable. And now I'm going to sort this data frame. So you can see much better the duplicated elements. So now I'm using the sort_values method that we learned in this course. And now I open square brackets. And here I'm going to specify the columns that I want to sort by. So first I want to sort by the product column and then by the type name column. So here I write type name and product. Now I run this one. And here we got this data frame with duplicated values, but now sorted by product and type name. So now we can clearly see the duplicated values. For example, this one that starts with 15-AC, the notebook 15.6, and is exactly the same as the second one here. So, both have the same values. And the same goes for this uh this one here with this one, the same values, and also this one has the same values. And that's how you find duplicates in two or more columns.

Here, one little detail I almost forget is this uh keep argument that is set to first by default. So here this uh duplicated method has this keep argument. So by default, the duplicated method will only keep the first duplicated element that is listed in a column. And we're going to see this keep argument in more detail in the following example.

Okay. In this second example, our task is to find the cheapest and most expensive laptop of each company using the sort values method and duplicated method. And here I'm going to explain you much much better how the keep argument works. So we're going to start by sorting the data frame. So here we write the name of the data frame df_laptops. Then we use the sort_values method. So we write sort_values, parentheses, square brackets, and then the columns we want to sort by. So company, the first one, and the second one is in this case the price. So the name of the column is price euros. Here I'm sorting by the price of the laptop because we want to find the cheapest and most expensive laptop. And this is going to make more sense when we add the keep argument in the duplicated method later.

So now I'm going to override the data frame df_laptops here. So I write df_laptops equal to this data frame sorted by company and price. So now I'm going to show you the new data frame. So here is the data frame, and now it's sorted by company and also by the price. So the cheapest laptop is first, and the most expensive laptop is at the end. Keep that in mind because I'm going to use that concept to explain you much better how the keep argument works in the duplicate method later.

So now let's continue here. And now let's check all the values in the company column. So here I'm going to use the value_counts method. So I'm going to use, I'm going to write uh df_laptops and now I write value_counts. Now parenthesis, and here I write the name of the column I want to evaluate. So company. So now I run this, and here we can see the elements inside this column, and we can see that all the companies have at least two duplicated values. For example, here Dell has 297. And if we scroll down, we can see that even Huawei has two duplicated values. So we can say that all the values inside the company column are duplicated.

Okay. Now let's continue this. And now let's find the duplicated values inside this company column. So now we write the name of the data frame df_laptops, then duplicated. Now I open parenthesis, and now I indicate the column I want to evaluate. So in this case, company, so now I run this, and we get this uh series, and as you might remember, this duplicated method has by default the keep argument which is set to first. This means that only the first duplicated value is going to remained while the orders will be removed. So here, just to remind you that I'm going to add the keep argument, and I'm going to set it to first. I don't need to do that because the keep argument has this first value by default, but I'm just writing it so you don't forget about it. So now I'm going to set this to a new variable which I'm going to name duplicated_first. So this is my new variable. Now I run this, and now I have all of this filter inside that duplicated_first variable.

So now what I'm going to do is to filter the data frame based on this variable. So I write the name of the data frame df_laptops. Now open square brackets. And now I paste the value of this variable. Now I'm going to run this one to see the data frame filter. So now I got here the result. And here I get the data frame with duplicated values in the company column. So now I want to show the data frame with non-duplicate values in the company column. And to do that, we have to use the not operator. So this one. And what this does is to get the opposite result of this filter. So here this variable give us that duplicated elements. But if we use that not operator, we're going to get that non-duplicated elements in this company column. So now let's check this out. So I'm going to run this one. And as you can see here, we got in the company column uh different values. So here first says Acer, then Apple, then Asus, and as you can see, none of these values are repeated. So none of them are duplicated. All of them are different.

So now to check this much better, I'm going to select two columns. The first one is going to be company, and the second one is going to be the price. So here I choose the price because our task is to find the cheapest and most expensive laptop. So here I select these two columns, and now we have this uh more compact data frame. And here as you can see here, all the uh the elements inside the company column are different. And also these are the lowest laptop prices per company. So all of these laptops are the cheapest in each company. And we didn't get this by accident. We got this because we used the source values method. And also we set this first value to the keep argument inside the duplicated method.

Here I'm going to scroll up a little bit to explain you much better what we did here. When we sort it by the prices and by the companies. What we did here is to show the cheapest laptops first. So the cheapest were first, and the most expensive were in the last position. So here then when we set here keep equal to first, it means that only the first duplicate value is going to remain in the data frame, that is only the laptop with the cheapest price will remain in the data frame, and that's why all of these values inside this data frame represent laptops with the cheapest price per company. So for example, here the cheapest Apple laptop costs €898, and that's it.

Now we can even check if this uh data frame actually have duplicated values or not. So here we only have to use the value_counts method. This is optional. You don't have to do this. But I'm I'm going to show you that this one doesn't have any duplicated values. So I run this one. And now, as you can see, all of the elements inside the company column only have one value. So here, Acer has one, Lenovo has one, and so on. So there isn't any duplicated values here.

Okay, that's how the keep argument works when it's set to first. Now let's check how it works when keep is set to last. So here, I'm going to copy the same code we use here because it's actually the same. We only have to modify the argument keep. So here is the same code. Now I'm going to change it here. Instead of first, I'm going to write last. Now I'm going to set this equal to a variable that I'm going to name duplicated_last. So this is my new variable. Now I run this. And now let's show a data frame with non-duplicated values in the company column. And in this case, since we're using keep equal to last, we're supposed to get the most expensive laptop per company because first we sorted the data frame ascending. So first were the laptops with the cheapest value or cheapest price, and at the end we got the laptops with the most expensive price. And if we say here keep equal to last, this means that we're going to get only the last duplicated value. So now let's check if this is true. Now I'm going to filter this by writing df_laptops. So here, so now let's verify if this is true. So I write here df_laptops. Now I open brackets, and here I write this duplicated_last to make the filter. And now I have to write this not operator. So here I'm saying non-duplicate values. So here I'm going to get non-duplicate values. So I press Ctrl+Enter. And now we got this data frame. And now let's select two columns to make this uh data frame smaller. So first the company column and also the price column. So it's price euros. So I write it here. Now I run this, and now we got this data frame, and all the values inside the company column are unique. So there isn't any duplicated values. And also here we got the most expensive laptop per company. So, for example, the most expensive Apple laptop costs €2,858. And we could get this data frame by setting the keep argument equal to last and also by using the sort_values method.

All right. Now, let's see how the keep argument works when it's set to false. So, now I'm going to copy this code and I'm going to set it here. So instead of last here, I write false, and I'm going to change the name of the variable. So in this case, it's going to be duplicated_false. Now I run this code and I scroll down here and I'm going to make the filter here. So I write df_laptops, then open square brackets, and inside I write the name of the variable that we created here. So it's duplicated_false. So now I write this, and now to get that non-duplicated values, we use that not operator. So we write here the not operator, which is this one. And now we press Ctrl+Enter. And as you can see here, this data frame is empty. And this happens because we set the keep argument equal to false. And this means that we're not going to keep neither the first duplicated value nor the last one. And as you might remember, the company column only has duplicated values. So here I'm going to uh write the code so you can remember in case you forget about this. So I write df_laptops, then value_counts, and now I write the name of the column company and press Ctrl+Enter. So as you can see here, all the values inside the company column are repeated at least twice. So for example, Huawei is repeated at least twice. So all of them at least have one duplicated value. This is why we got this empty data frame because all the values inside the company column are duplicated. And that's it. In this video, we learn how to find duplicate rows with a duplicated method.

Okay. In this video, we're going to learn how to drop duplicate elements with the drop_duplicates method. So here I have already pandas imported and also the CSV file that we've been working with so far. And here I have the data frame. Now let's see how the drop_duplicates method works.

Okay, the drop_duplicates method helps us remove duplicated elements in one or more columns. So this is similar to the not operator we seen before when we use the not operator to get that non-duplicated values, and this can also be accomplished with the drop_duplicates method. So now let's have a look, and first to use this method, we have to write the name of the data frame as usual, and here I write df_laptops and now that drop_duplicates, now I write parentheses, and now I open square brackets, and inside I write the name of the column I want to evaluate. So here the name is going to be company. So here we're going to remove duplicated elements inside the company column. So I run this, and now we shouldn't get any duplicated element in the company column. So to verify this, I'm going to select only the company and price euros column. So here I run, and now we can easily see all the elements. So we can see uh that in the company column we don't get any repeated value. So Apple, HP, Acer, and so on. Well, none of them are repeated, and actually we can verify this with a value_count method. So here value_counts, and here inside parenthesis we write company. So we run this, and all of them only have one element. So no duplicated elements. So now I'm going to uh get back the data frame here. And now our task is going to be to find the cheapest and most expensive laptops. So here in the drop_duplicates method, we also have the keep argument. So let's see how it works. So first we have to sort the data frame ascending by company and price. And as you might remember, this helps us put the cheapest laptops first and the most expensive laptops at the end. So here we write df_laptops, sort_values, and inside parenthesis we write the name of the columns we want to sort this data frame by. So we write company inside quotes and then we write the name of the uh the other column which is price euros and it's here. So we run this, and now this data frame is sorted. So let's override the values inside this data frame. So here I run, and now the df_laptops data frame is sorted. So now we can get the cheapest laptops by setting keep equal to first in the drop_duplicates method. So here we write df_laptops, then drop_duplicates, and now we write the name of the column which is company, and we add the keep argument. So here keep equal to first, actually we don't need to add this keep equal to first because this is set by default, but I add it here so you don't forget about it. So now let's run this one. So let's select only the columns we're interested in. So here I select the company and price euros. So I write square brackets and then paste this. So now I run. And here we can see all the laptops with the cheapest values per company. And now if we want to find the most expensive laptops per company, we only have to set this keep argument equal to last. So here I copy. And now I'm going to paste it here. So I paste it. And now I write keep equal to last. So now I run this one. And we see here all the most expensive laptops per company. And this is the same result that we got when we used the not operator with a duplicated values in the previous video.

All right. Now I want to show you two more arguments, and the first one is the in_place argument. I think you're quite familiar with this one. So this one saves all the changes we made to the original data frame. So when this is equal to true, we're saying that all the changes that we're making to this df_laptop data frame are going to be saved. However, when this is set to false, which is the value by default, we're saying just to return a copy. So now we want to save the changes we made. So I'm going to set this equal to true. And the other argument I want to show you is that ignore_index. And this one by default is equal to false, but we're going to set it equal to true. And this is going to ignore all the original indexes that the data frame had. So for example, here the original index of this row is 1,189. But if we set ignore_index equal to true, this is going to be zero. And the second one is going to be one, and then two, three, and so on. So now let's run this to try these two new arguments. But first, I'm going to delete this selection I'm making because we're not getting a copy because we set this in_place equal to true, but we're only saving the changes we're making. So, we're updating this data frame. So, here I delete this. And now I only have this. So, I run this. And now this data frame should be updated. So, now I I'm going to show it here. So I write df_laptops. I show it here. And now we have this data frame. And as you can see here, it starts with 0, 1, 2, and so on. And now I'm going to select only the columns we're interested in. So I write company and comma and here price euros. So now let's clearly see what's going on here. Here we got the last duplicated elements. And also we ignore the original indexes. And that's it. Now it's time to solve an exercise to put all these concepts into practice.

All right. In this exercise, our task is to find the biggest and smallest screen size in laptops for each company using the sort_values and duplicated methods. And here we have to use also the keep argument so we can get the smallest and biggest screen size.

Okay. Now you can pause the video and try solving these exercises on your own, and then you can continue watching the video to see my solution.

Okay. To solve this exercise, let's start by sorting the data frame. So I write df_laptops. Now sort_values, and here I'm going to write the name of the columns I want to sort by. So first is the company column and then is the inches column because we want to analyze the screen size. So here I sort by these two columns. And now I'm going to overwrite the values inside this original data frame. So now I run this. And now in our data frame, the small laptops should be placed first, and the biggest laptops should be placed at the end. So now let's choose the smallest laptops per company setting the keep argument equal to first. So here I write the name of the data frame, then drop_duplicates. So now I open parenthesis, and here I write the name of the column I want to analyze. So in this case is company. Now I add the name of the argument keep equal to first. And here I run this code. And in this data frame we got, we shouldn't have any duplicated element in the company column. And to clearly see the laptops with smallest screen sizes per company, we only select these two columns. So here I'm going to copy it. And now I open square brackets and paste this. So now I run this. But before we see the results, let's first read the CSV file. So we make sure that we have the original data frame here. So here I have the code that reads the

laptop price CSV. So, here I'm going to read it again. And we do this because we made some changes in the previous examples. So, we want to start from scratch, and here I read this CSV file. So, this is the original data frame. And now let's run this again. And also this one.

So, now let's see the smallest laptops. And here we got the smallest laptops. And in case we want to get the laptops with the biggest screens, we have to copy this and only modify the keep argument. So, here I'm going to paste it below. And now here I modify this. And now it's last. Now I run this. And here we got the laptops with the biggest screen. So, for example, the Apple laptop with the biggest screen is 15 in. And here above we got that the Apple laptop with the smallest screen is 11 in. And that's it. I hope you successfully solve this exercise.

All right, in this video we're going to learn how to get and count unique values using the unique and nunique methods. So, first we're going to start by importing pandas as pd and reading the laptop price CSV file. And now we have this data frame. And now we're going to count some unique values of some columns in this data frame. So, here I'm going to explain you first what the unique method does. So, this one returns unique values of series object, and the uniques are returned in ordering of appearance.

So, here let's get unique elements in the company column. So, to do that first we have to write the name of the data frame, in this case DF_laptops, then write square brackets, then the name of the column, in this case company, and this is the syntax of the series, and now we have to use the unique method. So, we write .unique, and that's it. It is as simple as that. So, now I run this one, and here we get an array, and this array contains unique elements inside the company column.

And now let's see another example, but now with the inches column. So, as you might remember, this inches column represents the screen size. So, now let's do this. So, we write again the name of the data frame followed by square brackets. Now the column is inches, and we write unique. We open parenthesis, and now we run this, and here we get an array, and the array has all the screen sizes. So, 13.3, 15.6, and so on.

Now we can get how many unique elements are in a column. We only have to use the len function. So, here I write len. Now I open parenthesis, and inside I write this. So, this is the code we created before. So, now if we run this we can get the length of unique elements inside the inches column. So, here we have 18 unique values inside the inches column. And now if we write company instead of inches, we're going to get how many unique elements are in the company column. And here we have 19. So, there are 19 unique values inside the company column. And that's it for the unique method.

Now let's check how the nunique method works. And this one returns the number of unique elements in an object, excluding null values by default. So, this is basically the same as the len function. But here we have a method in pandas. So, let's use this nunique method. So, we write first the name of the data frame followed by a square brackets and now the name of the column. So, company first, we're going to know how many unique elements are in the company column. So, first we have here the company series, and now we write nunique. Now we add parenthesis, and we run this, and we got 19. So, there are 19 unique elements in the company column, and this result is the same as the one we got here with the len function.

Now let's get the number of unique elements in the inches column. So, we copy this code and I paste it here. And now I just have to write here inches. So, now I run this, and here we got 18 unique elements in the inches column. And it's the same we got before using the len function. And that's it.

In this video, we'll learn how to get and count unique values using the unique and nunique methods.

Welcome back. In this video, we're going to learn the difference of selecting data with the methods loc and iloc. In pandas, we can select data by using the methods loc and iloc. Both have some similarities but also have some differences. And in both methods, to get the data we want, we only need to pass the indexes and the columns.

Okay, before showing you how these two methods work with code, we're going to learn some core concepts and also we're going to see some examples. So, first the loc method. So, the loc method is label based. So, we have to specify rows and columns based on their index and column labels. So, this means that we have to write the name of the index or the name of the column in order to get the data we want with the loc method. So, our first argument will be the name of the index, and our second argument will be the name of the column.

And now let's check the iloc method. And this method is integer position-based. So, we have to specify rows and columns by their integer position values. And this one is a zero-based integer position, which means that it always starts with zero. So, the main difference between the loc method and the iloc method is that with the loc method we only need the column and index labels. But with the iloc method we need to specify the integer position values. So, we have to locate a data inside a data frame and then see in which position is the row or the index and in which position is the column.

Okay. Now to understand this much better, let's see an example. So, here we have this data frame that I named df, and this data frame has three columns, fu, bar, and bas, and also has six indexes, and the first index is zero and the last one is five. So, here this 0, 1, 2, 3, 4, and 5 that is on the left, those are indexes, and those are the index labels or the index names, but here on the right we can see the index position of each of them. So, the first element has a zero index position, and the last element has a five index position. Here I wrote the name of the index using letters and not numbers in purpose because I wanted to show the difference between the index label and the index position. The index label is the actual name you see in the index. It can be words or numbers. But the index position can be only numbers and always start with zero.

Okay. Now let's see some differences between the loc and the iloc methods. So, in the first one you can introduce the label or the name of the index or the column. But in the second one you only can introduce the position of the index or the position of the column. So, here, for example, if you want to select an element with a value, you will see some differences between these two methods. So, with the loc method you have to write the name of the data frame df followed by loc with square brackets, and inside this square brackets you have to write the name of this index, in this case I want to get the value of the index zero, so this one. So, that's why I specify that index label. So, here I wrote df.loc and then zero inside square brackets. But in the second method, so the iloc method, we have to write df.iloc and inside square brackets we write the position of this index. So, in this case the position is position zero. So, the number, so here we wrote the number.

Okay. Now let's continue. And to select elements with a list is similar. So, we write df.loc. We open square brackets, and inside we write the list. So, in this case the elements will be the index labels. So, in this case zero and two. And for the iloc method we write iloc and then square brackets. And inside square brackets we write the list. But now the list has the index position. So, zero and two in numbers.

Okay. Now to select elements with the slicing we have to write df.loc, and inside square brackets we have to write the syntax start:stop. So, in this case the start element is zero. Then we have the colon, and then the stop element is two. So, these two elements are my start and stop. And with the iloc method we write df.iloc and inside square brackets we write the position of the index. In this case the numbers zero and the number two. But here there is a big difference between these two methods. So, in the loc method the start and the stop element are included in the slicing. However, in the iloc method the start element is included. So, zero is included in this slicing, but the stop element is not included. So, is excluded from the slicing. This means that we're going to get only the data that belongs to the index zero and index one using the iloc method. However, we're going to get the data from index zero, one, and two if we use the loc method because the start and stop elements are included in the slicing. So, this is a big difference that you have to keep in mind when you use either the loc or the iloc method.

So, now to finish this video, let's see how the following lectures will work. So, as you've seen in this video, the loc and the iloc methods are quite similar. The big difference is between the name label that the loc method uses and the index position that the iloc method requires. In the loc method lecture, we're going to have an average lecture, which means that I'm going to teach you how this method works with examples, and we're going to do it together. However, when we get to the iloc method lecture, we're going to do something different. So, this lecture is going to be an exercise for you. So, the examples we're going to see in the loc method and the iloc method are going to be the same. So, you have kind of translate the code we use in the loc method to select elements with the iloc method. After you try to solve these exercises on your own, we're going to check my solution and compare the results. And that's it. In this video, we learned the differences between the loc method and the iloc method.

Okay. In this video, we're going to have a look at the data set we're going to work with in this section. So, first we're going to import pandas as pd as usual. So, I run this first line of code, and then we're going to read the CSV file that we'll use in this section. So, to do that we use the read_csv method. So, I write pd.read_csv and here I open parenthesis. And now I open this quotes. And now I'm going to write the name of this CSV file, which is players. And now I'm going to complete this with tab. So, here's the name players_20.csv. So, I'm going to press control enter. You see the values. But here I added this s by mistake. I'm going to delete it. And here I run again. And now everything is fine. So, here is the data frame. And now we can see that this data frame contains information about soccer players. And here we can see different information about the players like the age, the height, the weight, nationality, and so on.

Okay. Now I'm going to name this data frame, and I'm going to set this equal to df. So, I write df = pd.read_csv. So, I run this, and now I'm going to show the data frame, and also I'll show you the columns we'll use in the next videos. Okay, now we have the data frame, and the first thing we're going to do is to set short_name to the index. So, this short_name is a column that contains the names of these soccer players. And if we set this to the index, now instead of 0, 1, 2, and 3 in the index, we're going to get the names of the players here. So, we can identify the rows easily, and that's going to help us a lot in the following videos. So, here I'm going to copy the name of the data frame, but actually I don't need it because it's just df. And here to set a column as an index we have to use the set_index method. So, we write set_index, we open parenthesis. And now we write the name of the column inside quotes. So, here is the name of the column. So, I copy it. And now I'm going to paste it here. And if I run this we can see that the new index is the short_name column. So, here we can see the names of the players, and now we can easily identify each row because it's not a number but now a player.

Okay. Now this set_index method created a copy of this df data frame. And if we want to save the changes and update this data frame, we have to overwrite this data frame. So, we can write this df = df.set_index, or we can also use the inplace argument. So, I write inplace = True. And now if I run, we're going to see that all the changes were saved. So, we're supposed to get the data frame with the index short_name. So, let's have a look. I run this, and here we can see that this index is named short_name and contains the names of soccer players.

Finally, we're going to select some columns that we're going to use in the following videos. So, here, if I scroll down, you're going to see that the data frame has around 100 columns, and those are a lot of columns. So, we're going to select like seven columns. So, here we're going to use the columns that start with long_name until club. So, these seven columns. So, right now I'm going to copy this seven columns. And now to select these columns, we're going to use double square brackets. So, here double square brackets. And now I write the name of the columns, or actually I'm going to paste them. And I have it here. So, now I add quotes and also comma to separate each column, and after this I'm going to explain you what each column means so you can understand much better the data set we're working with. So, now I only have to add quotes here in the club, and now it's ready. So, now I select the columns in here, I run, and you can see that there are only seven columns. So, now let's have a quick look at each column. And here first we have the long_name, and this is the full name of the player. So, here this is the short_name, and this is the long_name. Then we have the age. Then DOB is the date of birth of each player. Then we have the height, and this one is in centimeters, and the weight is in kilograms. Then we have the nationality, and the club each player played for by the year 2020. And that's it. Now I'm going to just override the value of this data frame. So, we can only have these columns. So, here I overwrote the value, and now here we have the data frame.

Okay. Okay. Now your task is to check this data frame in detail so you can easily understand the following videos.

Okay. In this video we're going to learn how to select elements by index label with the loc method. And we're going to start by importing pandas and doing all these things we did in the previous video. So, here I have the code that we did in the previous video. And now I'm going to run these three cells. And now here we have the data frame we created in the previous video. So, we have the seven columns we selected, and now we're going to start by selecting with a single value. So, here we're going to follow this syntax of the loc method, which includes the square brackets and then the row label or index label and also the column label.

So, our first example is to get all data about the player Lionel Messi. So, here to do this we're going to write the name of the data frame, which is df, then loc, then open square brackets as it says here. And now we write the name of the player. So, in this case it's Lionel Messi. And actually I'm going to copy the name because it's written in a different way. So, I make sure everything is correct. So, here I write or I paste Lionel Messi or L. Messi, and here I run, and we get all the data about this player. So, we have his full name, the age, and all this data.

So, now let's get a particular value of this data that we got. So, for example, we can get the height of this player by writing df.loc and now we have to write the index name. So, in this case is the name of the player. So, I write L. Messi, and then we have to write the column label. So, we want the height of this player, and the height is inside this height_cm column. So, here I paste the name of this column, and it's here. So, we follow the syntax, and now we have this. So, we run this, and as you can see, now we got the value of 170, and that's the height of this player in centimeters.

So, now let's continue with the next example, and in this case we have to get the weight of the player Cristiano Ronaldo. So, now I'm going to copy this and I'm going to paste it to have it as a reference. So, now here in the index we have to write the name of the player. And by the way, here we set the name of the column short_name to this data in purpose so we can use this data as the index. Ignore the numbers 0, 1, and so on. So, here this short_name helps us easily find this data with the loc method. So, now in this case we want the name of the index, and in this case is Cristiano Ronaldo. And now I'm going to copy and I'm going to paste it here. So, here the index name or the row name is Cristiano Ronaldo. And now we want the weight in kilograms. So, here I write weight_kg, and now it's ready. So, now I run this one, and we can see that the weight of this player is 83 kilograms.

Okay. Now let's get all rows inside the height column. So, so far we get here a specific value of this height, and also the weight. But now we want all the rows about this column. So, we can do that by using a special symbol or a special sign. So, this sign is this colon, and this allows us to select all the elements inside a row or inside a column. So, now let's see how it works. First we write the name of the data frame followed by that loc. And now here we want the height in the column. So, here we write a comma, and we write the name of this column. So, here height_cm, and then here in the index we want all the rows. So, in this case we write the colon, and this indicates that we're going to get all the elements, and all the elements in the index this time. So, here we run this, and now as you can see, we got this series, and this series is about the height of the players. So, here we have the information about the height of all the players listed here. So, for example, Lionel Messi is 170 cm, then Neymar is 175, and so on.

And now we can use this symbol in the column labels. So, I'm going to show you here. And we have to get all the columns that correspond to the index Lionel Messi. So, here I write df.loc, open square brackets. And here I write the name of the index, which is the name of the player L. Messi. And now we have to write the name of that column. So, in this case we want all the columns. So, we need the colon sign, and I write it here, and now it's ready. We run, and we get all the columns that correspond to the index Lionel Messi. So, here we have the data, and this one looks similar to the one we got here that we obtained by writing only the name of Lionel Messi, and yeah, it's the same, but in this case we use this colon that helps us get all the columns of this index.

Okay, now let's see how to select elements with a list of values. In the previous section we selected elements with single values, but we can also use lists. So, in this case we want to get all the data about Lionel Messi and Cristiano Ronaldo. So, here we have two elements. So, we can introduce these two

elements inside a list. So now we create a list with square brackets. Now we open here the quotes and we introduce the first element. So in this case, I do messy. And now I write this blank space. And now we write the second element, which is Cristiano Ronaldo. So now I write here, and now we have to put this inside the lock method. So here I write df.lo, open the square brackets that belongs to this lock method. Now we introduce this list. So here I copy and paste it here. And now with this, we're going to get all these elements inside this list. So I run this, and now we can see this data frame. So now we have the data about Messi and Ronaldo and all these columns here. So that's how you select uh elements with a list of values.

And now let's get the height of Messi and Ronaldo. So in this case, we can create the list and also add a column. So here we got the seven columns, and we can get or specify one column. So I'm going to show you here, and first I copy this code we created here, and now here is the name of the index. So now we only have to write the name of the column we want. So here we have to write the name of the height column. So here is height cm. So I paste it here. And now instead of getting this data frame with all these columns, we're going to get only one column. So here I run this. And now we only have the height of these two players.

And now let's get the height and the weight of Leonel Messi. So here we have to create a list for the column names. So in this case, we should have a list with two column names. So I going to create a list. So I open square brackets. And now I write the name of the height column, and it's here height cm. So I open this quotes, and then the second element is weight kg. So here kg, and now this is my list. So now I write df.log and put this inside this log method. But first, I'm going to write the name of the index, which in this case is also the name of the player. And here is l do messy. So I write it here. Now open quotes. And now I add this blank space. And now I can add this list that corresponds to the column names. So it's here. And now if I run this, we get the height and the weight of this player.

All right. Now let's get the height and weight of Messi and Ronaldo. And this is going to be like a mix of the previous examples. So here I write df.lo lock square brackets, and inside I write first the list of the index names. So in this case, the index are Messi and Ronaldo. So here I have it here from the previous example, and this list has Messi and Ronaldo. So I copy this list and I introduce it here as the index names. So here we have our first list, and then we have to write the column names, and here it's the height and the weight, and here I have it inside a list. So it's height and weight. So you now copy and paste it, and now we have these two uh elements, and these are inside the log method. So I run this one, and now we can see that we have this little data frame with the height and weight of Messi and Ronaldo.

Okay. Now we're going to learn how to select a range of data with a slice. And here we're going to use this syntax that it's here, which is start colon stop colon a step. And as a side note, this is contrary to usual Python slices. So in Python slices, this stop element is excluded, but here in pandas, both the start and stop are included. So this is kind of an exception. So here you just keep in mind that this start and the stop are included. So that said, now let's continue with the first example. Here we have to make a slice in the column labels. So here I'm going to show you the data frame. So you get a better idea of what we're going to do here. And I'm going to use this little data frame. So here we're going to make a slice between the columns H and the column club. So I'm going to struct this 2 4 6 columns. So I can do that by slicing these columns. And I'm going to show you how to do it here. So here I can use this syntax. So first we write the start element or the start column. In this case, this is going to be the H. Then we add the column, and then we add the column that is the club. So here I write club. And now we have this syntax start column stop. So here h column clap. And now we have to put this inside the log method. So here I write df.log square brackets. And first we introduce the name of the index that we want to select. And this time I have this one created and is a list that has messenger Ronaldo as elements. And the name of the list is players. So here I write the name of this list. So players. And now we have to write the name of the columns. So here I copy or cut this slicing. And now I paste this slice. So we have this slice from H to club. So now we can run this. And now as you can see, we have the slice between H to club. So only the columns between these two columns were selected. And as you can see, the clap column is included in this slice. So that's how you make a slice with column labels.

And we can also make a slice with index labels. And I'm going to show how to do it here. Okay. In this example, we want to slice index labels that are between the top one and the top 10 player. So this data set sorts the players based on their scores in the FIFA game. So the first player or the first row that is in this data set is the best player according to this FIFA game, and the 10th player in this data frame is the 10th best player in this game. So here we can get the slice that represents the top 10 players by selecting from index zero to index 9. But first, we have to get which player is in index zero and which player is in index 9. So here we have to use the index attribute. So I write df.index. And now to get the first 10 elements, we write here the square brackets. And now uh this column and then 10. So this represents the 10 first elements in this index. So here if I run, you can see all the indexes, and if I write now the square brackets, we're going to get only the first 10. So here we have the first 10, and we have that the best player in this game is Leonel Messi, and the 10th best player is Muhammad Salah, which is here. So now we can make a slice by writing both Leonel Messi and Muhammad Salah. So the start element will be Messi here, and then the stop element will be M. Salah. So I'm going to write it here. So first is going to be the first element or the first index, then colon, and then the 10th element. So I write it here hammet sala. And now what we have to do is put this inside a log method. So here I write df.log then square brackets. And now we cut this slice and now I paste here. So now this is the index or the name of the index, and then we have to write the name of the columns, and I created here a list that is here, and the variable is named columns. So I copy this one and now I paste here. So now if I run this, you can see that we get the first 10 players in this data frame. So from Leonel Messi, which is index zero, to Muhammad Salah, which is index 9. And that's how you select a range of data with a slice in the columns or in the index.

Finally, let's see how to select elements based on a condition. So in the previous videos, we learned how to make conditions in pandas, and now let's make conditions and also select elements from these conditions. So here we have this condition that says select players with height above 180 cm. So to create that condition, we have to write here the name of the data frame. So I write df, which is the name of the data frame, and then open square brackets, and we write the name of the column we want to evaluate. So in this case, is the height column. So I copy this one and paste it inside square brackets. Now we have to compare this with the 180 cm, and we only write the numbers here to compare it. So here greater than 180. So here we have this condition, and we have to put that condition inside the log method. So here I cut this and put it here in the first argument. So here is supposed to filter now our index. And now we have to write the name of the columns we want to select. So in this case, the name of the columns are here inside my columns variable. So I copy and paste it here. So now we're selecting elements that satisfy this condition in the index and that belong to these columns inside this columns variable. So I run this, and we can see here in this data frame only players above 180 cm. So, for example, Leon Messi is not here because he is 170 cm. So he's not here.

Okay. Now let's select elements based on multiple conditions. So here our task is to select players with height above 180 cm and from Argentina. So here is condition one, the height, and the second condition is the nationality. So here we write both conditions, and the first one we actually build it here. So here is my first condition, the height above 180. So I paste it here. So first condition is done. And now we have to write the second condition. So nationality equal to Argentina. So here I copy this one, and now I write the name of that column, which is nationality, but I want to see how it's written. So it's here nationality. So I paste here, and now this has to be equal to the country Argentina. So I write here, and here we have condition one and condition two. So to satisfy both conditions together, write the parenthesis here to separate them first, and then to satisfy these two conditions together, we have to write the and operator. So the and operator is represented by this symbol. And here we have condition one and condition two. So with this, we're going to satisfy both conditions. And now we only have to insert this multiple condition inside this log method. So I cut this one. Now I paste it here. And this is going to help us select elements based on this condition. So now I only have to write the name of the columns. And here instead of writing the name of the columns one by one, I'm going to write only the column symbol, which indicates that I want all the elements. So here I wrote this column sign, and now I press Ctrl enter to execute this code. And here we have players with height above 180 cm from Argentina. So, as you can see here, we don't have Cristiano Ronaldo anymore because he's from Portugal, but we have other players from Argentina with height above 180 cm. And that's it. In this video, we learned different ways to select elements with the log method.

Okay, in this video, we're going to learn how to select elements by index position with the iloc method. So as usual, I imported pandas and run these three first cells to get this data frame that has these seven columns and this short name column in the index. So it looks like the one we created in the first video. And now we're going to uh select elements with a single value. But in this case, we're going to use the iloc method. So all these exercises are the same as the examples we've seen with the lock method. So I recommend you to resolve these exercises on your own, and you can pause this video, and after you try it yourself, you can continue watching the video to see my solution. So you can download this Jupyter notebook file that is in the notes of this video to start solving these exercises.

Okay, let's start with the first task. And here we have to get the height of Leonel Messi. So we have to write df do iiloc and then square brackets, and inside instead of writing the names of the index or the name of the column, we have to write the position. So here what we have to do is to write the position of Leonel Messi, and this one is position zero in the index because here, as you might remember, Leonel Messi is first here in the first row. So first row position zero. So here zero and then the height is the position of the column here. So this is zero, this is one, this is two, and height is position number three. So we write three, and that's how we do it. So now we run this code, and we get the height of Leonol Messi.

Then we have to get the weight of Cristiano Ronaldo. And I'm going to copy this one so we can do this faster. So here first the index of Cristiano Ronaldo here is one. And now the column of the weight I think is here. So it height is number three. So weight is number four. So here instead of three, we write four. And now we run this, and we get that the weight of Ronaldo is 83 kilogram.

Then we have to get all the rows inside the height column. And in this case, I'm going to use the column symbol, and is this one. So this represents all the elements either in the rows or in the column. So here we write df and then iiloc then square brackets, and since we want all the rows, we have to introduce this one first as the first argument, and we get all the rows. Then here we have to write the position of this height column. So here the position is three because we did before in this exercise. So is three, and we paste it here. So we paste it, and we run now, and we get all the rows inside the hide column.

Next, we have to get all the columns that correspond to the index Leonel Messi. So we write df iloc, and then we write the index that corresponds to messy, which is index zero, and then we write uh the colon symbol because we want to get all the columns here. So if we write the colon symbol, we're going to get all the elements that are inside the columns. So here we press Ctrl enter, and now we get all the elements or all the columns that correspond to the index Leonel Messi.

Okay. Now let's select elements with a list of values using the iloc method. So the first task here is to get all data about Messi and Ronaldo. So we use df.iloc. And now we open square brackets. And since we want to get the data of two players at the same time, we have to create here a list. So here is my list. And now I write the index of Messi, which is zero. And then the index of Ronaldo. So one. Then we have to put this list inside this uh square brackets. So here we paste it. And now we press Ctrl enter. And here we have all the data that corresponds to these two players.

Next, we have to get the height of Messi and Ronaldo. So we write df.iloc. And here we have to write the indexes that corresponds to Messi and Ronaldo. And here we have it from the previous exercise. So I paste it here. And now we have to write the position of the column height. So in this case, is position uh 0 1 2 and three. So we write three. Now we press Ctrl enter, and we get the height of Messi and Ronaldo.

Next, we have to get the height and weight of only Lionel Messi. So here again df.iloc, and then here we write the index that corresponds to Messi, which is zero. And here the positioning of height and weight. So height is three and weight is four. Now we press Ctrl enter, and we get the height and weight of Messi.

And finally, we have to get the height and weight of both Messi and Ronaldo. So it's like a mix of the previous exercises. So let's do it here. df.iloc then square brackets. And we have to introduce first the list that corresponds to Matthew and Ronaldo. So here I have this one. So I paste it here. Now the columns that represent the position of height and weight, and it's this one. So we copy and now we paste it here. Now Ctrl enter, and we have this little data frame that shows us the height and weight of both Messi and Ronaldo.

All right. Now let's select a range of data with a slice. In here, you have to remember that we have to use this syntax start colon stop, and here the start is included, but the stop is excluded. So keep that in mind. So here we have to get the slice of the columns that are between age and club. And here we have the index we have to use. So first we translate this players variable. So instead of writing the name of the players, we write the index or the position. So here is zero and one, and then we have to write the here the df.iloc as usual, and now we have to write square brackets, and inside we paste the index of the players. So here players, then we have to write the slice that we want to make. So now let's see which position corresponds to the age in the club column. So here I'm going to show this data frame, and here we have h. So this is zero, and this is one, and here club is 2 3 4 5 6. So h is one, and club is six. So now we write here I'm going to delete this one because we don't need it anymore. And here is one and six. But as you might remember, this last element, this top element is excluded. So we have to write this one + one. So if we write this, we get seven, and the seventh is going to be excluded. So we get the number six. So that's why we do this. So 6 + 1 is 7. So we get this 1 column 7, which represents a slice from the H to the club column. So now it's ready. So we run this one, and we get this slice from H to club, and we get the index of Messi and Ronaldo.

Okay. Now let's get a slice of the first indexes. So here first we start translating these columns variable. So instead of writing the names, we write the position. So one 2 3 and four. So this one, as you might remember, H is position one. And here the column dab is next to h and also hide and also wait. So 1 2 3 and four. So it's here. And now we have to write the syntax. So df.iloc then square brackets. And now what we have to do here is to uh write the slice that we want. So in this case, we want the first 10 indexes. So we do this because we want to get the top 10 players. So if we want the first 10 elements or the first 10 indexes, we have to write here zero, which is the start element, then colon, and then the stop element, which is in this case 10 because this is excluded, and we're going to get the number nine. So we get from zero to nine. So the top 10 players. So this is my slice, and now I'm going to write the name of the column that we want to get, and this one is inside the columns variable. So I write columns, and now I run this one, and we get this slice from 0 to 9, and is here. So we have uh the top one player, and we have the top 10 player and all the data that is inside these columns that we specified in this columns variable.

Okay. Okay. Now let's select elements with conditions. And here a quick note because the iloc method cannot accept boolean series but only a boolean list. We have to use the list function to convert a series into a boolean list. So we're going to see this in this first example. And here our task is to select players with height above 180 cm. So here first I'm going to translate this uh columns variable, and this one is the same as the one we saw before. So 1 2 3 and four. H is one, and then do is next to h, so it's two, and then three and four. So here we have to write the condition first. So df square brackets, then the name of the column, which is

Uh, the column we want to evaluate here is height. So, height_cmters. Now, we compare this one with 180, and this is my condition. But, as I mentioned before, this is a boolean series, and we have to convert this into a boolean list. So, here we write list and parenthesis, and we use this list function. So, now we write the df.iloc, then we open square brackets, and now what we're going to do is to insert this uh list inside the first argument here. So, it's there, and now we have to specify the columns we want to get. So, I write columns. Now, I press Ctrl+Enter. And here we got this data frame that has only players with height above 180 cm.

Okay. Now, this final task: we have to select players with height above 160 cm. And these players are from Argentina. I think I didn't write it here, but anyway, it's from Argentina. So, this is condition number two. And now condition number one, which is height above 180 cm, we have it here. So, I copy this one. And now I paste it. And now condition number two is the nationality equal to Argentina. So, I write nationality. Then, here instead of greater than, I write equal to. Now, uh we compare this with Argentina. So, I write Argentina, and now we add this parenthesis. So, condition one with parenthesis, and now condition two with parenthesis, and in the middle we write the and operator. So, now we have to make this a list. So, I write list, open parenthesis. And now what we have to do is to write the df.iloc method as usual. Now, I open square brackets and I insert this list here as my first argument. So, here I wrote it, and now I want to get all the columns. So, I write the column sign, which is this one. Now, Ctrl+Enter, and here we got all the players with height above 180 cm, and these players are from Argentina, as you can see here. And that's it. I hope you successfully solve these exercises.

Okay. In this video, we're going to learn how to set a new value for a cell in a data frame, and we're going to start by importing the pandas library and also reading the CSV file and doing the changes we made in the previous videos. So, here I run these three first cells, and now we have this data frame with these seven columns we selected before and with the short_name as index. So, now I'm going to scroll down, and we're going to see the first example. So, let's see how to set a value to one cell. And now let's say we want to update the height of the player Leonel Messi. So, here I first have to locate the height of the player. So, I write df.loc. Now, open square brackets. And now I write the name of the player. In this case, this one, and then we have to write the name of the column. So, is this the name of the column? And now I write here. So, it's here now. And now we have to press Ctrl+Enter. And the current height is 170. And now let's say we want to update this to 175. So, we only have to write equal and then 175. And then we press Ctrl+Enter. And if we show this data frame here, we can see that the data frame has a new value inside the index L. Messi and height_cm column. So, here it says now 175, which indicates that this data was updated.

Okay. Now, let's see how to set a value to an entire column. So, let's say that we want to update the height of all the players to 190 cm. So, we can do that by using some things we've seen in the previous video. So, first we have to locate the whole column, and we have to use the loc method. So, here I write df.loc, open square brackets, and now to get all the rows we have to use the colon. So, here I write colon, and now I write comma, and this indicates that we're going to get all the rows. So, now I write the name of the column I want to get. In this case, it's going to be height because we have to set the height to 190. So, I copy this, and now I copy height. Now, paste it here. And now we write this equal to 190. So, first I'm going to show you what this is going to return. And here we can see that we get the rows of this height_cm column. And these are all the rows because we use the column here. So, now we set this equal to 190. And all this data is supposed to be 190. So, we run. And now I'm going to show this data frame. And now you can see that all the data inside the height_cmters, it's 190. So, here we updated all the data inside this column.

Okay. Now, let's see how to set a value to an entire row. And this is similar to setting a value to an entire column because we're going to use the colon sign. So, it follows the same idea. And here, our task is to get all columns that correspond to a player ranked last in FIFA. So, here I'm going to show you the data frame. And as I mentioned before, the player listed first in this data frame is the best player according to this FIFA game. And the last one is the worst player in this FIFA game. So, in this case, it's this one, Pan Sim. And we have to get this element, and we can get this element with the iloc method. So, we write iloc, and now we can write the name, we can write the name of that index. So, in this case, it should be this one, uh 18,000, and here I can copy this one. So, I can paste it now, but we can also use negative index. So, here instead of writing this uh number, we can write minus one, and minus one represents the last element in a list or in this case, in this uh indexes. So, now we have to write comma and write the name of the columns we want to get, but in this case, I want to get all the columns. So, we write the column symbol, which is this one, and we're going to get all the columns here. So, I press Ctrl+Enter, and this is all the data about this player. So, we're going to update this data, and now I want to update with null values. And to set null values, we have to use the numpy library. So, first I import numpy as np. And now to create null values, we have to write np.nan. This nan represents null values. So, here I run, and as you can see here, well, only says nan, but if now I set this equal to this data here that we got by selecting with iloc, we can update the values here with null values. So, I run this, and now let's see what happens. I'm going to show the data frame, and it's here. So, I scroll down until I find the last row, and it's this one. So, it says Pan Simmon, and we can see that it only has null values because we updated with this np.nan, and here with this iloc, we located that last row.

Okay. Now, let's see how to set value to multiple cells. So, we have to set a value for items matching the list of labels. So, in this case, let's say we want to update the height of two players. So, first, as you might remember, we have to write a list with these two players inside this list. And I'm going to do this here. So, I write square brackets. And now I write first the first element, which is L. Messi. I'm going to update the height of this first player. And the second player is going to be Cristiano Ronaldo. So, I write it here. And this is my list with these two players. So, now to update this uh the height of them, we have to use the loc method. Then, we insert this list here inside. And now what we have to do is to indicate the height column. So, I'm going to write it inside a square brackets to indicate this is a list. And here I opened um these quotes. And now I write the name of the column. So, here height_cm. So, I paste here. Now, I run this, and we get this little data frame. So, now I want to update the height to 175. So, these two values are going to be 175. So, I press Ctrl+Enter. Now, I show the data frame, and we're going to see that the height of Messi and Ronaldo is now 175, and the rest is 190 because of the changes we made before.

Finally, let's see how to set a value for rows matching a condition. So, here we want to change the height only for players with height above 180 cm. So, first we write this condition. So, we write df, and here the column we want to evaluate. So, in this case, the column is height_cm. So, we want this to be greater than 180. And now to change the values based on this condition, we have to write df.loc, and inside this we have to write this condition. So, this condition is going in the first argument, and here we have to add the name of the columns we want to obtain. So, here I have these columns inside this uh columns variable. So, I copy and paste it here. And now I'm going to press Ctrl+Enter to see what we get. So, is this uh data frame with only players that have height above 180, and now we can set this data equal to zero. So, all the data inside this data frame will be zero. So, I press Ctrl+Enter. And now I show the data frame. So, now we can see that all the data for almost all the players is zero because their height was above 180. But Leonel Messi and Cristiano Ronaldo are the exception of this rule because we change their height to 175. So, 175 is less than 180. So, this condition was false, and this value wasn't set to these two heights. And that's it. In this video, we learn how to set a new value for a cell in a data frame.

Okay. In this video, we're going to learn how to drop rows and columns from a data frame using the drop method. So, first we run these three uh cells here that transforms our data frame as we did in the previous videos. So, now we scroll down, and here I'm going to show you how to drop rows using the drop method. Okay, this drop method has different parameters, which means that we can use different ways to use this drop method, and here I'm going to show you uh one way, which is using the axis parameter. So, here I'm going to start by writing df.drop. So, this is the syntax of the drop method, and now I write the index or the row we want to get rid of. So, in this case, this is going to be uh let's see the first one, too, Leonel Messi, L. Messi. So, I want to delete all this row. So, I write the name of the index, and I write it here. So, this is my first argument, and I write it here. So, now I press comma, and now the drop method doesn't know if this first element is a row or a column. So, here we have to specify in the second argument, and here we use the axis parameter. So, we write axis then equal to, and here we have to write number zero because zero represents rows and one represents columns. So, if I write zero, it means that this first element, L. Messi, is a row or it's an index. So, we want to drop this index L. Messi. Okay. So, here now we press Ctrl+Enter to run this one, and we get here this data frame, but the row that had the index L. Messi is not here. So, we dropped it.

Okay. Now, the second option is using the index parameter. So, here we write df.drop. And now we use the index parameter. So, we write index equal to. Now, open this uh square brackets. And now we write the name of the element, in this case, L. Messi, which is this one. So, I copy and paste here. And that's it. So, we're telling this drop method to drop this L. Messi index. So, here now I'm going to comment this out, and I'm going to run this one. So, I press Ctrl+Enter. And as you can see, we get the same data frame. So, we get the same result that we got here with the df.drop but with the axis parameter. So, here we got the same result. So, you can choose any option you want. And now let's see the second example. And in this case, we have to drop two or more rows and update the data. And to update the data, we use the inplace uh parameter. So, here we write df.drop. Now, we open parenthesis, and here we write the name of the elements. In this case, I want to get rid of the first and second rows. So, in this case, uh the row L. Messi and Cristiano Ronaldo. So, I'm going to delete these two rows. So, I write first the square bracket that represents a list. And now here I write first the name L. Messi, and now I paste it here. And the second element is going to be Cristiano Ronaldo. So, I copy and paste it here. And by the way, we open a list here because we're working with two or more rows. So, we have two or more elements, and that's why we have to write this list. And now we have to specify that these are rows or indexes. So, I write axis equal to zero. And then to update this data, we have to write inplace equal to true. And here we're going to update the data because we're using this uh parameter. Before, we didn't update the data because here the drop method only creates a copy. So, to save all the changes we make, we have to use the inplace parameter. So, here now I run this one. And as you can see, we don't get anything, but if we now uh print this data frame, we can see that the rows Messi and Ronaldo are gone. So, the first row here is Neymar Junior.

Okay. Now, let's see how to drop columns with the drop method. And this is similar to dropping rows. So, first let's see how to do it with the axis parameter. So, we write df.drop. Now, parenthesis. And the column I want to delete is the long_name. So, I want to delete this column. So, here I paste it. And now I have to specify the axis. So, we tell the drop method that this is a column. So, here we write axis equal to one because one represents columns. And now we run this one. And as we can see here, this data frame starts with the H column. So, this means that the long_name column is not there. And now let's do this, but now using the columns parameter. So, here it's the columns parameter. And now I use df.drop, drop, and here open parenthesis, and now I write columns. So, this is the parameter, and now equal to, and here I write the name of the column. So, here I paste this column, and that's it. That's all we need because when we use the columns parameter, that drop method knows that this element long_name is a column. So, that's enough. And now I'm going to comment this out. And here I run. And as you can see, we don't have the long_name column.

Okay. Now, let's see how to drop columns by position. And in this case, we want to drop the last column. So, the first thing we have to do here is to locate the last column. And we can do it by using the columns attribute. So, here I write df. And now I write the columns attribute. And here I open square brackets twice. And if I want to get the last column, we only have to locate it here. So, this is not the original data frame. So, I have to write it here. df. So, now let's find in which position is this club column. So, this one is position zero, then one, two, three, four, five, and six. So, this club is in the position six. So, now I delete this. And here I write six. So, if I run this, we get this result, and it has the club element or the club column. So, it's fine, this approach. But what we can do now is to use negative numbers. So, if we write minus one, this means that we want the last element. And this is much better than using the six that we had before because sometimes the last element is not six but seven or five depending on how many columns we had before. But if we write minus one, we can make sure that the last column is going to be selected because minus one is always the last column. So, here I run this, and here you can see that we got the same result. So, it's working fine. And now to drop this uh column, we have to use the drop method. So, we write df.drop. And now I open parenthesis as usual. And now I insert this one here. And now I add the axis. So, we tell the drop method that this element is a column. So, here axis equal to one is column. So, now I press Ctrl+Enter, and as you can see, we don't have the club column anymore. So, we deleted the club column.

And now let's check the last example. And here we have to drop two or more columns and then update the data. So, this one is simple. We already did it but with rows. So, we only have to write df.drop. And now I open parenthesis. And since we want to drop two or more columns, we open square brackets that represent a list. And now here I write the two columns. And in this case, it's going to be the columns long_name and DOB. So, here I paste this one. And the second is DOB. So, date of birth. And now these are the two columns that we're going to delete. So, here we add axis equal to one, which indicates these are actually columns, and now we add the inplace parameter. So, we update the data, and we don't create a copy. So, here it's ready, and now we press Ctrl+Enter, and we didn't get anything. But if now I print the df data frame, we can see that the long_name column and the DOB column are not here. And that's it. In this video, we learn how to drop rows and columns using the drop method in pandas.

Okay. In this video, we're going to learn how to create random samples using the sample method. So, first I'm going to import pandas and run these three first cells. So, here I run these cells, and now we have the data frame ready. So, now I'm going to show you how the sample method works in pandas. So, the sample method helps us extract some random elements from a data frame and also series, and this can help us when we want to analyze a small portion of our data frame. So, here our first example is to extract 10 random elements from the nationality column. So, here in my data frame, I have this column name nationality, and here we have the countries where these players were born. So, what we're going to do is to get 10 random elements from that column. So, here we have to write first the name of the data frame, which is df, then square brackets, and here I'm going to copy and paste the nationality. So, here nationality, and now we have to write sample and then parenthesis, and inside we have to specify the number of elements we want to extract from this data frame or in this case, this series. So, in this case, I want to extract 10 random elements from this series. So, we write 10, and that's it. So, if I run now, you can see that I get 10 random elements from this data frame. So, here I get 10 uh soccer players, and also here the nationality. So, this is the index, and these are the values. And now if I run again, you will see that we're going to generate other 10 random elements. So, let's see. I run, and here we have another 10 elements. And now here we run again. And you can see that we generate 10 random elements every time we run this cell.

Okay. Okay, now we can control the random number generation here with a sample method by writing the operator name random_state. So, here I write random_state. And now we have to write any number we want. For example, I write the number 99. And now if I run this, we get new 10 random elements. But if I keep running this, we can see that these 10 random elements remain the same. So, they don't change no matter how many times I run this cell. So, this is how the random_state works. But if now I change the number here, for example, I write one.

Now, this is another random state, and we're going to get another 10 random elements. So, I'm going to leave it with 99. So, this is my random state, and that's how the sample method works.

So now, let's see the second example, and in this example, we have to extract a random 20% sample of the data frame. So, here I have the original data frame, and this one has 18,000 rows. So, if we want that 20%, we just have to divide the number of rows uh by five. So, here let me show you. We write df.sample, then we open parenthesis, and here we could write this number divided by five. So, this divided by five, and we get our sample.

But now we can do something different. We can add a new parameter to specify the portion or the fraction we want to get. So, let me show you here. We need to introduce the frac parameter. So, I write frac, and this stands for fraction. And now I write equal to, and here I write 0.2. So, 20%. Now I run this, and here we get uh the 20% portion of this original data frame. So, here this is 3,600 rows, and yeah, that's like uh this 18,000 divided by five, so the 20%.

Okay, now we can control here the random number generation. So, here we can add this uh, let me see here, random_state. So, this is the parameter that helps us control the randomness of the sample method. So, we write random_state, then equal to, and in this case, I'm going to write any number, let's say uh 12. And now I run, and now I get the 20% with the random state 12. And if I keep running here, we get the same uh 20% sample. So, it remains the same.

Finally, let's see how to increase the sampling rate, which is known as upsampling. So, when we want to increase the sampling rate, we have to specify the frac parameter greater than one. So, for example, if frac is equal to two, this means that we want to get a 200% of the original data frame. So, we want to double the number of rows. But here, keep in mind that when the parameter is greater than one, so for example, when we want to double the number of rows in a data frame, we have to set the replace parameter equal to true. So, that's like a rule. Every time we set the frac parameter greater than one, we have to set the replace parameter equal to true here. So, keep that in mind.

And now I'm going to do this here. So, I press DF.now that sample and now open parenthesis. So, now we want to double the number of rows in this data frame. And now we have to write frac equal to two, and here I'm going to add the replace parameter because here two is greater than one, and we have to follow this rule. So, two greater than one, and parameter has to be equal to true. So, now here what we have to do is to add this new parameter, replace equal to true. And that's it. So, now we run this one. And here I got an error because I didn't write this sample correctly. So, now I fix it. And now I run this one. And here we got this data frame. And this data frame should double the number of rows of the original data frame. So, let's check it out. Here there are 36,000 rows. And let's see how many rows the original data frame had. And here 18,000. So, yeah, it doubled the original number of rows. And now here again we can add a random state in case you want to control this. So, random_state, and here you can write any number. So, I'm going to write 99 in this case. Now I press Ctrl+Enter, and this data frame or this sampling that we're getting is going to remain the same. So, it's not going to change, as you can see here. And that's it. In this video, we learn how to create random samples with the sample method.

Okay. In this video, we're going to learn how to filter a data frame using the query method. So, we're going to start by running these three cells. And now I have this data frame that we created in the previous video. So, here is a data frame. And now I'm going to show you how the query method works. So, the query method helps us filter a data frame the same way we did in previous videos. But in previous videos, we use something called boolean slicing. But in this case, we're going to use the query method. And this is a different syntax. So, I'm going to show you here. And first, let's solve this task that says select players older than 34.

So, to solve this with a query method, we write df.query. Then we open parenthesis. And here we open these quotes, and inside we write this condition. So, first the name of the column, which is H. Let me see here. So, H, and we have to compare this with 34. And as you can see, this syntax is different from the syntax we've been using with the boolean slicing. But let's try it out. So, here let's run this one. And in the data frame that we got here, the Hs are greater than 34. So, here we have 37, 36, and yeah, so we successfully filter this data frame with a query method.

And now I'm going to show you how we will do it with the boolean slicing in case you don't remember how this works. So, with the boolean slicing, first we have to create a condition. So, here we write df, then open square brackets, and now we write the name of the column. So, here H, now we compare this with 34. So, we write greater than 34, and now we put this inside uh square brackets. So, here I write df[ ], and this condition is inside. So, here this is called boolean slicing, and this is going to return the same data frame we get here with a query method. So, now let's check it out here. I comment this out, and I run this one, and this is the same data frame we got before, but now with a boolean slicing. So, both have the same result, but different syntax, and you can choose the one you find most practical or the one that you find easy to remember.

Okay. Now, in this second example, we have to select players older than 34 from Italy. So, this is some kind of multiple condition, and we're going to do this with a query method. So, here I write df.query. Now parenthesis, we open this quote, and inside we write the two conditions. So, here we only write the name of the column. So, H greater than 34. And now we write the second condition. So, nationality here equal to Italy. So, here Italy should be inside quotes. But since we already use this double quotes here, we have to write single quotes to avoid any conflict between these two. So, here I used single quotes for Italy, and now it's fine. So, here we have condition number one, which is this one, and this one is condition number two. So, now we have to add an operator, and in this case, we have to add the and operator. So, we have to do this because we want to satisfy both conditions, and as you might remember, the and operator is represented by this symbol that I'm writing right now. So, we've been using this symbol, and in this case, with the query syntax, we don't use that symbol, but a different symbol. So, here instead of writing that symbol, we only write the word and. So, here we set this condition and this other condition. So, that's how it works with the query method. So, now it's ready. And now let's run this one. And here I got the data frame. And in this data frame, uh the age of the players are greater than 34, and all the players are from Italy, as you can see here. So, we successfully filter this data frame with this multiple condition.

Okay. Now, as an exercise, you have to write the equivalent boolean slicing of this query method. So, you have to do something like I did before here. This is the equivalent of this uh query method. So, you have to do something like this, and you can pause the video, and then to check my solution, you can download my script that is in the notes of this video.

Okay. Now, in this third example, we have to add a not operator to the first example. So, let me copy the code of the first example, which is right here. So, I copy this one. Now I paste it. And now we have to add a not operator. So, so far the not operator that we've used is this one. So, we've used this operator in this course so far. But now in this query method, we have to use a different operator. So, instead of writing this symbol, we have to add first parenthesis, and now here write the word not. So, this is the not operator, and this is being applied to the H greater than 34. So, if we set not this, we're going to get H less than 34. So, let's check it out. I run this one, and as you can see in this data frame, all the players have age less than 34. So, this is how we do it with a query method. And now you have to write the equivalent boolean slicing of this query method.

Okay. Now, in this task, we have to convert the height to meters and then select those with height above 1.8. So, we can do that with the query method. We only have to write df.query, now parenthesis, and inside quotes, we have to do this operation. We have to transform the cm to meters because the height now is in cm. So, to convert this to meters, we only have to write the name of the column here and then divide it by 100. So, here we have the height in centimeters, and if we divide it by 100, we get the height in meters. So, now what I'm going to do is to compare this with the height here 1.8. So, I write 1.8. And now this query is ready. So, we can run this one. And as you can see in the data frame, all the players have a height above 1.8.

All right. Now your task is to write the equivalent boolean slicing of this query we created here.

Okay. Now our last task is to select players that were born after 1990. So, to do that, first we have to check the data types of the columns. So, here I'm going to see the data types of all of these columns. So, here I write DF, and I'm going to use an attribute that we learned in this course, which is the dtypes. So, this one helps us get the data types of all the columns in this data frame. And now I'm going to see this one, but here above. So, I run this, and as we can see here, the date of birth, so DOB, has a data type object, and this isn't good because here the date of birth, so DOB, is a time, so it has the year, the month, and I think the day, and this column should be a datetime type, and we can convert the data type of this column by using a method that is called astype, and I'm going to show you here how it works. So, first we write df, then the name of this column, DOB, and now we write astype. So, this is a new method, we open parenthesis, and here inside we write the type we want to convert to. So, here I write datetime64, and this is a time type that is common in pandas. So, here this is my time type or datetime type, and now I'm going to override here this DOB column. So, it's here. So, now I'm updating the data type of this column. So, we do this because we want to extract the year of this column, and we can only do that if this column has a datetime type. So, that's why we're converting this column to a datetime type. And why we want to get the year because here in this task we have to select players that were born after 1990. So, if we compare the years, only the years, we can know which players were born after 1990.

Okay. And once we convert this data type, what we have to do is to get access to the year attribute of this column. And we can do this by writing the name of the column. I'm going to show you here in this cell. So, here DF[DOB]. And then to get to the year, we have to write first the dt attribute. So, this is some kind of datetime. And now to get to the year, we write .year. So, this is the year attribute. So, now I'm going to run this. But first, I have to convert the column to datetime type. So, here I paste it. And now I run this one. And as you can see here, I got only the year of each date. For example, Cristiano Ronaldo was born in 1885. And here is the year. So, we extracted only the year by using this year attribute. So, here we can do the same but with a query method. So, I'm going to show you how to do it with a query method. So, first we write df.query, open parenthesis, then this uh quotes, and inside we have to write the name of the column. So, instead of writing this uh series syntax, we have to write only DOB. So, here we have to write DOB's attribute. So, .dt and then .year, as we did here before. So, it's the same, just with the syntax that we need for the query method. And then we have to compare this with 1990. So, here we're saying that the year is greater than 1990. And that's my query. So, now I can run this one to verify that all the players were born after 1990. So, I run this one, and as you can see here in the data frame, all the players were born after 1990. For example, Neymar Jr. 1992, and this one 1991, and yeah 1997, and we filter this data frame successfully. Now your task is to write the equivalent boolean slicing of this query here. And you can use this code I wrote here to help you. And that's it. In this video, we learn how to filter a data frame using the query method.

In this video, we're going to learn how the apply method works in pandas. So, I already imported pandas, and I read this data frame that we're going to work with. So, now I scroll down in here. I'm going to show you how the apply method works in pandas. So, the apply method helps us apply functions or operations to a column or to a data frame. We can apply built-in functions as well as our own functions. So, I'm going to show you how to do both. First, we're going to use the numpy function and apply it to a series. So, in this case, we're going to apply a function to the H column or the H series. So, this one. So, to do that, we write first DF, and then we write the name of the series. So, H, and then we apply this method. So, we write .apply. And now we open parenthesis. So, now inside parenthesis, we have to use this function. And in this case, I'm going to use a numpy function. So, first I'm going to import numpy as np. So, here I have it, and now we have to write the function we're going to use here. So, in this example, I'm going to use the square root function. So, here we write np.sqrt, which stands for square root. So, now if we run, but first let me show you how this looks like without the apply. So, here I run, and here we have the ages of these players. But now if we apply this function, we're going to get the square root of each H. So, now I press Ctrl+Enter, and we get the result. So, this is a square root of the age of Messi, for example, and this is the square root of the age of Ronaldo.

Okay. And that's how you apply a function to a series. And now I'm going to show you how to create your own function and apply it to a data frame. So, here I scroll down, and now we're going to create our own function that calculates the BMI of a soccer player. So, here I write def calculate_BMI. So, now I write the parameter here, and it's going to be row. Now I press here the column symbol. Now I press Enter, and here I write return. And now I'm going to write the operation that I want to perform here. So, to calculate the BMI, we have to divide the weight in kilograms by the square of the height in meters. So, here we have to convert this height into meters and then get the square. So, here I scroll down, and now I'm going to create this uh formula. So, the weight divided by the square of height. So, meters, and here this number two. And now here in the formula, I'm going to write row, which is this one. By the way, this represents the data frame. I just didn't write df. So, to avoid any confusion with this df data frame. So, I wrote row, but this represents like the data frame. So, now row, and then we open square brackets. So, we're going to make a selection of a column. And now here I'm going to select the weight column. So, here I'm going to copy the name of this column. And now I paste it here. And now here we have to introduce the other uh column, which is the height. So, I copy this one. And now this is in cm. So, we have to convert this to meters. So, we divide this by 100. And now to get the square, we have to uh open parenthesis here. And now write this symbol twice, which is multiplication. And if we write it twice, we get the square. So, here if I write two, it means that we get the square of this number. So, now if I want to divide this, I write this operator. And now we get the weight divided by the square of the height in meters, as you can see here. So, now this is ready. And I'm going to run this one. But first, I'm going to delete this. And now it's ready. So, I press Ctrl+Enter. And now I can apply this function using the apply method. So, here I'm going to write the name of the data frame, df, then .apply, open parenthesis, and inside I can specify the name of the function as I did here with this numpy function. But in this case, I'm going to write the name of my function. So, calculate_BMI. So, I write it here. And now I have to add one more parameter, which is axis. And this means how we want to make this operation or how we want to apply this function. So, here in our function, we want to make operations with multiple columns. So, here I have column number one and column number two. So, we want to make some operations between two or more columns. This is why we have to write here axis equal to one. In this way, we can make operations with multiple columns as we are specifying here. So, now it's ready. And now I'm going to run this cell. And here you can see that we get the BMI of each soccer player. So, here I have 24.9 for Messi and 23.7 for Ronaldo. And that's it. In this video, we'll learn how to apply functions to a series and to a data frame.

Okay, in this video we're going to learn how the lambda function works and how we can use this function inside the apply method. So first I imported pandas and now we have the data frame as we had it in the previous videos and now I'm going to scroll down and here I'm going to explain you how to use the lambda function. So first to give you a simple explanation, let's start by creating the average or the basic function we learned in the Python crash course. So okay, let's create a Python function that sums two values, a and b. So here I write def sum_values, which is the name of my function. Now I introduce the parameters. So a and b, then colon, and then I want to sum these two uh parameters or these two values. So I write a + b. Now I assign this to a variable which I name x and then I return x value. So here return this x variable which is here. So this is my basic function. So this is the function we've been using so far in this course. And now if I want to call this function, we write the name of the function and then we pass in two arguments. So here I want to sum two and three. And now if I run this I should get five. So here I run it. And now we get five because 2 + 3 is five. So now this is how the

basic function works. And now I'm going to create this same function, but now using the lambda function.

So here in the lambda function, we have to first write the name of the function, which is going to be uh, let's say `sum_values_lambda`. So we know this is the lambda function. Now, equal to, and here we have to first write the lambda key. So we write `lambda`, and now you can see that this is in green. So this is the lambda key, and now we have to write the input and then the output. So first the input. So here my input is `a` and `b`, as we wrote here before. So `a` and `b`, and then colon. So we write colon to separate the input and the output. And now I'm going to write the output. And the output is going to be the operation we want to perform. So in this case, `a + b`. So we want to sum these two values. So here I run this one, and now we created this lambda function. So this lambda function is supposed to get two values, and then it returns the sum of these two values. Okay.

Now I want to show you how this lambda function looks. So I copy and paste it here. And now I'm going to show you how the average function looks. So here you can see that it says `function`, uh, and then the name of the function. And here it says `function`, but here we have `lambda`. So this is a lambda function, and this is just the average function or the basic function we learned. And now I'm going to apply this lambda function. So here I write parenthesis, and now I pass in two arguments. So in this case, it's going to be two and three. So we introduce these two values, and we should get the sum of these two values. So five. So I run, and yeah, we get the number five. So here we created this lambda function, which is the equivalent of this function. So just a recap: now this is the input, which is the same as these parameters here, and this is the output. So what we're going to return, then the name of the function we have to assign it here, and this is the name, and here this is the equivalent of the name of the function here in blue.

And now you might be wondering why we need to use the lambda function. The lambda function is useful when we want to create a temporary function. So a function that we're going to use only once and then we're not going to use anymore. So that's one of the cases when we use the lambda function over the average function. And also sometimes we use it when we want to simplify things. So it's like a one-liner compared to the average function that has more lines. So it's much simpler. Okay.

Now I'm going to show you how to use the lambda function with the `apply` method. So here I scroll down, and our first task is to use the lambda function to convert the `height_cm` series to meters. So here what we have to do is first write the name of this column. So `df` here, square brackets. Now I copy and paste this `height_cm`. So it's here. And now we write `apply`. This is the `apply` method. And now inside parenthesis we write the `lambda` key to create the lambda function. And then we write the input. So here we can write any variable. I'm going to write the letter `x`. And this is going to be my input. And now I write colon. And now I have to write the output. So in this case, the output is going to be `x` divided by 100 because we want to convert cm to meters. So here `x / 100` is how we convert centimeters to meters. And this `x` represents one row of this uh series. So we're going to apply this uh lambda function to each row in this series. So now I'm going to show you how this works. So I run this one, and we have the height, but now in meters. So before, Messi was 170 cm, but now it's 1.7 m. And that's how to use the lambda function with the `apply` method. And in this case, we don't need to set a name for this function as we did before. Here we set a name of the function. But here when we combine the `apply` method and the lambda function, we can leave it as it is without the name.

So now I'm going to show you an alternative to this `apply` method. So how we can achieve this uh same task without the `apply` method. So this is really simple; we only have to write the name of the series and then divide it by 100 to convert this to meters. So now I comment this out, and now let's test it out. So here I run this one, and as you can see, we got the same results. And now you might be asking yourself why we need to use the `apply` method if there is a simpler alternative. So in this example, the `apply` method looks unnecessary, but there are some cases where this `apply` method with the lambda function is really useful. And I'm going to show you this in the following example.

And our second task is to use the lambda function to convert the `long_name` series to uppercase. So here let's have a look at this series. So I print this data frame. So this `long_name` column has the full name of the soccer players in uppercase and in lowercase. But here I'm going to delete this one. We want to get the full name in uppercase. So we can do that with the `apply` method and the lambda function. So we write first the name of the series or the column we want to get. So this case is `long_name`, which I have to paste it here. And now we write `apply`. Now parenthesis. And now I write `lambda`. So I'm going to use the lambda function. And I write the variable. So `x`. This is my input. And now the output is going to be `x`. And here we're going to use the `upper` method. So we learned this method before. And we only have to write `upper` and then parenthesis. And here we can use this `upper` method because this `x` is a string. And why this `x` is a string is because here this `long_name` series contains letters. So contains strings. So as you might remember, this `long_name` contains the full name of a soccer player, and that's a string. So here the `x` represents one element of that series, and this means that this `x` is a string. So a string has this `upper` method, and that's why we can use the `upper` method. So here I run, and now as you can see in the result, the names are in uppercase. Okay.

And now I'm going to show you an alternative to achieve this, but now with the `str` attribute. So here I have to write the name of this series. But now we cannot use this `upper` method right away because if we paste this, and now I'm going to comment this out. So now if we do this, we're going to get an error because here uh this is not a string, but this is a series. So here when we do this `apply` with `lambda`, here this `x` represents a string because this is an element inside this series. So that's one of the benefits of the `lambda` in the `apply` when we use them together. This `x` represents one element of this series, and one element of this series is a string because these are names. However, here this is a series. So we cannot use the `upper` method because the `upper` method only works with strings. So here what we have to do is to get access to the string attribute, and we do that by writing `str`. So this is the string attribute, and now we can use the `upper` method because we got access to the string attribute. So now if I run this, we can see that we got the full name in uppercase. And here we could see one of the benefits of the `apply` method when we use it with a lambda function because here when we use the `lambda` in `apply`, we didn't need to specify any attribute. But here we have to specify an attribute, and that means we have to remember all the attributes an object has. And sometimes when you're a beginner, that might be complicated. I remember many times that I didn't remember all the attributes, and sometimes I did something like this, and as a result, I got errors. So here when you use the `apply` and the lambda function, it's much easier; at least it's easier to remember than many attributes here.

So here I'm going to show you another example, and in this case, we have to use the lambda function to get the year of the `dob` series. So here I'm going to show you this data frame again, and the `dob` column or `dob` series is this one. So here we have the date of birth, and we want to extract only the year. So this one. So to do that, first we have to um convert the data type of this column into a date. So if we get the data types here, we will see that the `dob` is an object, and this should be a datetime type. So we can convert this data type into a datetime by using the `astype` method. So here I'm going to write `df`. Now the name of this column, which is `dob`, and now I write `astype`, and this method allows me to convert that data type. So in this case, I want it to be `datetime64`, which is the standard datetime in pandas. So now I'm going to overwrite here this column, and I'm going to change the data type. So here I run this, and now if I show the type, we're going to see that it changes. So here now the date of birth column has a different data type. So in this case, it's datetime. And now this is ready. So here I'm going to delete this one. And now that this is a datetime uh type, here we can extract the year. So now what we have to do is to write the name of this column. So `df.dob` and then `apply lambda`, and now the input, which is the `x`, now colon, and now `x.year` to get the year. Since this is a datetime type, we can get access to the `year` attribute. So we only have to write that `year`, and that's it. So here we can run this one, and as you can see in the result, we get only the year. So for example, Messi here we have the date of birth that only represents the year. And here Ronaldo was born in 1985, and so on. So here we did this with the `apply` method and with the lambda function. But here there is an alternative using attributes. So we only have to write the name of the column followed by the `dt` attribute that represents that datetime. So `d` is date and `t` is time. And now we can get access to the year. So here I'm going to run. I'm going to comment this out. And now I run. So I did it. And now here we got only the years. And as you can see, is the same result. So here we can see again one of the advantages of the `apply` and `lambda` function because here we didn't have to use the `dt` attributes. But if you wanted to use the alternative, you need to know that there is a `dt` attribute because if you don't know about this attribute and you only do this, you're going to get an error because here this again is a series, and you need first to get access to the data type. So here we have to write `dt`, and now after that `year`, and that's how it works. But if you don't have any idea about this `dt`, you wouldn't be able to do it this way, but only with the `apply` and the `lambda` function. So here is one of the benefits of this lambda function with the `apply` method.

Finally, in this task, we have to apply the lambda function to a data frame in order to calculate the BMI. So we did this before to calculate the body mass index, but in this case, we're going to do it with a lambda function. So first we create this lambda function, and here I'm going to create a function. So first the input `x`, and now the output. So the output is going to be here `x`. Now we open quotes, and we need two uh columns. So the first one is the weight and the second one is the height. So here I'm going to print the data frame. So I have the columns. So here I copy first the weight and I paste it here. So here we're selecting the weight column. And now I'm going to do the same. So I copy this one and I paste here. And now I copy the height column. So here we have the weight and the height. So now I convert the height to meters. So I divide it by 100, and now I use parenthesis, and here this symbol twice with the number two to get the square, and now we have to divide the weight by the height. So here I write this symbol, and now we have this, but here we have to add a parenthesis. So now it's ready. So we have this lambda function, and now we have to insert this inside the `apply` method. So write `df.apply(`, parenthesis, and now we insert this as the first argument, and as a second argument, we have to use the `axis` parameter. So this `axis=1`. So here now is ready, and I'm going to run this one to see the results. So here I run it, and we got the BMI of each player. So here Messi has 24.9, and Ronaldo has 23.7. And that's it. In this video, we learn how to use the lambda function with the `apply` method.

All right. In this video, we're going to learn how to make a copy of the data frame using the `copy` method. So here I have the data frame that we're going to make a copy of. So is this one. And now I'm going to show you how to do it. So we're going to use the `copy` method. And here to make a copy of this data frame which has the name `df`, we only have to write the name of the data frame. So `df`, and then `copy`. Now we open parenthesis. And here by default, the `deep` parameter, which is a parameter of this `copy` method, is set to `True`. So this means that all the modifications that we make to the data in the original data frame is not going to be reflected in the copy and vice versa. So here I'm going to show you with an example. If I make a copy, and now I set this copy to `df_copy`, and now I run this, and if now we update a value of this original data frame `df`. So let's say we want to update the height of Lionel Messi. So we do `df.loc`, then square brackets, then `'Lionel Messi'`. Now the name of the column which is `height` here. So I copy and paste it. So I paste it here. And now if I set this to another value. So let's check. Right now Messi has 170. So if we set this to 180. Let's see how this is going to affect the copy and also the original data frame. So if we now print the `df` data frame, which is the original data frame, we're going to see that here in the height we have Messi, and the new height is 180. So this makes sense because we just updated this height. But now if we print the copy data frame, let's print it here, we're going to see that the value of this height is still 170. So this means that we created an independent copy. So this `df_copy` is independent from this `df` original data frame. So any update we make to the `df` data frame is not going to be reflected to this `df_copy` because both are independent. That's usually the behavior we expect from copies. But if for some reason you want that the copy and the original data frame are dependent, you have to set this `deep` parameter equal to `False`. And I'm going to show you here below. So here I copy the code. We're going to make a copy. But in this case, we're going to modify that `deep` parameter, and we're going to set it equal to `False`. So here I write `deep=False`, and this is going to create something called a shallow copy, which means that all the changes that we make to the original data frame will be reflected to the copy. So both are going to be like dependent. So now I'm going to update a value, and in this case, let's update the height of Cristiano Ronaldo. So I write `df.loc`, and now I write `'Cristiano Ronaldo'`, and then we have to write the name of the column we want to update. So in this case, it's going to be the `height` column. So here I copy and paste it. So now as we can see here, Ronaldo is 187, and now we want to change this to let's say 200. So 2 m. Okay. Now before I set this new height, first I'm going to make the copy. So here I make this shallow copy. And now here I'm going to actually I'm going to name it `df_shallow_copy`. So now I run this again. And now this copy was created. And I'm using this `df_shallow_copy` variable. And now I'm going to set this height of Cristiano Ronaldo equal to 200. So now I run this one. And now let's see how the `df` data frame and the `df_shallow_copy` data frame look. So here let's print them. So first `df`. And now we can see that in the original data frame here the height of Cristiano Ronaldo is now 200. And this makes sense because here we set the height to 200. But now if we print the shallow copy here, we're going to see that the height of Cristiano Ronaldo in the shallow copy is equal to 200. So this means that the changes we made in the original data frame `df` were also reflected in this shallow copy. So this `df` and `df_shallow_copy` are dependent, and this happened because we set the `deep` parameter equal to `False`. Okay.

Now I'm going to show you another way to make a copy. And in this case, it's going to be with a simple assignment. So here this works like this. We want a copy named `df_new_copy`, and we set this equal to `df`. So this is the simplest way to make a copy with a simple assignment. And now we run this. And now let's update another height, but in this case, of Neymar Jr. So we write `df.loc`. Now the name `'Neymar Jr.'`. And now the `height` column here I quickly copy. And now I paste it here. So now let's see. The height of Neymar Jr. is 175. So let's update this to 190. So we update the original data frame. And now let's compare the copy and the original data frame. So here I print the `df`. And now we can see that the height of Neymar Jr. is 190. And yeah, that's correct. But now if we print the copy `df_new_copy`, we're going to see that here the copy was also updated. So the new value is 190. And this means that this copy is a shallow copy because all the changes we made in the original data frame were reflected in this copy. And this is the equivalent of making a copy using the `copy` method with `deep=False`. So just keep that in mind when you do something like this. And that's it. In this video, we learn how to make a copy of a data frame using the `copy` method.

Welcome back. In this video, we're going to see different ways to make pivot tables. If you're an Excel user, probably you made many pivot tables in the past. In pandas, we can also make pivot tables. And in this case, we use two different methods, the `pivot` method and the `pivot_table` method. In this video, we're going to see the difference between the two of them. So first let's see what's the `pivot` method. So the `pivot` method reshapes data based on columns values, and it doesn't support data aggregation. So this means that this is not the regular pivot table you will see in Excel because you can only reshape data with the `pivot` method, and you cannot do anything else. To explain you better what the `pivot` method does, I'm going to show you an example. So here we have a little data frame, and this one has six rows and four columns. And as you can see here, there are many duplicate values. For example, in the column `foo`, the `1` value is repeated at least twice, and the same goes for the `2` value. Also, in the column `bar`, you can see that the `a`, `b`, and `c` is duplicated. So when we have this type of data frame, we can reshape it to have a different view and to make a better analysis. In this case, we can use the `pivot` method, as I'm going to show you right now. You only have to write the name of

The data frame followed by the pivot method, and then specify three arguments. So the first one is the index. In this case, I'm going to reshape this data frame choosing the column "fu" as an index. This means that the column "fu" will be in the position where is right now the numbers from zero to five on the left.

Next, you have to define the columns. So, these are the new columns that we're going to see in our new data frame, the one that we're going to reshape. So in this case, I'm selecting the data inside the "bar" column as new columns. This means that A, B, and C will be the new columns in our new data frame.

And finally, we have to choose the values we wish to show in this new data frame. So in this case, I'm choosing the "best" column. So all the values inside there will be shown in our new data frame. So this is the column that I'm selecting. And now I'm going to show you the result of this pivot method. So here it is.

And as you can see here, we have the "fu" in the index as I told you before. And the a, b, and c that are data from the "bar" column now are columns in this new data frame. Also, all the data inside this "B" column is the only data that is displayed in this reshaped data frame.

And now let's see why it's sorted this way. So why one is here, two is here, three is here, and so on. So here the value is defined by the index or row and the column. So between index one and column A is one. And why that happens? Because if we go to our previous data frame or the original data frame that is here, we can find that here is one, A, and the value that corresponds to that pair is the number one.

So let's pick another one. For example, five. Here we have two and B. And if we go here to our original data frame, we have that two and B, the value that corresponds to that pair is five. So that's why this value is here. And that's how this new data frame was reshaped. Okay.

And finally, we have the pivot table method. And this one creates a spreadsheet-style pivot table. So this is similar to the pivot table that we will find in Microsoft Excel, for example, and this one supports data aggregation. To explain more about the pivot table method as well as the pivot method, we're going to see some examples in the next video, and this time we're going to write some code so you can understand much better what we're doing.

All right, now it's time to see how the pivot method works in action in pandas. So first, as usual, we import pandas as pd. So here I import this library, and then we're going to use a different data set to work with this pivot method. So to read this data set, we use the pd.read_csv method, and inside parentheses, we write the name of this data set. So in this case, it's "GDP.csv" that you can find in the notes of this video.

So this is the new data set, and now let's have a look. I'm going to run this one. And as you can see here, we have data about GDP per capita that is in this column. And basically, this is how the GDP grew over the years for each country. So here I'm going to tell you which are the columns we're going to use for this example.

So first, we're going to use the "country" column that contains data about different countries. Then we're going to use the "year" column that well contains different years and the "GDP per capita" that it's in this column. So basically, what we want to do in this exercise is to obtain a different view of our original data set. So this data set that we're reading here with pandas has this view, but we want to get a different view to have a better analysis.

So the goal of this exercise is to see the evolution of the GDP per capita over the years for each country, and then we're going to put the country names in the columns. So the only data we're going to show in our new data frame is going to be the GDP per capita that it's here. So I'm going to show you now this with code, and let's write it here. But first, let's assign a variable to this data frame. So here I'm going to write "df_GDP". So this is the name of my data frame, and now I'm going to show it, and it's here.

So now I'm going to copy this data frame and to use the pivot method, I'm going to paste this one and now write ".pivot". Now we open parentheses, and now as you might remember from the previous video, we have to introduce three different arguments. And if you don't remember the three different arguments we have to introduce here, you can only press the Shift and Tab keys on your keyboard, and you will get this. And here you can see the three arguments I'm talking about.

So first, we have to write the "index" argument. So we write "index=", and as I told you before, I want the "year" column to be the index of my new reshaped data frame. So I'm going to set this "year" as the index of my new data frame. So I write here "year". Next, we write a comma and press Shift and Tab to show this. So the second argument is the "columns". So we write "columns=", then equal and open quotes. So here, as I told you before, I want the countries here listed in the "country" column. I want each country to be an independent column. So, for example, here, let's say we have the United States. So I want the United States to be column number one, then column number two, China, then Australia, then Spain, and so on. So each country should have one independent column. So that's what we want. And to get that, we have to set the "country" column here to the "columns" argument. So here "country", and that's it. Now again, Shift plus Tab to show this window here. And now the third argument is "values". So here I'm going to write "values=", open quotes. And here the only data I want to show here in my new data frame is going to be the "GDP per capita", which is the one that is here. And now I'm going to copy this one and paste it here. So remember our goal, our goal is to see the evolution of the GDP per capita over the years for all the countries listed here in this column. So here we're going to execute this code and let's see the result. So here, Ctrl+Enter, and as you can see here, I have the new view of this data frame, and it looks much better. It's more readable because we can see the GDP evolution over the years for each country.

So now let's verify if everything is correct. So here we have the index "year", and here we have the "year" as index. So everything is fine. Then the columns should be "country". And now we have each country in the columns. So it is correct. Next, the values are the "GDP per capita". And yeah, we have here the intersection between the row and a column is a value that corresponds to the GDP per capita of that country in that year. So everything is working fine. And there you have it. This is how the pivot method works in pandas.

Okay. Now let's see how the pivot table method works in pandas. So in this case, we're going to work with a different data set. And to read it, we're going to use the method pd.read_excel because in this case, the data set is not a CSV file, but an Excel file. So we use read_excel for an Excel file. So in this case, the name of this data set is "supermarket_sales.xlsx". "xlsx". And this is what we're going to see after you run this.

And here you can see that we have different columns about what a specific person bought in a supermarket. And here, well, we have the branch, the city, the gender, and different data. So here to make a pivot table, we're going to first name this data frame. And I'm going to name it "df_sales". And now I'm going to show it here. And okay, now it's here. Okay. The goal of this task is to see how much female and male spend their money in this supermarket. So to do that, we're going to use the pivot_table method in pandas. So first, I'm going to copy this data frame. And now I'm going to paste it here. And now we're going to make a pivot table and add an aggregate function because remember that the pivot method doesn't support that. So we're going to use the pivot_table this time. And now we're going to introduce some important arguments. So the first one is the index. And in this case, if we want to see how much male and female spend in this supermarket, the index is going to be the gender. So here I'm going to copy "gender" here, and it's going to be here "index=gender". So this is the first necessary argument, and the second one is going to be the aggregate function. So we have to write "aggfunc=" and then write the aggregate function we want to perform. So in this case, it's going to be a sum. So we write "sum", and now everything is ready. So what we're supposed to get here is the information about the sales here in this data frame. But now divided by gender. So we have the female category and then the male category. So let's verify this. I'm going to run this one.

And as you can see here, we have this summary table or pivot table. And now it's divided by gender. So we can see how much female spent here in the "total" column. And also how much male spent also in the "total" column. And here in the "quantity" column, we can see how many products they bought, how many products female and male bought in this supermarket. And one detail you might have noticed is that only the columns that contain numerical data are displayed here. So, for example, here "branch" and "city" that contain only text aren't here in this pivot table because here in the "aggfunc" argument we indicated that we want to sum, and when we sum values we cannot sum text but only numerical data. So only the columns that have numerical data are displayed in this new pivot table. Okay, that's our first pivot table.

And we can do even more. For example, we can select a pair of columns that we're interested in. So let's say we only care about the "quantity" and the "total" column. So we want only those columns. So we can get that. I'm going to copy this one. And to show you how to get only those two columns, I'm going to add a new argument. So here I'm going to write, and in this case, the name of the argument is "values". So I write "values=", and in this case, I'm going to select the "quantity" and the "total" columns. So I open square brackets because I'm going to select two or more columns, and inside I write the name of the columns. So first "quantity", I write here, and then "total". So here two. So we're going to get the same pivot table, but in this case, only the "quantity" and "total" columns are going to be shown in this table. So I'm going to execute this one. And here I got an error because I didn't include this comma. So I'm going to add it here. And now everything should be fine. And yeah, we got the same pivot table, but only the "quantity" and "total" columns are displayed here. And here we can clearly see that female spent more than male in this supermarket.

But we can get even more detail here. So far, we know that female spent 167,000 in this supermarket. But with pivot tables, we can even know in which product lines this money is spent. So let me show you here. We can see how the money is spent in this "product_line" column. So we only have to add a new argument to this pivot_table method. So I'm going to show you here. First, we copy this. And now I'm going to paste it here. And we're going to make a pivot table that says how much male and female spent in each category or well product line. So we add a new argument, and this one is going to be the "columns" argument. So I write "columns=", then open quotes. I add the comma, and here I write the name of this column that is "product_line". So I scroll up, I copy this column, and then we're going to see in which category is spent the money. So "health and beauty" or "sports" and so on. So now I scroll down and here I paste it. And before I run this code here, we only want to display the "total" because we only want to see where the money goes, not the quantity. So only "total". So I delete the square brackets too. And with "total", we're going to see where the money goes divided by gender. So here I run because it's ready.

And now as you can see here, we can see how much female and male spent in each product line. So we can quickly see, for example, that female spent more money on fashion accessories than male. And that kind of makes sense. And also in sports, women spend or female spend more money than male. So we could easily see all of that by using the pivot_table method in pandas. And this is similar to the pivot table you will find in Excel. And that's it. That's how you make a pivot table in pandas.

All right, before showing you how to make visualizations with pandas, first, we have to check the data set and also we have to make a pivot table so we can easily make the plots with pandas later. So first, we have to import pandas to read this CSV file, and well, I have this "import pandas as pd". So we just run this code, and now let's read this new data set. So as you might remember, to read a CSV file, we have to use the read_csv method. So we write "pd.read_csv", and then we write the name of the CSV file. So in this case, the name is "population", and I'm going to use this "population_total.csv". So I pressed Tab to get this, the name. So we have now the name, and now I'm going to assign this to a new variable. So the variable is going to be "df_population_row". So this row data, and now we're going to have a first look at this data set. So I paste this. And now I'm going to run this too. And now we have this data frame. So here, as you can see, we have the population of many countries throughout the years. So, for example, we have China here, United States, and India. So we have the population, and here I wrote the name "row" because this data set was extracted using some web scraping techniques, and then it wasn't modified. So now we have to make some changes to reshape this data frame. So we make it easy for us to make visualizations with pandas later. So what we have to do here is to make a pivot table to reshape this data frame, and that's what we're going to do here below. So we're going to make a pivot table, and we're going to use the pivot method. So as you might remember, the pivot method returns a reshaped data frame organized by given index column values, but it's a pivot without aggregation. So this is what we want. So we only want to reshape this data frame. So we're going to start by dropping null values. So we do that by writing the name of the data frame. And now I'm going to just copy the name and I paste it here. And now to drop null values, we have to use the dropna method. So I write ".dropna". And then we have to run this. And as you can see here, we have the result. And it's a copy from this data frame. But if we want to save the changes that we make to the data frame, we have two options. The first option is to use the "inplace" argument. So I write "inplace=", and then set this to "True". So if we do this and we run, all the changes that we make to the data frame are going to be saved. And the second option is to do something like this to override the content inside this data frame. So we do something like this. We write "df_population_row = df_population_row.dropna()". So we're overwriting the content inside this data frame. So I'm going to choose the first option just to reduce some code. So I write "inplace=True". And now I run. And this new data frame shouldn't have any null values.

Okay. Now it's time to make this pivot table. So first, I'm going to show you what I'm going to do so we have a better idea before writing the code. So here we have the original data frame, and what we're going to do is to reshape this data frame. So I want the "year" to be in the index. So the "year" column, I want it to be here in the index instead of 0, 1, and so on. And then I want the "country" column or the country, the values inside the "country" column, I want it to be here in the columns. So, for example, I want China here in one column, then United States in another column, and then India in another column. And I want the "population" data inside the data. I wanted this to be the only data here. So to do that, we have to use the pivot method. And that's what we're going to do here below. So let's do it here. So first, we have to write the name of the data frame, which is this one, and then write ".pivot", then we open parentheses, and here let's see the arguments that this pivot method accepts. So I press Shift and Tab to get this helpful, let's call it cheat sheet. And now we have the arguments that this pivot method accepts. So first is the "index", then the "columns", and then the "values". So as I told you before, the "index", I want it to be the "year" column. So we have to write "index='year'", then comma, and let's check another argument. So the next argument is the "columns". So I want the columns to be the "country". So the data inside the "country" columns. So here I write "columns='country'". So "country", and now the last one, I think is "values", and yeah, it's "values". So I want the values to be the "population" data. So let me see if that's correct. And yeah, it's here. So "population", and I'm going to press Enter here. So it looks much better. And now "population" is here. So I have the three arguments, the "index", the "columns", and the "values". And now I'm going to reshape my original data frame. So here I press Ctrl+Enter. Now as you can see here, we have the countries in the columns. So here we have many countries, uh, it's from the first country Afghanistan to, and Pandora, Argentina, Uruguay, and many other countries. So we have also the "year". So it's here the "year", uh, from 1955 to 2020. So we can see here the evolution of the population throughout the years for all the countries in this data set. But as you can see, there are many countries. So what we can do here is to select just some countries so we can simplify our visualizations later in pandas. So here I'm going to select some columns. But first, I'm going to name this new data frame. I'm going to give it a name. So I'm going to name it "df_pivot". So this is my new data frame. Now I'm going to rearrange this. And now it looks much better. So now I'm going to run this. And now let's select some countries. So I copy this "df_pivot" data frame. And now we open square brackets, double square brackets to select two or more columns. And here let's write some countries. So first "United States", then let's say "India", then "China", two more countries, "Indonesia", and last but not least, "Brazil". So here we have the five countries. So I run here, and we have these five countries and the population from 1955 to 2020. So great. Now we simplify this data frame. And now I'm going to overwrite the content inside the data frame "df_pivot". And I'm going to write here "df_pivot = df_pivot[ [...] ]". So I'm overwriting the content. So I press Ctrl+Enter, and our new "df_pivot" is here. So we

Have it here. And now I'm going to show it to you. And this is our new DF_Pot data frame. And that's it. Now our data is ready. So we can use it to make gray visualizations with pandas. And that's what we're going to do in the next video.

Okay, now it's time to make some visualizations with pandas. And here I have the data frame that we created. This is the pivot table we created in the previous video. And as you can see here, we have five countries in the columns. And here we have the year in the index from 1955 to 2020. So what we're going to do now is to make our first visualization. So I scroll down here, and the first one is going to be line plots. So here, first to make this visualization, I'm going to copy the name of the data frame and I paste it here.

So now to make plots with pandas, we have to use the plot method. So we write plot, and now I open parenthesis, and one necessary argument we need to introduce is the kind argument. So I write kind now equal to, and here I have to write the kind of plot we want to make. So in this case, it's a line plot. So we write line, and this is actually the mandatory argument we have to introduce here, and now we can run this code. So I press control enter, and as you can see here, I have the line plot. So in this line plot, we can quickly see the evolution of the population throughout the years. For example, China and India, which are green and orange lines, they had some uh fast-growing population, while United States, Indonesia, and Brazil uh they have a lower population, and also the population didn't change so much in the past 50 years.

Here we can add more arguments to this plot method to customize this line plot. So here we can introduce another argument, which is the x label, and this x label is what you can see here. Here uh when we created this line plot uh by default it was assigned this year label, but we can change it. So, for example, let's say we have we want to write year, but now with a capital letter. So we write Year here. And now let's say we want to add a new label here in the y-axis. So here we have only to write y label and then equal to, open quotes, and here we have to write the name we want. So in this case, I'm going to write only population. And finally, we can also add a title. So we can add any title we want. In this case, I'm going to write uh well the name of the argument first, title, then equal to, and then the name of the title. It's going to be let's say population uh from 1955 and to 2020. So this is the title. So let's run this. And now as you can see here, we got the title Population 1955 to 2020, and the X label and Y label were modified too.

Finally, we can add one more argument. In this case, the argument is the size of the figure. So to change the size of this figure, we can add the argument name figsize. And this is a tuple. So we have to open parenthesis. And now to edit the size, we have to add two arguments. The first is the size of the x-axis, and the second the size of the y-axis. So in this case, I'm going to set it to a uh and then four, which means that the x-axis is going to be uh large while the y-axis is going to be short. So here I'm going to run this code, and let's check it out. So here the figure has a different size, and that's how you can customize this line plot.

Okay. Now let's make a bar plot with pandas. So the first thing we have to do is to select only one year. So the bar plot only accepts just one year, and we can plot there the population of different countries. So let's select one year of this data frame we have before. So I'm going to copy the name of the data frame so you can check it out again. So this is the data frame, and we're going to select one year. So to do that, we have to use the index attribute and then the isin method. So first I'm going to show you the index method in case you don't remember. So here that index, sorry again, that index attribute allows us to see all the index in this data frame. So we have here from 1955 to 2020. So that's what the index attribute does. And now if we use the isin method, we can filter out some index. So here let's say we want to select only that 2020. So I copy 2020. And now here I write equal to, and first I'm going to make the selection. So it's here. And now I'm going to show you what's the result. So here I press Ctrl enter. And the result is this little data frame that only contains the population in the year 2020. So this is important because the bar plot is supposed to show only the population in this year.

So here we have it, and now what we're going to do is to name this data frame. So here we write equal to, and then let's give it a name. So I'm going to name it DF_pivot_2020. So here I press Ctrl enter, and now I'm going to show this new data frame. Well, again here, and here one little detail I have to tell you is that when we make bar plots, we have to put text data in the index. So here the name of the countries should be in the index. So to do that, we have to use the transpose method. So this transpose allows us to switch rows and columns and vice versa. So here we can easily do that by writing the data frame, the name of the data frame, and now .T. So if now we run this code, we can see here that we have this. So now the year 2020 is in the column and not in the index anymore, and the country names are in the index here. So this is the format we need to have before making the bar plot. So now I'm going to overwrite the content in this data frame. So I write df_pivot_2020 equal to this same data frame but .T. So here I run this, and now it's time to make the bar plot. So here I copy the name of the bar plot, and now I use the plot method. So I write plot again, open parenthesis, and the first argument is the kind. So I open quotes, and we write bar. So now it's ready, and we can run it. So as you can see, we have a basic bar plot, and it has some default values like uh the name of this x label, and also the default color is blue. And we can customize this bar plot a bit more. For example, I want a different color. So I write the color argument and then open these quotes. And let's say I want it to be orange. So I write orange. And also we can change the X and Y label. Actually, I can copy this here. So I can save some time here. So X label and Y label are here. And let's paste it here. So X and Y label. And finally, I can add also the title which was here. So I copy and paste it. But in this case, the title is a bit different because in this case, it's not from 1955 to 2020, but it's only 2020. So here I have only 2020. And now let's run this to see the results. So you can see here we have a title, the X and Y label, and the bar plot is in orange. So that's how you customize the bar plot.

All right, so far so good. Now let's go one step further by making bar plots grouped by n variables. So here we have to select a group of years to make these bar plots grouped by n variables. So I'm going to copy this code we used before to select only the year 2020. I'm going to copy this, and in this case, I'm not going to select only one year but a group of years. So let me show you here, instead of choosing only 2020, I'm going to show you the pivot table again so you can easily understand. So instead of choosing only 2020, I'm going to choose some other years here. So I'm going to delete this, and I'm going to write it here. So let's say 1980, 1990, then [Music] 2000 and 2010, and well finally 2020. So we have a group of years here, and we're selecting this using the index and isin method. So here I'm going to give it a different name. In this case, since it's a sample, I'm going to write df_pivot_sample. Now, I'm going to first I'm going to show you this one so you can see what this looks like. So, now we have five countries and now five years. So, now I'm going to assign this to my data frame. So df_pivot_sample. I run this, and now we have this new data frame. So it's time to make this grouped bar plot. So here we write the name of the data frame and then the plot method. So we write that plot, and now let's add the first argument which is kind and equal to bar. Now we run this, and as you can see here, we have the the plots or the bar plots grouped by year. So here's 1980 and 1990 and so on. And you can also add the same arguments we added here. So, for example, I can add the X and Y label. So I can do it here. I'm going to do it fast. So here I run. And as you can see here, we have the we modify the X and Y label. And that's it. That's how you make bar plots with pandas.

Okay. In this video, we're going to learn one of the most common charts that we can make in pandas and actually in any other visualization tool. And these are pie charts. So before we make this pie chart, first let's give a look to the the data frame we're going to use. And in this case, to make a pie chart, we're going to use the same data frame we used for making the bar plot because it follows the same logic. So here I'm going to copy the data frame we created for the bar plot, which is this one, df_pivot_2020. So this is what we created before by using the index attribute and the isin method. So here I'm going to copy this and now I'm going to show you here. So so you can remember what's inside this data frame, and it's here. So here as you can see uh we have the column 2020, and the countries are in the index. So everything is fine. That's what we need. That's the format we need for making the pie chart. But there is one little thing we have to modify. And this is the column name because now it's 2020, and this is a number. It's actually I think it's an integer. So it's not a good practice to have numbers in columns. So what we have to do is to make this a string. And to do that, we use the rename method. So we write rename, open parenthesis, and now we use the columns argument. So we write columns, then open this curly braces, and now we write the name of the column we want to change, which is 2020, and we're going to make this integer uh value into a string. So we open quotes and write 2020. So apparently they are the same, but the green one is an integer, and the red one is a string. So now to make to save these changes, I'm going to write inplace equal to true. And I'm going to run this. So now we can make the plot here. I'm going to write the name of the data frame. And now I'm going to use the plot method. So here I write that plot. So the first argument is kind, and here I write pi. So the kind is pi, and now I run this, and here I forgot to include that y argument, and I'm going to write it here. So the y argument is supposed to have the data. So in this case, I'm going to show you here again the data frame. So the data is here in 2020. So we should write here 2020. So I'm going to delete this. And here in the Y argument, you write the column that has the data. So that's what we did. So now I run this. And now we finally have our pie chart. So here is the pie chart. So that's how you make a pie chart. If you want, you can even add another argument like the title, for example, here I can say that this is a population in 2020, but in this case in percentages. So write this, and now we have this title. So that's how you make a pie chart in pandas.

Okay. In this video, we're going to learn how to make box plots with pandas. A box plot is a graph that shows us how the data is to spread out. And the box plot shows five important numbers. First, it shows the minimum score of the data. Then the first quartile, also known as Q1. Then the median, then the third quartile, also known as the Q3, and finally the maximum value of the data.

Okay. Okay, now let's make this plot in pandas, and we're going to use the data frame that we created before that is named df_pivot. So here I have it, and I run it so you can remember what this data frame looks like. Okay, let's start by making a single box plot, and then we're going to make multiple box plots. So here since we only want to make a single box plot, I'm going to pick one counter here. So here I write df_pivot, open square brackets, then quotes, and then write the name of the country. So in this case, United States. So now to plot this box plot, we only have to use the plot method. So here we write plot, open parenthesis, and as usual, we have to write the kind argument. So here kind equal to, and in this case, it's going to be equal to box, and that's all we need. So now I run this, and as you can see here, I have the box plot, and here we can add more arguments to customize the box plot. For example, the color argument, I'm going to set it to green, and also I'm going to add a y label, and this one is going to be equal to population. So I run this, and now we have this label in the y-axis, and the box plot is green, and as I mentioned before, this box plot contains five important values. So the one we have here is the minimum value, and this one is approximately 1.7. Okay. Then the one I'm showing right now is the Q1, also known as the first quartile. It should be here 2.2. Then this one here in the middle is the median, and this one should be 2.7, and the one here is the third quartile. So Q3, and this one is the maximum value in the data.

All right. Now let's make multiple box plots. So here I'm going to write df_pivot. And in this case, I'm not going to select a specific country because we're going to make multiple box plots. So we're going to get one box plot per country. So here I just write df_pivot and then plot, then parenthesis. And now the first argument is kind, and here I'm going to set it to box. So now I can run this one. And as you can see here, I got five box plots, and each of them represent one country. Here, if we want, we can add also the color argument equal to green and also the y label argument. In this case, I'm going to set it to population. So, as you can see here, the box plots are in green. And now we have the population label. And that's it. In this video, we learn how to make box plots with pandas.

All right. In this video, we're going to make a histogram with pandas. But first, let's see what's a histogram. A histogram is a graph that organizes a group of data points into ranges. In the graph, these ranges are represented by vertical bars, and they show the frequency distribution of the data.

All right. Now, let's make this histogram. And here I'm going to use the df_pivot data frame. So here is this data frame, and here we can choose any country to make its histogram. So I'm going to choose here Indonesia, and I open square brackets and write the name here. So now I write plot to make the histogram, and here I add the kind argument. So I set this equal to hist, and then I can run this to make the histogram. So I run, and here I got the histogram of the country Indonesia. But that's not all. Here we can make multiple histograms in this plot. So here I add square brackets. And now we add one more country. So here I'm going to write United States. And now I'm going to run this. Here I wrote incorrectly. So I modify. And now I run again. And here we got a histogram of Indonesia and United States. And that's it. That's how you make a histogram using pandas.

In this video, I'm going to show you how to make a scatter plot in pandas. A scatter plot is used to plot data points on a horizontal and a vertical axis in an attempt to show how much one variable is affected by another variable. In this example, we will plot years versus population, and we will see how much the population is affected by the year. So to do that, I'm going to use the data frame that we had at the beginning. So this data frame that we got after reading this CSV file, and it has this format with country, year, and population in separate columns. And this format is going to help us make scatter plots easily because each row will represent a dot in the scatter plot. So here now I'm going to paste the value of this data frame. And now I'm going to make some little changes. So the first thing I'm going to do is to select some countries from this data frame. So to do that, I select the countries column. So I write country inside square brackets. And now I use the isin method. So here isin, parenthesis. And now I select some countries. So I open the square brackets. And now I'm going to choose some countries. So United States and also India, then China, Indonesia, and finally Brazil. Okay, I have these five countries. And now I'm going to set this to a new data frame that I'm going to name df_sample. So now I run this. And here I forgot to make the selection. So I'm going to select this data frame. So I open a square brackets and write the same name of that df_population row. And now we created this df_sample data frame. So let's see how it looks. So here I print it, and now we have this data frame, but now we only have five countries, and we reduce the number of countries because we don't want that the scatter plot is overpopulated. As I told you before, the row represents one dot in the scatter plot, and if we have many dots, we won't be able to see the dots clearly, but with five countries is fine. Now that they have this data frame, I'm going to plot this with dot plot, and now add the kind parameter. So kind equal to, and here scatter. Now the two other parameters we have to add are the x and y parameter. So here I write x and then I write y. So here x and y. And this represents the values that are going to be in the x and the y axis. So first in the x-axis, I want to show the year. So I write year. Now we have this one here. And now in the y-axis, I want to show the population. So I write population, and that's it. Now we can plot this scatter plot. So I run this, and here we get this scatter plot, and each dot represents a row in the data frame df_sample that we have here. Here we see that the dots follow a pattern, and this is not so good because scatter plots are much better when the dots don't follow a pattern. But I'm just making this scatter plot so you know how to make the scatter plot with the plot method. So now we can control the size of the dots. So here I could write s, the s parameter, and here let's write 80. So we make it bigger. And now we see that the dots are bigger. Also, we can control the color of this data frame. So I write here color. And now let's set it to green. And now let's see the result. So now we see that all the dots are in green, and this isn't so good because it's hard to recognize what category each

dot belongs to. So, for example, we don't know if this dot belongs to the country India or United States or Brazil. We only know that the year is between 1980 and 1990, but we don't know exactly the year. And also the population is between 1 million and 1.22 million. But we don't know exactly how much the population is for this dot. I could show you how to set different colors for each dot based on the category, but that will involve importing math lip and also writing a for loop and more lines of code.

Fortunately, there is an easy approach that helps us set different colors for the dots based on the category and also see the values behind each dot, and that's by making interactive visualizations. I'm going to show you how to do that in the next video.

All right. So far, we made a pivot table and many plots using pandas. And in this video, we're going to learn how to export the pivot table and also the plots we made with pandas. So let's start by exporting the plots we made with pandas. And to do that, first we have to import mattplot lib. So we write import mattplot lib.pyplot as plt. So this plt represents this mattplot lib.pyplot. So now we run and we imported matt lip. And now we can use this plt to save the plot. So we write plt.fig. And now we open parenthesis. And here we have to write the name of the file we want to export. And here I'm going to write my_test.png. So this is the extension and this is the name of the file. And now before exporting this file, I'm going to show you something here. So probably you notice that when we make the plots with pandas, we get these words here that says access subplot and all of this. So we can get rid of these words by using the show method. So we write plt.show() with parenthesis, and if we run this we're going to export this figure and also we're going to get rid of these words. So let's try. So I run this, and as you can see here, all those words disappeared, and also we exported the figure to a PNG file. And now this file should be located in the same folder where you have this Jupyter notebook file.

Okay. I'm going to open that file. But first, I'm going to export the pivot table. So here I copy this DF_pivot and I paste it here. And now to export it, we have to use the to_Excel method. So we write to_Excel. And now I open parenthesis. And here we write the name of the file where we're going to export this pivot table. So in this case, I'm going to name it pivot_table.xlsx. So this is the extension of excel and this is the name of this file. So now I run this, and now the pivot table should be exported.

All right. Now I'm going to open the Excel file and the PNG file we created. So it's here, and here we have the plot we exported and also the pivot table. So as you can see here, the plot looks exactly the same as the one we created here with pandas. And the pivot table is the same. So I'm going to show you how the pivot table looks. And here is the pivot table. And here is the pivot table we exported. I open it in Google Sheets and it looks exactly the same. And that's it. In this video, you learn how to export data frames as well as plots.

Okay. In the previous videos, we learned how to make visualizations such as line plots, bar plots, pie charts, box plots, and more with pandas. But all of them were static visualizations. This is good. But sometimes you want to interact with your visualizations to know the data behind a line plot, for example, or to know the values behind this bar plot. This is not possible with the traditional visualizations that we have in pandas. But there is a library named plotly that helps us make interactive visualizations in Python. Fortunately, this library works good with pandas. And in this video, I'm going to show you how to make interactive visualizations with pandas using plotly under the hood.

Okay. To easily create interactive visualizations, we need to install cufflinks. This is a library that connects pandas and plotly. So, we can create visualizations directly from pandas. In the past, you had to learn workarounds to make them work together. But now it's simpler, and you don't even need to learn the syntax to make visualizations with plotly. But you can use the same syntax we learned so far to make visualizations with pandas with some little changes.

Okay, first we have to install the plotly library. So to do that, here we write pip install plotly. Remember that you need to add this exclamation mark to run commands from Jupyter notebook. So this is a command, and we're running this to install Plotly. So I press Ctrl+Enter to run this. And now let's wait until plotly is installed. So just a couple seconds. Okay, I got the message that plotly was successfully installed. And now that I have plotly installed, I'm going to install another library, which is cufflinks. And as I told you before, this library connects pandas and plotly. So let's install this library. So I write this exclamation mark and then pip install cufflinks. So cufflinks. Now I run this, and here we have to wait a couple seconds. Okay, I got the message that it was successfully installed, and now we can start making interactive visualizations with pandas. So here I'm going to delete this cell, and let's import the libraries that we need to make interactive visualizations. So first we import pandas. So import pandas as pd. Then we import cufflinks. So import cufflinks as cf. Then we have to import display and HTML from IPython.display, and this will help us display the interactive visualizations in Jupyter notebook. So here I write from IPython.display import display, HTML. And finally, I'm going to edit the default configurations of these cufflinks by writing cf.set_config_file. And inside parenthesis, I'm going to change some parameters. First, the sharing parameter, I'm going to set it to public. Then the theme, I'm going to set it to ggplot. And finally, the last parameter is offline. And I'm going to set it to True. By the way, here I'm using the ggplot theme, but you can choose any theme you want. You only have to write here the command cf.get_themes(). And here you will see all the themes. Here I didn't get it because I didn't import cufflinks as cf, that I'm going to import here. So I import this; might take some seconds because it's the first time, and now I imported these libraries and now I run this, and here we get all the themes. So here is ggplot. This is the theme I'm going to use for this video. But you can choose anything you want. You only have to set it and then test it out and see if you like it.

Okay. Now before we make interactive visualizations, I'm going to show you the syntax, which is similar to the static visualizations but with some little changes. So let's say we have a data frame named df. So if we want to make an interactive visualization, we have to write the iplot method, which is .iplot() and then parenthesis. So this one looks similar to the plots we make before for static visualizations, but here is not plot but iplot. So the i is for interactive and well plot for the plotting that we're making, and inside you have to add the parameters, the kind parameter, which is similar to the static visualization, and overall the behavior of this iplot is similar to the plot method, but it will have some changes in the parameters or in the way it behaves. So let's review each visualization and let's see how the iplot method works.

Okay, to make these visualizations, we're going to use the same data set we use for static visualizations. And I have here the code so I don't have to write it again. And this is the name of the data set or the CSV file population_total.csv. And here we have the same data frame. So I'm going to run and I'm going to show you the data frame. So you remember how this one looks. So here we have the country, year, and population, and we're going to make visualizations. But this time, we're going to make interactive visualizations so we can see the data behind the plots.

Okay. Now, I'm going to copy and paste all the code we used in the previous videos to make the pivot table. So, we focus only on making interactive visualizations.

Okay, here I pasted the code, and these are all the operations we made to get this df_pivot data frame. So we read the CSV file, then we drop some null values, make a pivot table with a pivot method, and then select some countries. So here I'm going to show this df_pivot. So I run this one, and here we have this data frame that will help us easily make the interactive visualizations.

Okay. Now before we start, I'm going to delete these two lines of code or two cells. And here I'm going to start with the line plot. So I delete this one too. And here we start with the line plot. But this time it's going to be an interactive line plot. So we will be able to see the data behind the lines. So here let's start by writing the name of the data frame, which is this one. So df_pivot and then we write the name of the method. So .iplot(). So after this we write kind and then we write the type of plot we want to make. So here is line. And now this is everything we need to make this basic line plot. So let's have a look. I'm going to run this one. And here we got this plot, which is a bit big, but anyway, you can see clearly. And the cool thing about this interactive visualization is that if we point and drag this plot, we can see the data year by year. So here you see the population in China in 1870. Here in this blue line, you can see much better the numbers. And here we see the population in India in 1875. So here we can see how it grew over the years. And here we see the population in 2019. So this is cool because you not only see the plot but you can see the data behind, and you get more detail about this line plot.

Okay. Now let's say we want to add some labels here, the X label and the Y label. So we can do this also in interactive visualization. So we can add some parameters to the iplot method. So first let's add the X label. But in this case, the X label that we had here in the line plot, here this is the static visualization we made previously; here the parameter was called xlabel, but now in the interactive visualization this one is called xTitle. So keep that in mind. So I write xTitle in here. I'm going to name it years, and the other is going to be population. So here I write years and then yTitle. In this case, it's going to be population. So now I'm going to add a title, and here the title is just named title, and this one is going to be named Population and I want to add the years. So 1955 to 2020. So here it is, and I run this. And now we see the title and also we see the years in the x-axis and the population in the y-axis. And that's how you make an interactive line plot with pandas and also with plotly.

All right. Before we move on to the next visualization, I want to show you this bar that comes with plotly. And here we have different options. For example, we can download this plot as PNG with this camera icon or we can make zoom. So I click here, and let's see what happens. So now if I make this. So I made a selection, and you see that we have more details in these years for these two countries. So here let's make another selection and we zoom in again. So here to zoom out. Let me see how to do it. And I think I can do this with this zoom out button. So here I zoom out, and we get this line plot, but now it's a bit smaller. And here again zoom out, and we can see all the line plot, but now we have more years. So here we have 1940. And let's zoom in now. And you can see now that it fits much better. But now it's too close. But you can control this much better with this zoom in and zoom out options. But also you can autoscale with this option here. So you get the line plot as it was before. And finally, we got this reset axes. And let's see what this does. And I think this one fixes the axis position or the tick position. But anyway, let's check another one. So the pan option helps us move in this plot. So we can see other parts of the plot, as you can see here. And finally, we have this "produced with plotly". If you click on it, this is going to send you to the plotly website. So let's try it out. Here we see, yeah, the plotly website. And anyway, those are all the options in this plotly bar. And now let's make our next plot. So here I'm going to make a bar plot. So I write bar plot. And now let's make this bar plot. And as you might remember, we have to make some changes to the original data frame to make this bar plot. And to avoid repetition, I'm going to copy and paste those lines of code. So we can focus only on making this interactive bar plot.

Okay, now we have the code that helps us make this data frame df_pivot_2020 that we used to make the bar plot. And now I'm going to run this code. And here the first line of code helps us select only the year 2020, and the second one transposes the data frame, which means that we switch the rows and the columns. So here I'm going to show you the data frame. So I print it, and here this is the data frame. So with this data, we're going to make the bar plot, and in this case, it's going to be an interactive bar plot. So first we write the iplot method. So .iplot(), and inside we write first the kind parameter. This one is equal to bar. And then we're going to give it a color. So I'm going to set it to light green. Then I change the name of the labels, the X and the Y label. So I write xTitle=years and then also the yTitle. Remember that in static visualizations with pandas this is called xlabel, but in interactive this is called xTitle. So here I write yTitle, and in this case, it's going to be the population. So with this, I'm going to run the code, and now let's see the visualization that we got. So this is the bar plot. But now if we select one bar, we get the value that it has. So, for example, here in the traditional or static visualization, we only get this. So we don't know how much population India has in 2020. We can say that the population in India in 2020 is approximately close to 1.4 billion. But we don't know exactly how much is the population. But if now we select the India bar in this interactive visualization, we get that the population is 1.38 billion. And we can see also the other countries. So, for example, United States 331 million and China 1.43 billion and so on. In this case, it doesn't make so much sense to use the zoom in or zoom out. So now let's move on to the next visualization, which is making bar plots grouped by two or more variables.

Okay, here I pasted this code that selects some years. So the years between 1980 to 2020, and we're going to use these years to make multiple bar plots. So here I run. And now I'm going to show you the df_pivot_sample. So it's this one. So now we have multiple years, and this is going to help us make multiple bar plots. So now I write the iplot method. So here I write .iplot(), then the first parameter is kind, and I set it to bar, and let's see how this one looks. So here I have this multiple bar plots or bar plots grouped by five variables. Here we can see in detail the population of each country in the year we want. So let's pick this one. So it's India in 2020, and it's 1.38 billion. Let's pick one more. And, for example, this little one, and it's United States in 1990. The population was 252 million.

Okay. So far we made interactive bar plots and also interactive line plots. And you can argue that maybe it's not so useful to make interactive visualizations in bar plots because the detail is not so much. I mean, you only get one value. But maybe in line plots it's more useful because you get more values behind one line. And the next visualization we're going to see is more relevant when it's interactive.

Okay. The next visualization we're going to see is the pie chart. And here I have this first line of code that helps us change the name of the column 2020. So here I run this one. Now let's make this interactive pie chart, and then I'm going to explain to you why I think this pie chart is more relevant when it's interactive. That one is static. So first I copy this data frame. And now I'm going to use the iplot method again. Here I added the kind parameter. And now I set this equal to pie. Then I'm going to set the values equal to this column that is 2020. So now I write 2020. And now this pie chart is ready. So I'm going to run this one. And here I got an error because I forgot to add the labels. So I'm going to add the labels parameter. So labels= and now I'm going to show the df_pivot_2020 data frame. And here the labels is the country because here we have the names of the elements in this pie chart. So here I write country, and this is also our index. So here country, and now I run this, and I forgot to mention that we have to set this country as a column. So here, unlike the plot method for making pie charts, we need to set this country index to column. So in the plot method, we could make this pie chart with this country as index, but in the iplot method, we need to set this country index as column, and to do that, we have to use the reset_index method. So here I'm going to do this. So I write .reset_index(). And here you see that the country index is not anymore an index but now is a column. So now to save the changes, I'm going to override the values in this data frame. So now I run and let's see the results. I'm going to run this iplot method. So to make the pie chart, so I run this, and now we verify that this labels parameter only accepts columns and not indexes. So keep that in mind. This is a difference between the iplot method and the plot method.

Okay. Now we have the pie chart, and we can see in detail the percentage that each element takes. So, for example, India takes 37.9%, and the US takes 9.1%. So this is helpful. So we can see in detail the percentages. And here in the plot method, it wasn't possible. We could add a parameter that shows the percentages in each portion, but that might require more lines of code. But in this interactive visualization, we got this with the same lines of code. And to make this even much better, we can play with the visualization. So we can hide some elements. For example, I want to hide India. So if I hide India, we only see the United States, China, Indonesia, and Brazil. And also the percentages were calculated again. So all of them sum 100%. Because we're not considering India as an element. So let's show India again. And now let's hide China and let's say also India. And now we see in detail the proportion between these three countries without considering the countries with the largest population. And you can play as you want with this interactive pie chart. And this is something you cannot do with a static pie chart. So this is one of the advantages that the interactive pie chart has over the static pie chart.

Okay. If you like this interactive pie chart, I think you're going to like even more the next visualization.

Okay. The next interactive visualization is box plot. And this interactive box plot is way better than the static box plot because we can see in detail the parameters that the box plot has. For example, the Q1, Q3, the minimum value, the maximum value, and the median. All of

that we cannot see on the static visualization. But on an interactive box plot, we can easily see by just pointing at those values.

Okay. Now let me show you how to make a box plot. So first we write the name of the data frame which is df_pivot. Let me check it here so I don't make a mistake. So yeah, it's df_pivot, but unlike the previous plots, we're going to select only one country. So we make only a single box plot. So here I write df_pivot. And now I'm going to choose the United States country. So here United States, and now the D, and here I'm going to use the Iplot method. So the kind parameter is going to be equal to box, and here I'm going to plot it, and let's see the result. So here we have this box plot, and also if we point at this box plot, we automatically get all the values of this box plot. So we can see that Q1 is 219 million, the Q3 is 323 million, and we also see the maximum and minimum value. In this box plot, we don't have outliers, so we don't see the outliers, but if we had outliers, we will be able to see the values of those outliers too.

Okay. Now to move on to making multiple box plots, I'm going to customize this box plot much better. So here I write color. And now let's change the color to green. And also let's add one more parameter. So also I'm going to add a title for this one. So y_title equal to, and this is going to be population. So now we run. And now we get this box plot. And we see that it is green and also that the y label is population.

Great. Now let's make multiple box plots. And let's do it below. So here I write df_pivot. Now I plot, and here the kind parameter. So again box. Now we write the y_title to add a name for the y-axis. So this one is going to be population. And now let's run this and let's see the results. So here we see the five box plots, and each of them represent one country. So here we have United States, India, and all the countries, and we can see in detail all the values that correspond to these box plots. And this is cool because in the static box plot, you will have to calculate those values on your own, following some equations as I'm going to show you in some videos. But with interactive visualization, you don't have to do that. You only have to point at the box plot, and you immediately get the values you want.

Okay. Now let's make interactive histograms. And to do that, first I'm going to write the name of the data frame, which is df_pivot. Now I select one country. So in this case, United States again. So here I have United States. Now I use the Iplot method. Then I add parenthesis, and the kind is going to be equal to hist for histograms. Now I add a label for the x-axis, and this is x_title, then equal to population. Now I run this, and we get this histogram, and we can see the values inside each bin. So the first one has three values, as you can see there. The second one has four. The third one has four. And the last one has seven values. So here we can control also the number of bins. We only have to add that bins parameter. And now set it equal to a number we want. So now let's say we want three bins. So I write three. Now run. And instead of having four bins as we had before, now we have three bins. So the first one has three, the second one has eight. And now the third one has seven. So we rearrange these bins.

Okay. Now we can add even one more histogram. So we make multiple histograms in this one plot. So here the only thing we have to do is to add one more element to this selection. So I add one square brackets. And here I add one more country. So in this case, let's add Indonesia. So Indonesia here, and with this, we should get two histograms in this plot. So let's run this. And now you see here that we have two histograms. And the cool thing about interactive visualizations is that we can play also with this plot. So here I can hide Indonesia, and also I can hide United States. So we can see the histogram we want. And this is something you could do also with the box plots here. With the multiple box plots, you could hide any value you want; so you only focus on the country you want. And this is very cool because this is not possible with the static visualization.

Finally, we're going to see a plot that is more relevant when it is interactive. And I'm talking about the scatter plot. This one is more relevant because in the static scatter plot, you don't see the values behind all the dots that are in the scatter plot. But in an interactive visualization, you can see the values of all the dots that are in a scatter plot. And now I'm going to show you, so you understand much better this. So first df_pivot. So this data frame, and now I write the Iplot method. So that Iplot, and here the kind parameter as usual, equal to scatter. So double t, and now I add the mode; in this case, it is going to be markers mode. So I run, now I scroll down, and now we have the scatter plot. This one looks like a static scatter plot, but now if we select any dot, we will see in detail the value behind a dot. So, for example, China in 1985 had 1.07 billion population, and we can see all the values behind all the dots in this scatter plot. Again, you can hide any country you want. For example, I only want to show Brazil. So here I have the scatter plot of Brazil. And this scatter plot of population doesn't make so much sense because this follows a pattern. And usually a scatter plot is useful when the data doesn't have a pattern. But still, I think you get the idea of the usefulness of interactive scatter plots.

And that's it. In this video, we learn how to make interactive visualizations with pandas. We learned the advantages that interactive visualizations have over static visualizations. And also we learned the differences and similarities between the Iplot and the plot method.

Welcome to this section. In this section, we're going to learn how the groupby and the aggregate functions work. So we're going to group the data frame we have by some categories, and then we're going to perform some operations like counting the elements, calculating the mean, calculating the sum, and so on. But first, let's check the data frame we're going to use in these videos so we can understand much better the concepts we're going to see in the following videos. So here I'm going to start by importing pandas. So I write import pandas as pd. And now I run this one. And now we have pandas imported. So now I'm going to read the CSV file that we're going to work with. And in this case, it's going to be a car_sales.csv file. And this one shows all the information about car sales and different columns that we're going to analyze. So here I'm going to use the pd.read_csv method. And this one is going to help me read the CSV file. So now I open quotes. And now I write the name of this CSV file, which is car_sales.csv. Okay. Now it's time to read this CSV file. So I run this. And now we have this data frame. And we have different columns about this car sales. And here I'm going to select some columns that we're going to use in the following videos. And I'm going to explain you what type of information each column has while I'm selecting those columns. So here I'm going to write the name of the data frame, which is I don't have the name, so I'm going to assign it now. So in this case, I'm going to set the name df_cars for this data frame. And now I run. So now I have the name of this data frame, which is df_cars. And now I'm going to select the columns I want by writing double square brackets. And here I'm going to list the columns I want. So first I'm going to here print the data frame. And here we have the data frame, and I'm going to select some columns, and the first column I'm going to choose here is the manufacturer column. So here I open quotes and now I paste this manufacturer, and this column has information about the company that created this car. For example, Audi, Volvo, and so on. Okay. Now let's select the next column, and here I'm going to choose the sales_in_thousands column. So this one. So now I copy this and I paste it here. So this column has the sales of the car in thousands, and I suppose that's in dollars. Okay. Now let's select the following column, and this one is vehicle_type, and is this column. So now I copy and paste, and this column has information about the type of this vehicle, and there are two types of vehicle: passenger and car. And to know the types of values that is inside this vehicle_type column, we can use the value_counts method. And in case you forget about this method, we only have to use the name of the data frame. We write this and then value_counts. So now parenthesis, and now I have to add the name of the column. So here I copy and paste it. And after this we run, and we get two types: passenger and car. So more vehicles are passenger type, and a few of them are car type. So now we know about this. And now I delete this. And here I have the column. Yeah, it's here. So next I'm going to choose the price_in_thousands column, which is this one. And this one has the price of each vehicle. So I'm going to copy and paste it here. So I add this new column. And now I'm going to choose another column, which is the engine_size. Each vehicle has different types of engine size. And we're going to see that when we group the engine size later. And here I'm going to copy and paste it here. I write comma, then quotes. And it's here. And now the next column is the horsepower, which is this one. So I add it here. And okay, that's the horsepower. And now the last column is fuel_capacity, which is this one. So that's the last column we're going to choose. And with that, we're done. So this is the last column. Now I'm going to arrange this one here. So I press enter so you can see much better all the columns. And we have seven columns in total. And now I'm going to override the values of this data frame. So I write df_cars equal to this. So now it's there. And we can even make this much better. So you can see, and now it's ready. So now we have this df_cars data frame. And now I run. And let's see how this data frame looks. So here I have the data frame. And we only have seven columns now. And always it's a good idea to select the columns you're going to work with because sometimes the data frame has too many columns. And if you keep all the columns, it's hard to focus on the goal of your project. So it's a good idea to select all the columns you want before you start working, as I did. And now before we finish this video, I'm going to show you the unique values inside all the columns we have. And to do that, I'm going to use a method we learned in this course, which is nunique. So I write nunique, now parenthesis. And this is going to help me see all the unique values inside the columns. So here I run. And now we see that all the unique values. And for example, the vehicle_type has only two unique values, as we've seen before: the passenger and the car type. And the manufacturer has 30 unique values. So probably 30 companies. And this is important because when we group values in a data frame, we have to make some operations in those groups. And if we have more unique values, that means that we're going to have more categories, so more groups. And the more groups we have, the more difficult to analyze it.

Okay, that's it for this video. In this video, we reviewed this data frame and we got familiar with some columns that this data frame has. So now check this data frame on your own to help yourself understand much better the following videos.

All right. In this video, we're going to learn how the agg method works, and also we're going to use some aggregate functions in this method. So we want to start by calculating the sum of columns of this data frame that we had before. So in case you didn't watch the previous video, we created this df_cars data frame with cars information here, car sales information. So here we selected seven columns, and now I'm going to use this data frame. So now I'm going to calculate the sum of columns, and to do that, I only write the agg method. So I write agg, and then open parenthesis, and inside we have to write the function we want to apply to this data frame. So now I want to sum the values inside the column. So I write sum, and now I run this code, and we see the result. And here in the first column that is called manufacturer, we got this weird result. And this is because this manufacturer column has text data. And since it's text data here, we concatenated the strings. So we didn't sum the values, but concatenated the text. And we didn't want to do this, but we only wanted to sum the values. But for some reason, the agg method didn't completely understand what we want to do, and it concatenated the strings. But it doesn't matter. Let's check the next column. And this column has the sales in thousands. And here we got the sum of the values inside this column. And that's fine. So now the following column is the vehicle_type. And here this one has text data. And this is why we concatenated the values here. But anyway, it's only text data that we can exclude later. And here the next column, we have the total sum, and also in the other columns. So now this is how you sum the values inside a column. And now I'm going to calculate the mean of numeric columns. So maybe we're going to get again the text data like we got here in the manufacturer column. So let's see, I'm going to write here the name of the data frame, which is df_cars, you know, agg, and this parenthesis. So now we write the function we want to apply to this data frame. In this case, is the mean function. And here I run. And now there is a difference between these two functions because here we got text data inside the manufacturer column and vehicle_type. But here we didn't get it because here the mean also includes division, and probably division is not possible in text data. So this is why here it was omitted. So now we only got the columns with numeric data. For example, the sales_in_thousands. And we see that the sales is greater than the price_in_thousands. And then we have the average engine_size, which is three. And then the horsepower and the average fuel_capacity.

Okay. Now let's say that you want to apply not only the sum function or the mean function but both. So you want to apply both functions to this data frame. In that case, we can add them to a list and then use the agg method. So I'm going to do that here. We're going to calculate the mean and the count values. So here we're going to apply two functions. So first I write df_cars, then agg, then parenthesis, and now we open square brackets to indicate that we want to start a list. And inside we write the two functions that we want to apply. So in this case, I write mean, and the second is going to be count. So here I run, and we get two rows, and we get the seven columns we selected before. And here we see that in the first row that has count as index, all the values had numeric data. And in the second row that has mean as index, we have some null data because here we try to calculate the mean of text data, for example, in the manufacturer column, and that's not possible. So that's why we got this NaN, that stands for null value, and well, we didn't calculate that data. If you want, you can modify and type any other function. So, for example, the sum function instead of the count. So here I write sum, and now we get these values, and here we concatenate strings again, and anyway, you can include any function you want.

Okay. Now let's apply different aggregations per column. So we're going to calculate the mean and also the sum of the columns sales_in_thousands and also calculate the sum and max of the price_in_thousands column. So here I'm going to write the name of the data frame, and I write df_cars, then agg, and parenthesis. So now this has to be in a shape of a dictionary. So here we're going to use the lists again, but now to indicate the column that we want to work with, we have to open a dictionary and write it as a key. So, for example, here I open quotes, and now I have to write the name of the column I want to work with. So in this case, it's going to be the sales_in_thousands column, which is here. So I don't have it. So I'm going to print it here. df_cars. So I can copy and paste it. So I choose this sales_in_thousands. So I copy this one and now I paste it. So this is my first key. Now I have to write colon, and here the value. So the value is going to be in list format because we want to apply multiple functions. And now I'm going to delete this. And here I'm going to write the functions that we want to apply. So in this case, for the sales_in_thousands, we apply this first sum function, you know, I add it, and then the mean. So sum and mean. So there it is, and now we add the second pair of key and values. So comma, enter, and now we write the second pair. So here the key is going to be the second column. So price_in_thousands, and now I paste it here, and here we have to specify the aggregate functions we want to apply to this column. So here I have the square brackets that indicates this is a list, and now I write the first function, which is sum, and the second one is going to be max. So here I paste this one, then comma, and here max. Okay, now this is done, and what we're doing here is to apply the sum and mean functions to this sales_in_thousands column and then to apply this sum and max function to this price_in_thousands column. So we're applying different aggregation functions per column. So now I run, and now we see the results. So here we see three rows, and we see also null values because here we don't have a mean for the price_in_thousands. So you see here that we only have sum and max. And for example, here for the sales_in_thousands, we don't have the max function. So we only have sum and mean. But the two columns have one common function, which is the sum. And here we have values for both of them. And that's how you apply different aggregation functions to a column.

Now let's see how to aggregate over the columns. So we're going to make operations along the columns. So here I'm going to sum two columns, and in this case, I'm going to select these two columns first. So I write df_cars, and now let's select because if we don't select these two columns, the aggregation function we're going to apply is going to be applied to all the columns, and we don't want that. We only want it to be applied to two columns. So here I select those two columns, and the first one is going to be sales_in_thousands, and the second one is going to be price_in_thousands. So I copy and paste it, and now, well, we have these two columns, and now I'm going to apply the agg, which is the sum function, and this indicates that I want to sum the sales_in_thousands column and also the price_in_thousands column. So I want to sum these two columns, and now to specify that I want to make an operation along the columns, I have to add the axis parameter. So I write axis, and here I write 1, which means columns. And now I'm going to run this one, and

See the results. So here we see that we have a series, and there are 156 rows, actually 157, because here the zero counts. And now this is the same number of rows that the original data frame has. So I'm going to show you; here I print the DF_cars, and here we see 157. This means that we sum the values inside these columns. So, we sum each row inside these two columns.

And now you might be wondering why I didn't select each column and sum the two columns as we did in previous videos. So, I'm going to show you what I'm talking about. So, to sum these two columns, we could do this: So, DF_CARS and then select one column and then uh write this plus operator and then write the other column, to this price and thousand and this. So if we do this and I run, we get the same result here. I did this because I wanted to show you how to aggregate over the columns using the AG method. But also there is another reason why I did this: is because both approaches behave differently.

So here, uh in the first approach, you see that we have a value, a numeric value in the index two, and here in the same row we have a new value, and this is kind of weird because we should have the same result, but we have a different result. So I'm going to show you the data frame here and explain you what's going on. So here in index two, we see that we have a numeric data for the sales in thousand column, but for the price in thousand column we have a new value here: So NaN. So here in our first approach, uh here with the ag method, this ag method uh takes this null value as if it was zero. So here we got 14, and here this NaN is like zero. So we sum these two values, and we got 14, of course, but the second approach, so summing these two columns, making the selection is different. So the way it behaves is that if one value is null, the sum of two values is going to be null too, even if one value has numeric data. So that's how both behave, and you have to keep that in mind.

Okay, before we finish this video, I'm going to show you how to aggregate different functions and rename the index of the resulting data frame. So here I'm going to write again the DF_CARS data frame and here AG. So I'm going to use this method, and now here we have to use a different format compared to those we've used so far, and in this case we have to use tuples, which are represented with the parenthesis. So here the first element is going to be the name of the column, and the second element is going to be the aggregate function, and outside the tuple we write the name of the index that we want to set. So here I'm going to show you; first I open the tuple, and now I write the name of the column. So here I'm going to choose the sales in thousand. So I just copy this one and I paste it here. So this is my first element, and now I add the function I want to apply. So in this case, the sum function, and now outside this tuple I'm going to set it equal to the name of the index I want to set. So in this case, I'm not going to get this sum function as index, but I'm going to get the name I set here. So for example, if I want to set it equal to X, I'm going to get the X in the index. So here I'm going to show you an example here. Instead of getting the sum, the min and the max in the index, I'm going to get the name I set in the aggame, which I'm setting here. So this one, I'm going to get this x name. And now I'm going to apply another aggregate function in the price in thousands column. So here I just copy this, add a comma, and here I paste it. So in this case, it's going to be a y, a value or name, and here I'm going to write the price in thousand, which is here. So I copy. Now I paste it, and I'm going to use the same function. So you see how it works. So here I run this, and we get this uh little data frame, and here in the name of the indexes we get x, this one, and y, so this one, so we rename the name of the indexes in this data frame. Here you can see that we have null values in the price in thousand and also in the sales in thousand, and it looks kind of weird because here we know that the sales in thousand is 8,000 320 for X and Y because X and Y represent the sum function, so it should be the same, but here for some reason it doesn't recognize that X and Y belong to the same function. So now you might think this is not so useful, but in the following videos I'm going to show you how this is going to work with the group by method. This gets more relevant when we use both the group by method and the AG method. And that's it. In this video, we'll learn how the AG method works in pandas, and also we'll learn how to use some aggregate functions.

All right. In this video, we're going to learn how the split apply combine strategy works. And this strategy helps us group values in a data frame. So we build categories, and we can apply some aggregate function. So we see the values of those categories. So let's start with the split, and here let's split the data into separate groups. So here we're going to work with the vehicle type column, and let's see the values that this column has. So I think we've seen this in the previous video, but in case you forgot, here let's see the values. So I write vehicle type, and now I'm going to use the value count method to see all the values inside this one. So we see that the vehicle_type has two uh values, and the first one is the passenger and the second one is the car. So here we can split that data frame into these two categories. So the first category is going to be the passenger category. So here we make a filter, and here I write df_cars, then open square brackets, and then I write the name of the column. So this one, and now I'm going to write a condition: So this equal to and here passenger. So I copy and paste this. So now here this is my condition, and I'm going to assign this equal to the passenger uh let's call it filter. So this is my variable. Now I'm going to do the same, but now for the car type. So instead of uh passenger filter, I'm going to name it car filter. And now here this remains the same. And now this condition should be equal to car. So this is what we get. And now I'm going to comment this out. And now I assign these two conditions to the variables. So now I run. And here we have these two variables. I'm going to show you one of them. So here I have car filter, and here we see that we got boolean uh values in this series, and we're going to use that in the next step. So now I delete, and now the next step is the apply, and here we apply an operation or we apply an aggregate function. So we get the value of the category that will show in a data frame later, but here we make that calculation. So here I'm going to filter uh the data frame based on these conditions. So here I write df_cars, then open square brackets, and inside we write the condition. So here is the passenger filter first. And now I'm going to show you how it looks. So this is the filter that we made. So here we see that we only got the vehicle type equal to passengers. So if we go to vehicle type, we see that all of them are passenger. And now I'm going to choose a column to make the aggregate function. So here I'm going to choose the sales in thousands column. So this one. So now I select this, and here we get only this column. And now I have to choose an aggregate function. And in this case, I'm going to choose the mean. So I write that mean and then I open parenthesis. And now I calculate the mean of this column. So if I run this, we see the mean of the sales in thousands, but only for the passenger filter. And here I'm going to assign this to a variable. So I'm going to name it car or actually passenger because this is the passenger uh filter passenger let's say average. So this is my first variable. And now let's create the second one. So here I write df_cars, then open the square brackets. And now I select this car filter. So I paste it here. And now I'm going to choose the same column because we have to make the the operation in the same column. So it's this one. And now I paste it here. And the operation is going to be the mean. So mean and parenthesis. And now I'm going to assign this to a variable. And in this case, it's going to be named car uh average. And here I forgot to add the quotes. So here we have the quotes. And now this is ready. So here I run this code, and we got this. And now the last step of this strategy is combine. So we're going to combine the values of the ply step and the split step, and we're going to make a data frame. So here to make a data frame, I'm going to use the data frame method. So pd.DataFrame and now I open parenthesis, and to create a data frame I'm going to use a dictionary. So I open this curly braces. Now I write the name of the key, and in this case the name of the key is going to be that type we selected: So vehicle type. So I'm going to name this first column as vehicle type. And then I open a square brackets. And here we write the types of vehicles. So the first one is car and the second one is passengers. So here passenger. And now we add a second pair of key value. So here is going to be the sales in thousand. Then I write colon, and now I open this uh square brackets, and inside we write these two variables that we created here. So the passenger average and also the car_average. So I'm going to set it in the same order we we put it here. So first car and second passenger. So here first car and second passenger. So now this is ready. And what we're showing in this data frame is the categories here, the categories that we selected for these groups: So car and passengers, and also we're showing the values of the aggregate function that we're applying to those categories. So in this case, the average function. So now I press control enter, and now we see the results, and here we see the vehicle type. So here the car type has 80.62 62 in average in sales, and the passenger has uh 43.23 23 in average in the sales in thousands column, and that's how you group values in a pandas data frame, and also that's how you apply some aggregate functions, and I know these are too many steps, but fortunately there is an easier way to do this with less lines of code, and is by using the group by method, and with that we're going to do this with one line of code and faster, so I'm going to show you how the group by method works in the following video.

All right, in this video we're going to learn how the group by method works. And the group by method allows us group data into categories, and then we can apply some aggregate functions. So we see the values there. So for example, the sum, the average, or we can count the values inside each category. So we're going to start by grouping vehicle type and then calculate the mean. And this is the same we did before here uh in the previous video using the split apply combine strategy. In this case we used a lot of lines of code, but in this case we're going to use an easy approach with the group by method, and we're going to group this category vehicle_type, and then we're going to compare the results. So here let's start by writing the name of the data frame, which is df_cars, then groupby, and then here I write the name of the columns. So I'm going to use the vehicle type. So this one, now I paste it here, and now I apply the aggregate function. In this case, we want to calculate the mean. So we write mean then parenthesis, and that's everything we need to do. So let's review the syntax: First the name of the data frame, followed by the group by method, and inside we write the name of the column in which we want to make the groups, and then we apply the function, and that's everything we have to do. Now we run this, and here we got this data frame. And here we see two categories. So in the vehicle type there are two categories. Uh the first is car and the second is passenger. We only see columns that have numeric data. So here we calculated the mean or the average of each column. And now we can compare the result of this data frame with this uh little data frame that we got here using the split apply combine strategy. So here we only calculated the sales in thousands. So let's compare only that column. So here I have this data frame, and if we compare the columns we see that the value in car is 80.62, and here is the same, and for passenger is 43.23, and yeah it's the same in the sales and thousands column. So here we did the same thing, but now we use the group by method, and as you can see this is a shorter method because we didn't need to split apply and combine, but we could uh avoid those steps and only do all of that with the group by method. And here there is a little difference: Here we have the vehicle type in the index, but here we have the vehicle type in the column. So we can set this as index if you want here. So set_index and then we set this vehicle type as index. So both are the same now. Or you can here add a parameter as_index and here set it to false. So with this the vehicle type index will be a column. So now I run this, and here we see that this vehicle type isn't an index anymore, but now is a column, and that's how the group by method works. Now I'm going to show you a different example so we can understand much better this method. So now let's group by manufacturer. So I write df_cars.groupby and then inside parenthesis I write this name of the column. So here I paste it, and now let's run this without applying an aggregate function like in the first example. So let's see what happens. So here I only group by manufacturer, and we got this uh these words that says that this is a data frame group by object. So here when we use the group by method we get a group by object. So this is the result of running this code. This actually is not the final result that we're going to use, but we can use this object to get to other values we want. For example, we can get the name of the groups of this uh group by object. But first, I'm going to set this uh variable. So, I'm going to name it groupby_g. So, object. So, now I run this. And now this variable has this group by object. Now I'm going to show you this object. So I print this, and here is the same uh object, and now I'm going to get to the groups attribute. So we have to write the name of this object followed by that, and here groups. So this groups attribute allows us to get all the categories that are inside this group by object. So here I run, and this has the form of a dictionary because we see here the curly braces, and now we have all the elements. So here in the keys of this dictionary we have the categories or the names of the groups, and in the values we have the indexes. So for example, the category Audi has these indexes four, five and six. So this means that the four, five and six index has this all the category, and only those rows, all the rows have other category. So now this is the groups, but now since this is a dictionary we can get the name of the groups. So here I can use the keys method, and with that we get only the keys of this dictionary. So here I run. Now we only got the keys of the dictionary. So only the name of the manufacturers. So here there is Audi, Ford, Volvo and so on.

Okay. Now let's use the get_group method to get a specific group within the manufacturer column. So here I'm going to reuse this object. So I copy and paste. And now I'm going to use the get_group method. So I write get_group now parenthesis, and here inside we have to write the name of the manufacturer we want to select. So in this case let's say I want the Ford. So I write Ford. Now I paste it here. Now I run. And here we got the selection of Ford in the manufacturer column. So all the values inside manufacturer are Ford. So here we see only Ford. And if you check here the groups attribute we'll see that the indexes are the same. So let me see here I print now. And now let's look the word Ford. So it must be here. So it's BMW, and here is Ford. So we see 46 47 until 56, and these are the same indexes we get here. So from 46 to 56. So this is the selection that corresponds to the Ford category.

Okay, that's enough for the group by object. And now let's apply some aggregate functions. So now I'm going to uh here use the name of the data frame. So I write df_cars and now I'm going to group by the manufacturer. So groupby here here parenthesis. Now the name of the column here. So I paste it. And now I'm going to apply the mean uh function. So here the mean method. Sorry, here's the mean method. And now I run this. And here we get the uh data frame grouped by the names of the manufacturer. And here we have all the columns with numeric data. So we calculated it for each group in the manufacturer column. Here we could set this uh index as column as I showed you before. But this is enough. So now let's see the next example. And here we have to group by manufacturer and then calculate the sum. So this is similar. So you can try yourself, and I'm going to do it now. So here I write DF_CARS then groupby now parenthesis and now the name of the column. So here again this manufacturer which previously was a column so I paste it here. Now we have to write the uh function we want to apply. So sum parenthesis and this is enough. So I run and now we got a similar result that we got with the average, but in this case we grouped by manufacturer and then we sum the values inside each category.

Okay, finally let's see these three last examples, and all of them involve the count function. So in this case we're going to group by vehicle type and then count the values. So we write again the name of the data frame. So df_cars then we groupby vehicle type. So I write groupby then quotes and then we write this column. So it's here. And now I'm going to count the values inside this column. So we use the count method. And now I run. And here we have only two categories because as we've seen before this uh vehicle type there are only the car type and also the passenger type. In here we counted uh the values inside each column. So for example there are 41 values that correspond to the manufacturer inside the car type. And we see that we got the same number of element for sales in thousands, but for the price in thousands and engine size we got 40. So there is one element less. And this happens because some columns have some null values. So if a column has a null value we won't count it because they aren't actually values but they are NaN. If we want to count those NaN values, we could fill those null values with a number or with a text, and only in that case we will count those values. But as long as they are NaN we will count those values with a count method. So keep that in mind. Now let's

verify which columns have these null values, and we can do that by using the is null method and the sum method. So here I write df_cars, then is null (parenthesis), and then we sum here all the null values. So here we see that only the price in thousands, the engine size, the horsepower, and the fuel capacity have null values. And this makes sense because those columns have less values than the manufacturer and sales in thousand columns. And here we see that those columns, manufacturer and sales in thousands, don't have any null values. So these results make sense.

But what if you want to group by and then count the values, but then show the null values? Well, if you want to do that, you only have to add the dropna parameter inside the group by method. And I'm going to show you how to do that in this last uh example. So in this case, we're going to group by engine size. So here I write df_cars, then groupby(now parenthesis). And now we write the name of the column. So here is the engine size. And now I add a new parameter, so dropna, and we set it equal to false. So this dropna parameter controls if we want to drop the null values or not. So if this is set equal to true, this means that we want to drop the null values, and we're not going to count it if we apply the count method as we're going to do in this example. So if we drop the null values, they are not going to be counted. But if we u set this equal to false, so dropna equal to false, we're going to consider and include these null values in the group that we're going to make. And then we're going to be able to count these null values. So let's have a look at how it works. So now I run this, and here we see the result. So we see that there are many engine size types. So we have many groups, so one, 1.5, 1.6, and so on. And if we scroll all the way down, we will see that the last row is the NaN. So here we have the null values. And here we see the columns that have null values. And this dropna parameter might come in handy whenever you want to consider the null values in whatever operation you want to perform in the groups you created. And that's it. In this video, we learn how the group by method works in pandas.

Okay. In the previous videos, we learn how the group by and the aggs. And in this video, we're going to use both of them to perform more complex operations. Okay. Now, in this first example, we're going to find the minimum and maximum value on each column. So here, first we're going to group by vehicle type. And to do that, we write first the name of the data frame, so DF_cars. And then we group by the vehicle type. So, here (parenthesis). And now the name of the uh data frame, which is vehicle and then underscore type. And in the previous videos, we learned that if we want to get the minimum or maximum value, we have to use the min method. So let's try it out. So let's say we want the minimum value. So we use that min method. And here I run this, and we get the minimum value of each column. And if we want the maximum value, we only have to modify this and write max. So here write max, and we get the maximum value. That's a common approach to get the minimum and maximum value. But now I'm going to show you how to calculate both of them at the same time using the agg method. So here, instead of using the max method, I write agg, and now here inside I open square brackets, so we create a list, and here we have to write all the functions that we want to apply to this group. So now I write min inside quotes and now max, so max, and now this is ready. So if I run this, we get the minimum and maximum value of all the columns. And here we can add as many aggregate functions we want. And also we can customize the name of the columns, and also we can apply different functions to many columns. And I'm going to show you how to do this in the next example.

So here we're going to find the minimum engine size and the maximum horsepower. And to do that, first we uh group by vehicle type. So I copy and paste this one. And after this, we use the agg method. But now instead of writing the minimum and maximum functions inside square brackets, we'll use tuples this time. And to create tuples, we have to write (parenthesis). And here the first element is going to be engine size. So here I copy this column name and now I paste it, and after this I write the function I want to apply. So in this case, I want to find the minimum value, and here I write min. And now I'm going to set the name of this uh column that we're going to create here, and let's call it uh min_engine_size. So this name that I'm creating here is supposed to be instead of this uh column name. So instead of getting this default column name, we're going to get this customized column name. So now let's create a second uh tuple. So here I open (parenthesis). And now the first element is going to be uh this column horsepower. And in this case, I want to calculate the max function. So here I write max. And now here equal to and now I assign the name for this tuple and is max_horsepower, and this is ready. So let's review this uh syntax one more time. So first we group as we've done so many times in this course. We group by vehicle type, and after this we um use the agg method. And this agg method has a different syntax. And here we use tuples. And inside these tuples, we have to write first the name of the column in which we want to perform an aggregate function. And then the second element is the aggregate function we want to perform. So here engine size and min, and then we have to assign a name for this column. And in this case, we assign these two names. And that's everything you have to do. Once this is ready, we can run this. And here are the results. So here we got the vehicle type in the index because we grouped by vehicle type. And then we have here two columns. So first is the min_engine_size. And this is because here we set this name. And this column has the minimum value, so two and one. And if we check the previous data frame we created here, we can verify that those values uh belong to this um engine size column. So it's this min. So here we have two and one, and two corresponds to car and one to passenger. So it's the same here we have min_engine_size instead of engine size. And here is the minimum value. So the minimum for car and the minimum for passenger, and the same goes for horsepower. So here we calculated this time the maximum value. So here the minimum and here the maximum, and we can verify that those values are correct because here in horsepower here we have max, and for car it says 300 and for passenger it says uh 450, and these are the same values. So what we did here is to perform some specific aggregate functions to a particular column, so the minimum for the engine size and the maximum for the horsepower. So this is much better instead of just applying the aggregate functions to all the columns. So here we can focus only on the columns we want and also set customized column names.

Okay. And now to understand all the concepts we learned, let's solve this example. And this time we have to calculate the sum of sales in thousands and the mean of price in thousands. So it's basically something similar to the previous example, but now new columns and also new aggregate functions. So here I'm going to group first by the manufacturer. I didn't write here, but we have to group by the manufacturer this time. So I copy this. And now instead of writing vehicle type, I write manufacturer. So here to write it correctly, I'm going to use the columns attribute. So here I write cars, then columns. And now I run this. And now I copy manufacturer just to make sure I write it correctly. So I paste it here. And now I use the agg method, so agg(parenthesis), and then we create the tuples. So the first tuple will be um this one with the sales in thousands column. So we paste it, and here the aggregate function we want to perform is the sum. So now we set the name for this first tuple and is sum_sales. Then comma, enter. And here we create our second tuple. This time the column is price in thousands. So here price in thousands, and the aggregate function is the mean. So mean. Now we create the column and here is mean_price. So this is the name of the column. Now equal to and now this is ready. So we have here the uh the group object and then here is the agg method and here the syntax that you already know, and now I'm going to run this to see the result, and here we have a data frame, and in this data frame we grouped the data by manufacturer and then we sum the values in the sales column and then we calculated the mean in the price column. So here we have all the values for all the manufacturers.

Okay. Now I want to show you what's under the hood. So now we're going to do the same thing we got in this data frame but without using the agg method. So only with the group by and aggregate function. So you understand much better what we're doing now. So here we're going to calculate the sales in thousand, the sum, so the first column and only with the group by method. So here I copy, I paste it here. And now we sum the values in these groups. So we use the sum method. And now we select one column. So here first I'm going to run, and I'm going to show you here. If we sum, we get the sum of all the columns and for all these uh manufacturers. But now we only want one column. So we only want this uh sales. So here I copy and paste this one. So here I paste it. And now I run this, and let's compare the results of any value here. So let's take Audi. In this case, the value is 40. And now let's check here uh the sum_sales. So for Audi is 40. So this is the same, but here we didn't use the agg method but only the sum aggregate function, and also we select only the sales in thousands, and basically this is the same, here we just u use the aggregate function but without the agg. For mean price, we can do the same here. So I'm going to copy and paste because it's absolutely the same. So here instead of uh using the sum method, we only write mean. And now instead of sales in thousands, we have to use the price in thousands. So now I paste this. And now I comment this out. I run. And let's check now BMW. So this is 33. And let's check here BMW, and 33. So now it's correct. And I did this because I wanted to show you what's under the hood. So you can achieve all of this without using the agg method. But you should use this agg method to simplify things. But it's always good to know what's behind this shortcut. So here I showed you, and now you know how to do it with this group by and also how to do it with the agg method. And that's it. In this video, we learn how to use the group by and agg method together.

Okay. So far we learned how to apply a built-in aggregate function to our group by object. But in this video, we're going to learn how to apply our own function using the lambda function. So here we have an example of the built-in function. For example, here is a sum function. So here we're grouping by manufacturer and then summing the values inside these groups. So this is what we learned so far. So here we have all the values, and we have the sum in each column. Okay, this is what we learned so far. We have here the group by object, and now here we apply a built-in aggregate function, and here we got the sum of the manufacturers and in these columns. So here now we can apply our own function, and now to make an example, let's say we don't want to get the sales in thousands, but now we want to show the real value here. So instead of 79 for this uh value, we show uh 79,000. So all of them will be in the real value. So to do that, we have to multiply each value by 1,000. And we can do that by using the lambda function. And I'm going to show you how to do it here. So first I'm going to uh minimize this. And now I'm going to show you how the lambda function works in case you don't remember or you didn't check the previous videos. So basically, to create a lambda function, we only write lambda, and then we have to write the input. So here we write any random variable. So I write x. This is my input. And now the output. So let's say we want to output x + one. Now I assign a name for this function. So I write here uh sum_one. So here now if we apply this function, we write here sum_1 and then we write any value we want. So let's write one, and here we should get two because 1 + 1 is two. So now we run, and here we got two. So this lambda function is a one-liner function that we can uh create, and this is like a short version of the traditional function that we learned in the Python crash course, and the lambda function is preferred when we want to apply a customized function to a pandas data frame. So now now that I explained you this, I delete this, and now I'm going to do what I told you before. So here we're going to calculate the values in thousand. So I'm going to multiply this by 1,000. And here I start by writing the name of the data frame, so df_cars, and then we group by the manufacturers. So here I copy manufacturer and then I paste it. So I have this group by object. Now we sum the values, and after this we use the apply method, so apply, and inside (parenthesis) we write the lambda function. So lambda, my input is x, and my output is x * 1,000. So this is input and output. Now if I run this, we get the real values of this data. So for example, this one is 79,000, and it's not 79 anymore. So here this sales column is not in thousands anymore, but it's only sales because now we're showing the real data of this column. And now you can see that all the columns were multiplied by 1,000, but not all of them have this in thousands in their columns. So we are only interested in these two first columns. So now let's select only these two first columns with double square brackets. And now I write the name of these columns. So here price in thousand and now sales in thousand. So we only want to show those two columns because those are the actual columns that are in thousands. And that's how you create a lambda function and apply it to a group by object.

Okay. Now to understand this much better, let's see another example. This time we're going to subtract values from the mean of each group. So we're going to group again by manufacturer. And now we're going to create a lambda function that helps us subtract the values from the mean. So here I paste this group by object. And now I write apply, so that apply, and inside the lambda function. So now the input, so an x variable, and now x, which represents an element of my data frame, and now minus x.mean(). So here with (parenthesis), and before I do this, I'm going to show you the data frame again so you can see what the x and the x.mean() represents in this data frame. So the x represents a value inside this data frame, for example, this 16, this 39, or this 14. But the x.mean() represents the average values that are in these groups. So here we're grouped by manufacturer. So we get here, Acura, Audi, BMW. So here we calculate the mean of each value. So here, for example, Acura, the mean is I don't know, 10. And after we do that, we uh subtract 16 - 10. So we have to select the right average for each manufacturer, and then we subtract each value from the mean of each manufacturer. So here we're not calculating the mean of the whole column, but we're calculating the mean of each group. Okay. Now that this is clear, I delete this. And now I run, and here we have this result. So for example, in the engine size column, we got the first value which is -1, and this indicates that this value is under the mean. So this is less than the mean of the engine size. But here we have a 27, and this indicates that this value is over the mean. And as you can see, this function that we created shows us the values that are under the mean and over the mean. So we can recognize which are good values.

Okay, before finishing this video, I want to show you what's under the hood of this lambda function. So here I print the data frame df_cars. And well, we have these values. This is the original data frame. And now I'm going to calculate the mean of the category manufacturer. So we're going to group by manufacturer and then calculate the mean. So I'm going to do that just to show you what's under the hood of this lambda function we created. So here I write df_cars and then we group by the manufacturer. So here I copy and paste this, and after this I'm going to calculate the mean. So here we have the groups by manufacturer and then we calculate the mean. So now we have these values, and now let's uh make this operation. So let's subtract x - x.mean(). So this x first means the value inside this uh data frame. So for example, 16, and this x.mean() is the average in each group. So for this example, 16, the average is the average of Acura. So here let's check the average for this first example. So Acura, the average is 19, and if we subtract 16 - 19, we get a value close to -3. So now let's verify. So here in this uh data frame that we got after applying this lambda function, we see that in the sales in thousands column in the first row we have this -2.83, which is close to -3, and well this is great. So we verify that this is correct. So we only uh subtract these values from the mean. And now let's check one more example. So here let's pick Audi. In this case, is 20.39. And now let's check the average of Audi. So in the sales in thousands columns. So here all these 13.5, and 20 - 13 is 7. So now let's verify in this um data frame. So we check here the row with the index 4. And now index 4. And now we uh look for the sales in thousands. So here is 6.87, and this value is close to seven. So this is correct. And with this, I wanted to show you that aggregate function like the mean, when they are applied inside the lambda function, we apply this mean to the groups that we get after using the group by method. In this case, the group by method are grouped by manufacturer. So, we calculated the mean over the manufacturer groups. And that's it. In this video, we learned how the lambda function works, and we created more complex functions using the apply method and the lambda function.

Okay. In this video, we're going to learn how to filter values based on an aggregate function. And to do that, first we have to create a group by object. So here let's create the group by object. So I write df_cars, then the group by method. So I write groupby, then (parenthesis), and here I write the name of the column. And now I'm going to get the name of the columns by using the columns method. So I write it correctly. So manufacturer, I'm going to do this example with the manufacturer column. And now I use the filter method. So the filter method helps us filter values based on an aggregate function when we use it with this group by method. So here we only have to write filter and then (parenthesis). So now inside this (parenthesis), we have to specify a function, and we can create our own function and specify the condition that we want to

set for this filter. So let's do this.

So I'm going to create a function using this def keyword and the name of my function is going to be filter_funk then I set the parameter x then colon and we're going to return um this condition. So first x and now we select a column. So this x parameter represents a data frame and if we open this square brackets we're selecting a column. So here I want to select the column sales in thousands. So I'm going to create a condition based on the sales in thousands column. So I paste this and here we're going to compare the sum in the sales of these manufacturer groups. And to do that we write the sum method. And now we compare this with a value which is going to be the mean of the sales in thousand. And to calculate the mean here I'm going to use the mean method. So here I run this and now we see that the mean of the sales in thousands is 52. So this is the mean of the whole column. So now I use 52. And now this filter function is going to filter groups based on the sum of the sales in thousands. So this has to be greater than 52. So if we have a manufacturer with a total sales less than 52 that company or manufacturer is going to be filtered out.

So now what we have to do is to copy the name of the function and put it inside parenthesis. So here in my filter method we specify that we want to apply that function. So in this function we have this condition and we're going to filter based on that condition. So now I'm going to uh set a name for this data frame because what we get here is a data frame with this filter. So only the values that pass this filter are going to be in this new data frame. And I'm going to set this data frame uh name. So I write df_filter. And now this is my new data frame. So here I'm going to show you this data frame. And now everything is ready. We have the function, we have the group by object and we use here the filter method. So that's everything we need. So now I show you this new data frame and it's here. So I scroll down and all the values here passed this filter. So all the manufacturers passed this filter. So that means that the sum of the sales is greater than 52. So, Acura passed, Volvo passed, and so on. Okay, great.

Now, I'm going to show you which manufacturers were filtered out. And to do that, I'm going to use the same group by object. So, here I copy this. And now I paste it. So, this is my group by object. And now to calculate which manufacturers were filtered out, I'm going to use the sum uh aggregate function. And now I run this. So here we get the sum of these values and we get the columns and we grouped by manufacturer. So here I'm going to select only the sales in thousands because this is the column we use in this function. So here I select only this uh column and here we have this column and now I'm going to sort the values ascending. So I write sort_values and parenthesis. So we use this method and here we get the values of this sum in the sales column and now is ascending. And here we see that the first six manufacturers have total sales less than 52. So here to see it much better I'm going to use head and then six. So only we see that six first values. So here Porsche, uh, Jawward, SA, Infiniti, Audi, and VMW were filtered out.

Okay. Now let's see how many values these six manufacturer had in the original data frame. And to do that, I'm going to use the isin method. So here I'm going to um use the same column. So here I write manufacturer. So I copy this and now I paste it. And after this I use the isin method. So now parenthesis and inside I write the name of these six manufacturers. Okay, I pasted the six manufacturers and now I'm going to filter this data frame. So I copy the name of that data frame and now I add these square brackets. So now I run this to see this data frame. Okay, now this data frame only contains cars that belong to these six manufacturers. So we see Audi, BMW and so on. Now let's see the shape of this data frame by using the shape uh attribute. So here I write that shape and now I run and we see that the shape of this uh data frame is 13, 7. The first is the number of rows and the second is the number of columns. Great.

Now let's compare the shape of the original data frame and the shape of the df_filter data frame to see how many rows we lost in the filter. So here I write df_cars and now the shape attribute and here we see that in the original data frame we had 157 rows and now let's find out how many rows we have in the filter data frame. So this one I copy this and now let's paste it here. Now I use the print function to print both values and it's here. So I run and we see that in the original data frame we had 157 rows but in the filter data frame we have 144. So here we lost 13 values or 13 rows and those 13 rows belong to these six manufacturers because those manufacturers didn't pass this filter that we created in the function. So those have total sales less than 52. So they didn't pass this filter and that's why we removed these 13 rows from these six manufacturers. And that's it.

In this video, we learn how to use the filter method with the group by object.

In this video, we're going to see the data set that we're going to work with in this section. And first, we're going to import pandas as pd. So, here I run this first line of code. And now, I'm going to read two data sets that we're going to work with in this section. The first is a movies data set from IMDb. And the second is the ratings of those movies. So let's read the first CSV file. So I write pd.read_csv. Then parenthesis and here I write the name of the CSV file. So immov.csv. So now I read and now let's see what happens. So here we got a warning message and we also got the data frame. This warning message is recommending us to use this low_memory parameter and set it equal to false. So this is some kind of issue. So I'm going to set this low_memory equal to false. And now if I run this we'll see that we get the data frame but now we don't get the warning message. So this is an issue that we have with this data set. But if you add the low_memory parameter, everything will be fine. So now we have the first data frame and now let's see some columns that we're going to use for the section. So here this data frame has a lot of columns. Actually if we scroll all the way down, we can see here that it has 22 columns. And we're not going to use all of them. But we're going to use only five of them. And now let's see those five columns that we're going to work with in this section. And the first is the ID. So this IMDb title ID. This one contains the IDs of columns. And ids are unique. This means that we won't find duplicated IDs, but they are unique. So each movie has an ID. Okay. The next column that we're going to use is the title column. So this is the name of the movie. And then we're going to use the year column. So this is the year in which the movie was published. And also we're going to use the genre. And finally the country. So those are the five columns that we're going to use in this section. And I'm going to set this data frame equal to df_mov. So now I run this and here we save the CSV file into this data frame df_movies.

So now let's check the second CSV file. So this one is about ratings of those movies and I'm going to read this CSV file with the read_csv method. So I write read_csv then here I write the name. So, imdb_rating.csv and here is ratings. So, now it's ready. I comment this out and run this. So, here now we have the data frame and each column measures the vote in a different ways. So, for example, here we have the total votes. Here we have the mean and the median and so on. So, we have a lot of columns here, but we're going to use only the ids. This ID column is the same ID column that we found in this df_mov data frame. And the next column we're going to use is the total_votes. And finally, the last column that we're going to use is the mean_vote. So this is the mean of the votes for each movie. And now I'll give a name to this data frame. So I'm going to write df_ratings. So this is the name of this data frame. And now I run this.

Okay. Now let's select the columns that we're going to work with in this section. So I write first df_mov.columns to get all the columns. And now to overwrite the data frame. I'm going to write df_mov = df_mov but now with double square brackets. And inside I write the columns that we're going to select. So these are the columns I mentioned before and I just pasted these five columns. So the ID, the title, the year, the genre and the country. Now let's do the same with the df_ratings. So now I overwrite the data frame. And now I select the three columns that I mentioned before. To do so, I change the name of the data frame here. So I write df_ratings.columns. And now we get all the name of the columns. Now I'm going to select just the columns that I mentioned before. Okay, I pasted the three columns. And now everything is ready. So I run this code. And now let's see how the data frames look. So here I run the df_movies. And now we only have five columns. And now I copy and paste df_ratings. And we only have three columns. And as you can see, both data frames have one column in common. And this is the ID column. Keep that in mind because we're going to use that column a lot in this section. And that's it.

In this video, we explore the two data frames that we're going to work with in this section. Now, it's your time to explore more these data sets so you can understand much better what we're going to learn in this section.

All right. In this video, I'm going to show you how to concatenate data frames in pandas. In pandas, we can use the concat method to concatenate two data frames. And there are two different ways to concatenate data frames. We can concatenate data frames vertically and horizontally. And in this video, I'm going to show you how to concatenate data frames vertically. So first let's see uh these two data frames that we're going to use as examples. And here we have the data frame df1 and df2. And these two data frames have two columns in common. So those two columns in common are the ID and the H. In the first data frame, we see that the values inside the ID column start with the letter A and ends with the letter D. While in the second data frame, the values for the ID column starts with the letter E and ends with the letter H. So we have different values. Whenever we want to concatenate two data frames vertically, we need to have columns in common. In this case, the columns in common are the ID and the H columns. And by concatenating we will have a data frame that combines the data inside the df1 and df2.

Okay. To indicate in pandas that we want to concatenate two data frames vertically, we have to write the axis parameter and set it equal to zero. Zero means vertical concatenation. And if you go to the pandas documentation, you will see that this is known also as concatenating along the rows. So that's another way to call this concatenation. And now let's see how to concatenate these two data frames. So here if you want to concatenate vertically, you have to imagine that one data frame is over the other. This helps us visualize the final output of our data frame. So if we put df1 over df2, this is like concatenating df1 and df2 vertically. So here let's see the result of this. So if we concatenate these two data frames, we're going to get this final data frame which has uh this ID and H column and has all the data in both the df1 and the df2. So here we see that in the ID column we have data from A to D that corresponds to df1 and also data from E to H that corresponds to df2 and well all the data inside the H column was concatenated too. Another thing you have to keep in mind is that by default pandas will concatenate the name of the indexes even if they have the same values. So in dataframe one we have indexes from 0 to three and in dataframe two we also had indexes from 0 to three. So this means that the output data frame will have repetitive indexes and this is not a good practice. Later I'm going to show you how to deal with this kind of behavior. But now let's see the code that helps us concatenate vertically. So here is the code and we have to write pd. This pd comes from pandas and then we have to write the concat method. So we write concat with parenthesis and inside parenthesis we write a list and we start this list with a square brackets and as elements we write the two data frames. So first data frame one df1 and then df2. So here we can add also the axis parameter and set this equal to zero because we want to make a vertical concatenation but it's not necessary because the axis is set to zero by default in the concat method and we can omit that parameter. All right enough talking and now it's time to start writing code. So let's go to Jupyter notebook and here I have the code that creates the df1 and df2. So I'm going to run this code and now I have df1 which is this one is the same data frame I showed you before and also df2. So now let's use the pd.concat method to concatenate these two data frames. So here I write pd.concat then we open parenthesis and here we open square brackets. So first we have to write the df1 and then the df2. So here we can omit the axis parameter but in this case I'm going to add it. So axis=0. So this axis=0 indicates that we want to concatenate vertically. Okay. Now I run this and we see the result. So this is the output and we see that we have the same data frame I showed you before in the slides. So here we have the ID and the H column with the data of df1 and df2. So now I'm going to show you some other parameters you can add. And first is the ignore_index. So if you check here, you can see that there is a ignore_index. This helps us ignore the original indexes from the df1 and df2. So if we add this one, so ignore_index=True, we'll see that here we get 0 1 2 3 4 until 7. And we see that we don't get the 0 1 2 3 indexes twice because we ignored the original indexes. So this new data frame will have a new index ignoring the indexes from df1 and df2. And that's how you use the concat method in pandas.

Now we're going to see an exercise and we'll use this data frame that I showed you in the previous videos and we're going to use this IMDb movies data set and also this IMDb ratings data set. So in the previous videos we read these two data sets and we assigned the df_movies and the df_ratings uh values and in this video we're going to use these two data sets or data frames to solve an exercise. Remember that we also selected some columns. So we focused only on these five columns in the df_movies and these three columns in the df_ratings. Okay. In this exercise, first we have to extract a 50% sample of the original data frame. And this data frame is the df_movies that I showed you before. So this one. So we have to use the sample method to extract a 50% sample. Then we have to show the shape of the data frame that we created here. So after extracting a 50% sample, we create a df_sample data frame. And here we check the shape of that data frame. And then we concatenate the df_movies and df_sample data frames. In this case, we concatenate vertically. And that's what you have to do. So now try this yourself. And after you tried to solve this exercise, you can continue watching this video to see my solution.

Okay. To solve this exercise, we're going to start by extracting a 50% sample and we do that with the sample method. So we write df_mov.sample. So this sample method we learned previously in this course. And as you might know, this sample method has a parameter named frac and that helps us get a percentage of the original data frame. In this case, the df_movies. So we set this frac=0.5 and this means 50%. Then we only run this and here we have the data frame. So this is the sample and here we see that the number of rows is 42,000. So this is half of the number of rows in the df_movies. And now I'm going to set this data frame equal to df_sample. So now I run and I just created this df_sample. Now let's see the shape of this data frame. So I write that.shape. This is a shape attribute. And now let's check. And we see that it has 42,000 rows and five columns. Now let's see also the shape of the df_mov data frame. So here I copy and paste. And now instead of df_sample I write df_movies. So now I print. So we can see both values together. And now I run. So we see that yeah we have half of the number of rows of the df_movies and also we have the same number of columns and actually these are the same columns because we extracted a sample. So if you check here df_movies has these uh five columns title, year, genre, country and also the df_sample has the same columns and this is important because as you might remember when we want to concatenate vertically we need to have columns in common. So these two data frames have columns in common and that's why we can concatenate them vertically. So don't forget about that. And now let's concatenate these two data frames. So here I write pd.concat. Now we open parenthesis. Now I create a list with square brackets. And now I write the name of the first data frame which is df_movies. And now I write the name of the second data frame df_sample. Now we want to concatenate vertically. So I add axis=0. And remember that we can omit this axis parameter if we want because the axis parameter is set to zero by default. So now everything is ready. So we can run this and here we have the data frame that is the output of this concatenation. So this is the result and now we can verify that we successfully perform this concatenation by getting the shape of this data frame. So here I'm going to assign a name to this data frame. I'm going to name it df_concat_vertically. So it's here. And now I only run this. And now let's check the shape of this data frame. So here I write the name of the data frame followed by the shape attribute. And now we have 128,000 rows and this is the sum of 85,000 rows plus 42,000 rows. So this is the sum of the rows in dataframe one and dataframe 2. So with this we can verify that we successfully concatenated these two data frames because by concatenating vertically we sum the number of rows of the two data frames and the number of columns remain the same. And that's it.

In this video we learn how to concatenate vertically in pandas.

All right. In this video, I'm going to show you how to concatenate data frames horizontally and we're going to use the concat method as we did in the previous video. So, first we're going to see these two data frames. And here the second data frame df2, it's different from the previous example. So, in this case, this df2 has only one column and this column is named job. And this

Column is different from the columns that DF1 has. So here in the job column, we have the professions of people in data frame one. So we can say that this data corresponds to the same data we have in data frame one, but they are split. So as you can see, we have different columns, but the indexes are the same. So both have the indexes 0, 1, 2, and three. And this is a requirement to make a horizontal concatenation. So here we need to have the same indexes to concatenate horizontally. And in pandas, when you use the concat method, you need to specify that the axis is equal to one. This axis parameter equal to one indicates that we want to concatenate horizontally. And if you check the pandas documentation, you will see the sentence along the columns, which means concatenating horizontally. So it's the same. Okay.

Now we'll see how we can concatenate these two data frames. So basically, we need to put one data frame next to the other data frame. So we have DF1 on the left and DF2 on the right. So now if we concatenate these two data frames horizontally, we're going to get a data frame that has three columns. So the two columns of DF1 and the one column of DF2, and also four rows. So the four rows that are in those two data frames. So let's see the output of this concatenation. So if we concatenate horizontally, we're going to get this data frame. So the three columns and the four rows, and we could concatenate this because we have these indexes 0, 1, 2, and 3 in common. So now you see that we have the job information available in one data frame and also the ID and the age column.

So now let's check the code that will help us create this data frame. So if we want to concatenate horizontally, we have to write pd.concat, and inside, write data frames inside square brackets because this is a list, and then we have to specify axis equal to one, and that means that we want to concatenate horizontally. So by default, axis is equal to zero, which means concatenate vertically, but when we set it to one, we tell pandas that we want to concatenate horizontally, and that's everything you need to do. All right, enough talk. Now let's write code in Jupyter notebook.

So now I go to Jupyter notebook, and now I have the code that creates the data frames DF1 and DF2. So I run this code, and now I show you the data frames which look exactly the same as the two data frames we've seen before. So DF1 and DF2. So now let's concatenate these two data frames. So I write pd.concat, then parenthesis. Then we need to write the two data frames inside a list. That's important. Don't forget about it. So first df1, then df2. And now we have to specify the axis. So axis equal to one. So this means concatenate horizontally. Now I run this, and we get a single data frame with the three columns. So we see the ID, the age, and the job in this single data frame. Okay.

Now let's check some other parameters that might come in handy. So here you see the ignore_index. In this case, we don't need the ignore_index because we make a concatenation that is horizontal and we have the same indexes. Actually, that's the reason why we can make this horizontal concatenation. So it's not useful for this case. And now a parameter that might come in handy is the sort parameter. So this sort parameter allows us to sort the data frame. So if you set it equal to true, you will be able to sort this data frame. And that's it. Okay.

Now it's time to put everything we learned into practice solving an exercise. So here we're going to use the data sets that we read previously. So these are the IMDb movies data set and also the IMDb ratings data set. So in previous videos, we read these two data sets and we set it in these two data frames. So here we have the DF_movies and the DF_ratings data frames. We're going to use these two data frames for this exercise. And remember that we selected these five columns for the DF_movies and these three columns for the DF_ratings data frame. Okay.

Now in this exercise, your task is first to show the shape of the data frames that will concatenate. So the DF_movies and the DF_ratings. Then you have to concatenate these two data frames on the IMDb title ID column. So this is the column that these two data frames have in common, and you have to concatenate these two data frames horizontally. So after you concatenate these two data frames horizontally, you set a name to the concatenation and then you use the shape attribute to calculate the shape of this new data frame. So now is your time to solve this exercise on your own. And after you try, you can continue watching this video to see my solution.

Okay, let's start solving this exercise. So first we calculate the shape of the two data frames. So first I write df_movies.shape, and this returns the number of rows and the number of columns of this data frame. So now I'm going to get the shape of the DF_ratings too. So here instead of writing DF_movies, I write DF_ratings. So now I print both shapes. So I use the print function. And now I run the cell. And here we have the output. So we see that we have the exact number of rows in these two data frames, but the number of columns is different. So in the first data frame, we have five columns, and in the second data frame, we only have three columns. All right.

Now let's continue. And now let's concatenate the two data frames. So we use pd.concat(parenthesis). Then we open these square brackets, which is really important not to forget. And here we write the first data frame. So I copy and paste it. And then I copy and paste the second data frame. So DF_ratings. Then we have to specify the axis equal to one because this is a horizontal concatenation. And now everything is perfect, but we need to make a little change. So as I mentioned before in the theory, we need to have common indexes, and in this case, we have a common column which is the ID column, and we can set this column as an index. So we have a common index between these two data frames. So let me show you here. So here if I print the df_movies, we see that we have this ID column in this data frame, and now if I print the df_ratings, we'll see the same column in this data frame. So here is the data frame, and we see the same column. So this is the only column that these two data frames have in common, and we can use this column as an index. So if I use .set_index, we can set this column as an index. So here I write the name of the column. And now I run this, and as you can see, now this ID column is my new index. And we can do the same for the DF_movies data frame. So here I comment this out, and now I set that column as an index. And now to update the data, we have to use the inplace parameter and set it equal to true. So this helps us save all the changes that we made with the set_index method. So we do the same for the second line of code. So here I write inplace=True. And now with this, we're going to set the ID column as index for both columns. So now I run, and if we check any data frame, we'll see that the index is the ID column. So perfect. Okay.

And only now we can concatenate these two data frames because right now the index of df_ratings is the same index that we have in df_movies. So this is the requirement that we need to concatenate horizontally. So we satisfy that requirement, and now we can run this code. So I run this code, and now we have this concatenation. So this data frame has the four columns of the DF_movies and the two columns of the DF_ratings. And here the index is the ID column. So perfect. Now I'm going to set a name to this data frame, which is going to be here df_concat_horizontally. So this is the name of my data frame. And now let's check the shape of this data frame. So I write the name of the data frame followed by the shape attribute. So now we see that this data frame has 85,000 rows. So this is the same number of rows that the DF_movie and DF_ratings had. So that's fine because when we concatenate horizontally, we don't change the number of rows. Actually, the number of rows have to be the same because we should have indexes in common. But now in the columns, we see that this df_concat_horizontally has six columns. But the df_movies has five columns, and the df_ratings has three columns. So here we see that by concatenating these two data frames, we get the sum of these two columns. And now you might be wondering 5 + 3 is 8. So this is not equal to six. But here we set the ID column as index in both data frames. So we lost one column in each data frame. So this is why here we have four columns and here we have two columns. So now 4 + 2 is six. And with this, we can verify that we successfully concatenated these two data frames. And that's it. In this video, we learn how to concatenate two data frames horizontally.

All right. In this video, we're going to see how joins work in pandas. And first, we're going to see the inner join. The inner join returns matching values between two tables. In this case, it will be the matching data between two data frames. So if we have data frame one on the left and data frame two on the right, the data that we have in common is the data that will be in the middle. So the area with the white dots that you see in this image. So now let's check how the DF1 and DF2 look. And these are the data frames that we'll use in this example. So DF1 and DF2 have a column in common. In this case, it is the ID column, and also DF1 has the H column, and DF2 has the job column. But what is important here is that the ID column has some values in common between DF1 and DF2. So if you compare these two columns, you can see that the C and D values are the same between these two columns, while this A and B only belongs to DF1 and the E and F only belongs to DF2. So if we keep that in mind and we see again the diagram that we've seen before, we can see that the A and B is on the left, while the C and D is on the area with the white dots, while the E and the F is on the right. So this represents the data inside the ID column in these two data frames. So this is basically how the inner join works. So the data that only belongs to DF1 will go to the left, and the data that is in common will go in the middle, and the data that only belongs to DF2 will be on the right. Okay.

Now this inner join is possible because we have the ID column in common between these two data frames. So this allows us to make this inner join. And to write the code that produces this inner join, we have to use the merge method. So we write df1, which is the data frame on the left, and then we use the merge method. So .merge, and then inside parenthesis the first parameter is the second data frame. So df2, then we have to write the on parameter, and here we have to specify the column that both data frames have. So in this case, the ID column, and finally, we have to specify the how parameter, and here we write the type of join we want to perform; in this case, inner join. So we write inner, and when we make this inner join by writing this code and using this merge method, we get this data frame. So the output is this data frame that you see now on screen. And this data frame has the three columns. So the H belongs to the DF1, and the job belongs to DF2. And we only see two rows. We see the row that has the data C in the column ID and also the second row that has the data D. So this is the output after using the merge method.

Now let's write some code in Jupyter notebook. So let's go to Jupyter notebook. And now I have these two data frames. I have the code to create these two data frames. And now I just run this code. And now I can show you these two data frames. So I write df1, and you can see the first data frame, and df2, and you can see the second data frame. So now let's use the merge method to merge these two data frames. So we write df1.merge, then parenthesis, and inside df2, then on, this is the on parameter, and here we write the column that these two data frames have in common; in this case, the ID, then the how parameter; in this case, inner because we want to make an inner join, and that's enough, that's everything we need. So now we run this, and we get the same output I showed you in the slides. So these three columns and this data that corresponds to the C and D ID. Okay.

Now let's verify the shapes of these three data frames. So I'm going to assign this data frame a name. So I write DF_inner_join. So this is my data frame. And now let's check the shape. So first df1.shape. Now I copy this. I paste it twice. df2, df_inner_join. So now I paste it. And now I'm going to print the three values. So here I use the print function. And now it's ready. So now I run this code, and we see that the first two data frames have the same number of rows and columns. But the last data frame, so the data frame that represents the inner join, has only two rows and three columns. So this happens because these two data frames have two rows in common. So here we see these two rows that have in common, which are in the C and D rows. So these two rows, and here the three columns are the ID, the job, and also the age column which is here. So this is why we have three columns and two rows. And that's how you make an inner join using the merge method.

So now let's see an exercise to put everything we learned so far into practice. So for the exercise, we're going to use the df_movies and df_ratings data frames that we created in the previous videos. So now let's check the exercise, and it's here. So for the exercise, you have to merge the DF_movies data frame and the DF_ratings data frame. You have to explore the data frames and find the column that these two data frames have in common. And after that, you have to use the how parameter and set it equal to inner to make this inner join. So now you can try this yourself, and after that, you can continue watching this video to see my solution.

Okay, to solve this exercise, first let's see the two data frames so we can find a common column. So here is the df_movies. We see these five columns. And now let's print the df_ratings and let's see what column they have in common. So I just realized that the two data frames have this ID column in common. And now I only have to use the merge method. Open parenthesis and write this df_ratings inside parenthesis. Then in the on parameter, we can write the name of this column, which is IMDb title ID, and then we have to write the how parameter and set it equal to inner. But here there is a little detail you have to know. So here the default value that the how parameter has is inner. So if you press shift plus tab, you can see here in this little window here in how the default is inner. So you can use this to find the default values that all the parameters have, and in this case, the how parameter has this inner value by default. So this means that we can omit this how parameter because it's set to inner by default. So I can delete this. And now I can run, and we can see that this data frame has the columns of DF_movies. So these four columns belong to DF_movies, and these last two columns belong to this DF_ratings. And also the IDs that df_movies and df_ratings have in common are shown in this data frame. So only the common IDs are here in this data frame. And that's how you make an inner join between the df_movies and df_ratings.

Now I want to show you a different way to use the merge method. So this way is similar to the way we use in concatenation but now is in merge. So the only thing you have to do is to write pd.merge, then parenthesis, and write the data frames you want to merge. So df_movies and df_ratings. So this looks like the syntax of the pd.concat, but now instead of pd.concat, it's pd.merge. Now we have to add the on parameter and set it equal to the ID column. So now I set equal to this. I comment this out, and if I run this, we get the same result. And well, the IDs that these two data frames have in common are in this ID column. And that's it. In this video, we learn how to make an inner join in pandas.

All right. In this video, I'm going to show you how full joins work. And full joins allow us to join all the elements between two tables. And you can see much better how full join works in this diagram. So here all the white dots cover all the space in these two tables. So all the area covered by the white dots represent the full join in this diagram. And now let's see an example to understand this much better. So here I have the two data frames we've used in the previous video. And what we're going to do here is to join these two data frames, but in this case with a full join. So here now we have all the elements from the ID column. So here is the ID column of DF1 and ID column of DF2. So we can see that here C and D are the elements in common in this column. So A and B only belong to DF1, and E and F only belong to DF2. But we see that all the elements are taken by this full join because this full join takes all the elements in these two data frames. And we can see here that we have DF1 and DF2 again. And we see that A, B, C, D, E, and F are covered in this full join. So now let's see how to write code that produces a full join. So here again we have to find a column in common. In this case, the column that both data frames have is the ID column, and to make this full join, we have to use the merge method. So we write df1.merge, and inside parenthesis we write df2, and then on the on parameter we set it equal to the column that these two data frames have in common. So in this case, the ID column, and finally, in the how parameter, we set it equal to outer because this is a full join, also known as an outer join, and well, they are the same. So the outer join is the same as the full join or full outer join. Anyway, so now let's see the output of this full join. So what data frame we're going to get after using this code. So the output will be this. So this data frame has all the data that we had in DF1 and DF2. But as you can see in this data frame, only the rows that belong to the ID C and ID D have non-null values, while the rest have some null values. And this happens because the DF1 doesn't have the job column, and the DF2 doesn't have the H column. So this produces some null values when we make this full join. But this is fine. This is just how the full join behaves when we have this type of data frames. So now enough talk. Now let's see how to produce this full join using pandas. So let's go to Jupyter notebook. And now I'm going to show you the two data frames we're going to work with, which are the same data frames we've seen in the slide. So first the DF1 and then the DF2. So here's DF1 and DF2. So for a gentle introduction to this full join, we're going to merge these two data frames. And now we

Only have to write uh here DF1 merge, then parenthesis here DF2, then on and here the id because this is the column in common. Then how, and we set the how parameter equal to outer. And now this produces the data frame we want it. And here we see that uh the A and B ID have null values, and the E and F have null values too. While the C and D uh don't have null values because these are the elements in common between these two data frames. Okay, this is how the full join works.

And now to understand much better all the concepts we learned so far, we're going to solve an exercise. And in this exercise, you have to merge two data frames. In this case, the data frames DF movies and DF ratings. These two data frames we've seen in the previous videos. And now you have to find the column in common and perform an outer join. So now try to solve this exercise on your own, and after you do that, you can continue watching this video to see my solution.

Okay, to solve this exercise, let's explore this DF movies and DF ratings data frame. So here I print this DF movies, and now we have this data frame that we've seen in the previous videos. So now let's find which column they have in common. So here I print DF ratings. And as we can see, both data frames have this ID column. So I'm going to use this in the on parameter. And here I'm going to use the merge method first. So that merge, then open parenthesis. Then we write the second data frame. And now we write the on parameter. So on equal to this imdb title id. So now how in the parameter we set it equal to outer. So now this is ready. So we run this, and we get this data frame that is the full join uh of the df movies and the df ratings. So as you can see, these four columns belong to this DF movies, and these two last columns belong to the DF ratings, and this full join looks like the horizontal concatenation we learned in previous videos, and that's an alternative to solve this type of exercise too. But in this case, I chose to use the merge method because we're learning full joins. But you could also solve this exercise with the concat method.

Okay, that's enough for full joins. And now we're going to see a type of join that is similar to the full join but has some differences.

Okay, now it's time to see exclusive full joins. And exclusive full joins will pick only the values that are exclusive to the tables. So here I have two tables, and the exclusive values from table one and exclusive values from table two will be taken in this exclusive full join. So only this area that you see here on the left that is exclusive to the first table or first data frame and also this area that is with the white dots that is on the right that is exclusive to the second table or second data frame. So this is known as the exclusive area because it belongs either to one table or to the other table. However, the area that is in the middle is the common area, and this area is not considered in this exclusive full join because this area belongs to both tables or both data frames.

Okay. Now let's see an example. So we're going to use the same data frames here that we've been using so far. And now let's see how these uh elements will fit in this diagram. So now we see that the A, B, E, and F were taken in this exclusive full join because here in the data frame we can see that A and B only belong to DF1 and E and F only belong to DF2. However, C and D belong to both DF1 and DF2. So, this is why we didn't take this C and D because this is exclusive full join and we only want exclusive values.

So now let's see how we can make this exclusive full join uh with code. First, we need of course a column in common. So in this case, the column in common is the id column, and after we find this column, we need to use the merge method. So we write dfmer merge, and inside we write df2 and then also the on and the how parameter. In this case, the how parameter is going to be outer. So it's similar to the outer join, also known as full join. But in this case, we're going to add an extra parameter, which is the indicator. So this indicator helps us know which values are exclusive to DF1 and which values are exclusive to DF2. So after we add this indicator, we're going to get a new column in our output data frame. And this new column is going to be named _merge by default. And in this column, you will see values like left only, right only, and both. So the values that are left only corresponds only to the data frame on the left, in this case DF1. And the values that say right only corresponds only to the data frame on the right, which is DF2. And these are the two values that are exclusive to DF1 and DF2. So once we have this column, we're going to use the query method to select only the exclusive values to DF1 and exclusive values to DF2. And with this, we're going to filter out the values that these two data frames have in common.

All right. Now, let's see the data frame that we're going to get as output after running this merge method. And the output is going to be this. So we only get the A, B, E, and F uh ids, and the C and D were filtered out. Also, you see that we have some null values because the DF1 doesn't have the job column and the DF2 doesn't have the H column. And this is why you see here some null values.

All right. Now let's see how to make this exclusive full join using pandas. So now I go to Jupyter notebook, and again I'm going to use that DF1 and DF2 that we've seen before. So these are the same data frames we've seen in the slides and also in the previous videos. And I'm going to use these two data frames to make this exclusive full join. So here what I'm going to write is first DF1 merge and here DF2, then also we add the on parameter equal to id and the how parameter equal to outer, so this creates a full join, and that's fine, but now we want an exclusive full join, so we have to add the indicator equal to true, and here I copy and paste the code, and Now I add the indicator equal to true. So with this we create a new column called _merge, which is this one. And as I mentioned before, we have here three types of elements: Left only, both, and right only. So left only are the elements that are exclusive to DF1, and right only are the elements exclusive to DF2. But the both values are the elements that have in common DF1 and DF2.

Great. Now let's get only the left only and right only values from this merge column. And now I'm going to use the query method. So here I copy and paste, and now I'm going to use this query method. So I write that query, and after this I'm going to write this query. So here I press enter. So you can see much better the query I'm about to write. So to select only the left_only and right_only values, we first open these quotes and now write the name of the column. So in this case _merge, and now we compare with this uh equal sign. So double equal sign, and now we write left only. And now we have to do the same with the right only. So I copy this and I paste it here. But now instead of left only, I copy and paste right only. So now we need to add the quotes to specify that this left only and right only are strings. So here I add quotes, and here I add quotes again. And keep in mind that here I'm using single quotes, but here in this quotes that we wrote before uh here I wrote double quotes. And this is to avoid some type of conflict between quotes. And now I'm going to use the or operator. So we indicate that we want either uh left only or right only. So here I write or, and this is how we select the left only and right only values from the merge column. So here I'm going to run this code. So I press Ctrl enter. And now you see that this data frame that we have as output only shows the values that are left_only and right_only. And that's everything you have to do to make an exclusive full join.

All right. Now that you have a good idea of how to make these exclusive full joins, we're going to solve an exercise. So here in this exercise, we have to merge df movies and df ratings. And in this case, the type of join is going to be an exclusive full join. And well now explore df movies and df ratings. Those are the data frames we seen in the previous videos. So you have to find that column in common and then use the same steps we uh followed here. So first add the indicator, then add the query, and then make this exclusive full join. So now try to solve this exercise on your own, and after you do that, you can continue watching this video to see my solution.

Okay, to solve this exercise, first let's have a look at the data frames. We've seen this data frames before. So I have a good idea of the column that they have in common. But in case you don't remember, I'm going to print both data frames. So first the DF movies, now the DF_ratings. So as you can see, both data frames have this ID column. And now we're going to use this column to do this exclusive full join. So now I write merge, then parenthesis. And now I put this second data frame inside parenthesis. Then I add the on parameter, and in this case it's going to be equal to this imb title id. And then we add the how parameter. So here equal to outer. We press ctrl enter. And with this we get this full join. So this is a full join, and what we want right now is an exclusive full join. So here I'm going to um add the indicator. So I copy and paste. And now I add the indicator parameter equal to true. So now to see much better this I'm going to press enter here. And now it's much clearer. So I'm going to run this cell, and here we see the data frame. But now a new column was created, and is this _merge column. And here we can see that most of the values are set to both. And now I remember that these two data frames DF movies and DF ratings have the same ID. So all the values inside the IMDb title ID column are the same. And that means that all the values inside the _merge column are going to be equal to both. So all these values will be both, and this will affect somehow this exclusive full join. But I'm going to show you uh what's going to happen. But first let's add the query. So here I'm going to minimize this data frame. Now I paste this code, and I'm going to add the query. So I write query parenthesis. And now I'm going to copy and paste the same query we created before because it's the same query uh we're going to use every time we want to make an exclusive full join. So every time we want to make an exclusive full join we only use this uh code. So here this is ready, and now I'll press Ctrl enter, and we see that this data frame is empty, and we see only the column names, but there is no data in this data frame, and this happens because here in the _merge column all the values were set to both. So this means that all the values in the ID column are the same between the DF movies and DF ratings. So these two data frames have the same ids, and none of these data frames have exclusive ids. So this is why we got this uh empty data frame because these two data frames don't have exclusive values, and this is fine. We didn't make any mistake. This happened only because the IDs between these two data frames were the same. But this code that we wrote here is correct. And that's it for this video. I hope you successfully solved this exercise.

All right. In this video, we're going to see how left join works. Left join allows us to grab only the data that belongs to a data frame or to one table. So this is the data that is exclusive to data frame one or table one and also the data that data frame one or table one has in common with data frame two or table two. So both areas are covered by the white dots, and this represents the left join.

So now let's see an example, and we're going to use the same data frame we've been using so far. So this DF1 and DF2, and now let's see how this uh data inside the ID column will fit into the diagram. So here we have this A B C D E and F. And we see that only the data A B C and D is covered by the white dots. And this happens because this data belongs to the DF1. So here we see that in the ID column there is the A B C and D ids. So this data belongs to DF1, and the E and DF weren't selected because these two ids belong only to DF2. So this is why this E and F ids weren't selected in this left join. So the left join will return only the data that corresponds to the left table or left data frame. So in this case, the DF1.

All right. Now let's see how to make this left join in pandas. And as usual, we will need a column in common. And in this case, the column in common is the ID column. So both the DF1 and DF2 have this ID column. And now what we have to do is to use the merge method to create this left join. So we write DF1 merge, then parenthesis, and then we write DF2 because that's the second data frame. Then on the on parameter we set it equal to the column in common, in this case ID. And after this, in the how parameter we set it equal to left because we want to make a left join.

Okay. After we do this, we're going to get the following data frame. So this is the data frame that we get after making this left join. And as you can see here, all the data that belongs to the DF1, that is the ids A, B, C, and D, will be in this data frame. However, some data is missing. So, we see here some new values, and this happens because the DF1 doesn't have this job column, and this means that we don't get that data because that data doesn't exist. However, for the IDs C and D, we got the data that corresponds to the job column because this data also exists in the DF2. And since we have that data when we make the left join, we get this data that is doctor and a statistician. So this happens because we made a left join. And now that's enough for the theory. Now let's check how to make this left join. But now writing code in Python. So let's go to Jupyter notebook. And here I'm going to use the same data frames DF1 and DF2. And now let's make this merge. So I write df1 uh merge, and then I write parenthesis and write df2 inside parenthesis. So now df2, and now the on parameter will be equal to id and the how parameter uh will be equal to this left. So we set it equal to left because we want to make a left join. Now we press Ctrl enter, and now we have this data frame. So here we have some null data, and here we have the data that corresponds to DF2 in the job column, and also ids A, B, C, and D are in this data frame because this data corresponds to DF1.

All right, that's how you make a left join. And now let's put everything we learned into practice by solving this exercise. So in this exercise, we're going to use the DF movies and DF ratings data frames. And then we're going to follow these steps. First, we extract a 50% sample of the DF movies data frame. And then we merge this uh sample. So here we give a name to this sample data frame. We assign the name DF_OMOV sample, and we merge this new data frame with the DF ratings. So here the type of join is going to be a left join. And once you have this data frame, you have to get the shape of all the data frames involved in this exercise. So we compare the shapes and draw some conclusions.

All right. Now it's your time to solve this exercise on your own. And after you try to solve this exercise, you can continue watching this video to see my solution.

All right, to start solving this exercise, I'm going to extract a 50% sample of the df movies data frame, and I'm going to use the sample method. So I write df movies and then sample and then I open parenthesis. We learned this method in this course before. And now you might know that here we have the frac parameter, and this frac parameter allows us to extract a percentage or a fraction of the original data frame. So in this case, we want the 50%. So we set it equal to uh 0.5, and with this we get this sample. Now I'm going to give a name to this data frame, and the name is going to be df_mov sample. So we run this, and now we have this sample. So now let's merge the two data frames. And first let's write df_mov sample merge, then parenthesis, then the second data frame. And now I write the on parameter. In this case, the the column that they have in common is this uh ID I guess. Let me check here. So this data frame has this ID, and also these ratings I believe has this ID. So let's check, and yeah, they have in common this ID column. So here I write the id, and now as last parameter we add how, and now we set it equal to left because we want to make a left join. So now I run this, and this is the data frame we get after making this left join. So I'm going to give it a name. In this case, this is going to be equal to df left. And now it's time to compare the shapes of these data frames. So here I write df_mov sample. So I better copy and paste now that shape. So we get the shape of this data frame. And now I print this. So print, and now copy and paste this twice. So now I write DF ratings. I copy and paste it. And finally DF left. And after I get the shapes, we're going to compare the values. So now I run this, and we have these shapes. So here the first element is the number of rows, and the second element is the number of columns. And we see that the number of rows of the df_mov sample is around 42,000. The number of rows of the df ratings is 85,000. And the number of rows of the data frame we got after making the left join is also 42,000 or around 42,000. And this happened because we made a left join, which means that only the data that belongs to this df movies sample will remain, and the data that is not here but is only in the df ratings will be excluded. So we're using this id column in this on operator, and this indicates that only the ids that belong to the df_mov sample will remain in this dfc left, and that's why we got around 42,000 rows in the data frame that corresponds to the left join, that said all the number of columns were combined. So here we have four columns of the first data frame, two columns of the second data frame, and then the ID column that these two data frames have in common. So in total 4 + 2 + 1. So seven columns in this final DF_left data frame. And that's everything you need to know about left joins. And now let's see a different type of join. In this case, we're going to see the exclusive left join. And this is a bit similar to the left join, but now we're going to get exclusive values from the data frame that is on the left. So here, let's see the diagram. And now you see that the white dots only cover the area that is exclusive to the first table or first data frame. So here, let's see an example. So here we have the DF1 and DF2. And now let's see how the IDs fit into

The diagram that we've seen before. So let's check it out. And it's here. So we see that only the A and B ids are taken in this exclusive left join because these are the only ids that are exclusive to this DF1.

So here in the data frame we see that this ID A and this ID B is the only data that is exclusive in this DF1 in the ID column. So if you try to find this A and B ID in DF2, you won't find them because it's exclusive of DF1. And you see also that the rest of the IDs, so C, D, E, and F, isn't exclusive of DF1. So this is why these four ids were excluded in this exclusive left join.

So now let's see how to make this exclusive left join uh with pandas. So first, as usual, we need a column in common. In this case, this column is the id, and after we recognize this column, we have to use the merge method again. So we write df1.merge, and then we write the second data frame, in this case df2. Then we write the id in the on parameter, and then in the how parameter we have to set it equal to outer, and as a rule of thumb, we always set the outer value in the how parameter when we want to create an exclusive join. So either an exclusive left join or an exclusive full join or an exclusive right join. So every time we want to create an exclusive join, we have to use the outer in the how parameter.

Okay. After this, we have to use the indicator parameter and set it equal to true. So as you might remember, this indicator will create a new column called _merge, and there we will see the data that is exclusive to DF1, the data that is exclusive to DF2, and also the data that belongs to both DF1 and DF2. After we get this column, we're going to use the query method to get the data that is exclusive to DF1 only. So we're going to see in detail that when we write code in Jupyter notebook. But now let's see the data frame that we're going to get as output.

So this is the data frame that we get after making this exclusive left join. And as you can see here, the data that we get is only the data that corresponds to the A and B ids. So here we see that in the job column we got null values, and this is because the DF1 doesn't have this job column, and as a result, we got this uh null values.

So now enough theory, now let's write code in Jupyter notebook, and now let's create this exclusive left join. So let's go to Jupyter notebook.

And now here we're going to uh merge these two data frames DF1 and DF2. And to do that, we have to write DF1.merge, and then DF2, then the on parameter equal to ID, and then the how parameter equal to outer. With this, we create a full join. So I'm just going to show you here. But if we add the indicator parameter, we can set this indicator equal to true. So I just copy and paste this. Copy and paste. Okay. And with this indicator equal to true, we created this _merge column. And here we're going to pick only the left_only values because these values represent the values that are exclusive to DF1. So left_only means the data that is exclusive to the data frame on the left. So in this case DF1, and we're going to exclude the both and the right_only values because that data is an exclusive of DF1.

So now let's create the query. So here I copy and now I paste it here. So now I write that query. Now parenthesis. Now I press here enter. So you can see much better the query that I'm about to write. And here I open these quotes. Now _merge because this is the name of the column. Then we write double equal sign. And now I write the left_only because this is the data exclusive to DF1. So here I write single quotes and paste left_only. And with this our exclusive left join is ready. So now I can run this and we can see the result, and we get this data frame and on the _merge column we see that the data that we got has only the left_only values. So with this we can verify that this exclusive left joint was successfully performed.

And that's everything you need to do to create an exclusive left joint. And now we're going to put everything we learned into practice by solving this exercise. So in this exercise, you have to follow these steps. First, you have to make a copy of that DF_movies data frame that we've seen in previous videos. And then you have to set the first 1,000 values of the ID column as this ID that I created. So this ID that I created is TT1234567890. So this is an ID that I just came up with. And you have to set this ID to the first 1,000 values of this ID column. After you do that, you have to merge the data frame that you got after making the copy, which I named DF_MOV_2. And you merge this data frame with the DF_ratings. And the type of joint that you're going to make here is an exclusive left joint. And after this, you have to find the shape of all the data frames that are involved in this exercise to compare and draw some conclusions.

All right, now is your time to solve this exercise. And after you try to solve this exercise, you can continue watching this video to see my solution.

Okay. To solve this exercise, first I'm going to make a copy of this df_mov data frame. So I write df_mov.copy(). So I'm using the copy method we learned in this course. And then we set this equal to df_mov_2. So with this copy method, we make an independent copy. Now we run this. And now we created this df_mov_2. After this, to save the first 1,000 values to this ID that I created, I'm going to use a for loop. So here I write for index in, and here I'm going to loop through the indexes in this data frame that I created. So here I write df_mov_2.index:, and then press enter. So here I'm going to show you what I'm doing. So this index attribute returns all the indexes in this data frame. So here we're looping through these indexes. And now I'm going to write a condition. So I write if index < 1000 then execute the following code. So with this we're going to change the values of the first 1,000 rows only. So now I press enter. In here I'm going to write the code that will allow me to change this uh values of the ID column. So now I write the name of the data frame and here I'm going to use the loc method. So I write .loc, then open square brackets, and the loc method needs uh two arguments. So first the name of the index and then the name of the column. So the name of the index is this index that I'm using here when looping through this indexes list. So here I'm going to use this index. And now the name of the column is the ID column. So this one, imdb_title_id. So in case you don't remember this loc method, I'm going to quickly remind you what this loc method does. So when we use the loc method, we locate an specific data in the data frame. So here, for example, I write the name of this data frame and then I write .loc, and then let's say we want to get the data inside the index zero. So we press Ctrl enter and we get all the data that corresponds to the index zero. But now let's say that we only want the data that belongs to the index zero and also to the column id. So now I only have to write the ID column and then run. And we get this ID. And now we're going to change the value of these ids for only the first 1,000 rows. So now we set this equal to the ID that I created, this one that is tt1234567890. So this is the value and now that this is ready I'm going to run this code. So now I run and now let's see the data frame. So now I'm going to print df_mov_2. And now you can see that the first 1,000 rows of this ID column uh have this TT1234567890. So we successfully change the data inside the first 1,000 rows. And now I'm going to uh make this merge. So first I write the name of the data frame df_mov_2, then .merge, and then parenthesis, then I write the name of the second data frame and then the on parameter equal to this ID column and then the how parameter equal to the outer value. So here with this we get an full join. But now if we add the indicator equal to true and if we add also the query we can get this exclusive left join. So here I write that query and now inside parenthesis I'm going to write the same code I wrote here. So _merge == 'left_only'. So every time we want to make a left join we can use this uh query. So in this case this query help us get the exclusive left join. So we paste it here and now this is ready. So we can run this cell and see the result. So I run and now we have this data frame and on the _merge column we see that the data that we have is only left_only. So apparently everything is correct. So now I'm going to set a name for this data frame and the name I'm going to give this data frame is df_exclusive_left. So now I run this and now we have this data frame and now let's find the shape of this data frame. So first df_exclusive_left.shape. Now print( ), now parenthesis and now I copy and paste this twice. So now we have this and now I write the name of the DF_MOV_2 and also DF_ratings. So now I run this cell and now let's see the result. And here we see that the first and the second data frame have around 85,000 rows. However, the DF_exclusive_left data frame only has 1,000 rows. And this happens because this 1,000 rows is exclusive to that df_mov_2. And this 1,000 rows correspond to this ID I created. So this TT1234567890. So I created this ID in purpose because this ID didn't exist. So those ids were unique to this data frame. And this is why in the exclusive left joint, we only got this 1,000 rows. So this means that the other 84,000 rows aren't exclusive to DF_MOV_2. So probably they also belong to this DF_ratings data frame. And that's it. In this video, we learn how to make a left joint and also an exclusive left joint in pandas.

Okay. In this video, we're going to learn how a right join works. A right join allows us to grab the data of the second data frame or the second table. So consider this diagram. Let's say that the data frame on the left is the first data frame DF1 and the data frame or table on the right is the second data frame DF2. So a right join will grab only the data that corresponds to DF2 or table 2. So this data is the data that is exclusive to DF2 and also the data that belongs to DF2 and to DF1. So the data in common, this right join is represented by the white dots in this diagram.

So now let's see an example. These are the data frames that we've seen in the previous videos. So we have DF1 and DF2 and the ID column that is the column that these two data frames have in common. And now let's see how this data that is inside the ID column will fit into the diagram that we had before. So here we have the data that is inside the ID column. So A, B, C, D, E, and F. And now we see that the right join only covers the ids C, D, E, and F. And this happens because this data corresponds to that DF2. So C, D, E, and F belong to DF2 while A and B doesn't belong to DF2. So only the data that is exclusive to DF2 and also the data that DF1 and DF2 have in common will be taken by the right join, as you can see in this diagram.

Okay. Now let's see how to make the right join using pandas. So first we have to identify the column in common. So in this case is the ID column, and after we do that we have to use again the merge method. So we write df1.merge, and inside we write the second data frame, also the column in common in the on parameter, and then we write the type of join in the how parameter. In this case, this is a right join. So we write how='right', and after we write this we're going to get the following output. So this is the data frame that we get as output. So here we see that only the data that corresponds to C, D, E, and the F ID is in this data frame, and this happens because these ids belong to DF2.

And now let's see how to implement this right join in pandas. So let's go to Jupyter notebook. And here I'm going to use the DF1 and DF2 data frames. So let's write the code to make this right join. So first df1.merge( ), then parenthesis df2. So this is the second data frame and then the on parameter equal to id and the how parameter equal to right. So with this we make this right join. So now let's run this code and we get this uh data frame. So, it's the same output we've seen before. And as you can see, making a right join in pandas is as simple as that.

Okay. Now, let's solve an exercise to challenge ourself a bit more. And we're going to follow these steps. First, we're going to extract a 30% sample of the DF_ratings data frame. Then, we give a name to this sample. So, I'm going to name it DF_ratings_sample. And then we uh make a merge. So between the df_movies and this df_ratings_sample that we created in the first step. And in this case, the type of joint that we're going to make here is the right joint. And after you do that, you have to calculate the shape of the data frames. So we can know what's going on behind the right join.

Okay. Now is your time to solve this exercise on your own. And after you finish, you can continue watching this video to see my solution.

Okay. To solve this exercise, first I'm going to extract a 30% sample of the DF_ratings data frame. So I write DF_ratings.sample(frac=0.3). And now I give a name to this data frame. So I'm going to name it DF_ratings_sample. And now I run this. And I just created this data frame. So now I'm going to merge the df_mov and this new data frame. So I write df_mov.merge( ), then the name of my second data frame which is this new data frame I just created and then the on parameter and after this the how parameter. So on the on parameter we have to write the column these two data frames have in common and this column is the ID. I'm going to show you here. So I'm going to use the columns attribute and is here. So ID and now I paste it here and in the how parameter I set it equal to right because we want to make a right join. So this is ready and now I'm going to run this code. So I press Ctrl enter and now we made this right join. So we have the data that belongs to df_ratings_sample and also we got the columns that belong to df_movies. So now I'm going to name this data frame and I write = df_right. So this is the name of my data frame that we got after making the right join. And now I'm going to run this. And here I calculate the shape. So first df_right.shape. And now I paste twice. Then I copy the values of the data frame. So df_movies and the other is df_ratings_sample. And the last one is the df_right. So it's here. And now I print these three shapes. So I use the print function. And here uh I copy and paste. And now we have the shapes. So we see that the number of rows between the df_right and the df_ratings_sample are the same. And this happens because we uh made a right join. And with this we obtain only the ids that belong to the second data frame. In this case the df_ratings_sample. So all the data that corresponds to those ids will be available in the df_right. However the ids that only belong to the first data frame. So df_movies will be excluded. And this is why some of the 85,000 rows from DF_movies were not considered in this right join. And that's everything you need to know about the right join. Now let's see the next type of join.

Okay, now it's time to see an exclusive right join. And this type of join is similar to the right join. But here we're going to exclude the data that the data frame one and data frame two have in common. So the only data that the exclusive right join will grab is the data that is exclusive to the right table or the second table, also known as the second data frame. So this is the area that is covered by the white dots. So that area represents the exclusive right join.

And now let's see an example with the same data frames we've seen so far. So DF1 and DF2. And now let's see how the ids fit into this diagram. So here we see that only the E and F ids will be taken by this right join because this is the data that is exclusive to the DF2 or the second data frame. So now let's see these two data frames again and we can verify that the ids E and F belong only to the DF2 while the other ids either belong only to DF1 or belong to both DF1 and DF2.

So now let's see how to make this exclusive right join in pandas. So first, as always, we have to find the column in common, in this case the id column, and after this we use the merge method. So we write df1.merge, and inside parenthesis we write the second data frame, the column in common in the on parameter, and in the how parameter we have to choose the outer join. So in this case we have to write outer because as I told you before in some other video, as a rule of thumb, every time we want to make an exclusive join we have to write the word outer in the how parameter, and after this we have to add the indicator parameter and set it equal to true. This indicator will create a new column named _merge. And in this column, we'll see which data is exclusive to the DF2 or to the second data frame. So after we do this, we have to use the query method to pick only the data that is exclusive to DF2.

All right. Now let's see the output of this exclusive right join. And the output is the data that corresponds to the ids E and F. And we see that the H column is empty because the second data frame, also known as DF2, doesn't have this column. For that reason, we got null values in this H column.

All right. Now let's see how to implement this exclusive right join in pandas. So let's go to Jupyter notebook and here let's write the code. So I'm going to use DF1 and DF2 as I did in previous videos. And now I write df1.merge( ), then the second data frame df2, and then the on parameter here ID, then the how parameter equal to outer. So with this we make a full join or outer join. But now we want to make an exclusive right join. So we have to add the indicator parameter and set it equal to true. So this indicator will create the _merge column, and this _merge column will tell us which data is exclusive to the second data frame. And this type of data is represented by the right_only. So this means the data that only belongs to the right data frame, and in this case, the right data frame or the second data frame is the DF2.

Okay. Now let's create a query that only picks this right_only data inside the _merge column. So now I copy and paste this code here. And after I do this I'm going to create

a query. So here query then parentheses and now u I open these quotes then I write the name of the column actually I copy the name of the columns to make it faster. So now this column equal to and now the name of the data I want to pick in this case is the right underscore only and that's everything we have to do. Now we can run this cell and we're going to get only the data that is exclusive to the second data frame. So here you see that this data is exclusive to DF2. And with this we successfully created our exclusive right join. All right.

Now let's put everything we learned into practice solving an exercise. So in this exercise, you have to make a copy of the DF ratings data frame. And after you do that, you have to set the first 1,000 values of the ID column as the ID that I created here. This is an ID that I just came up with. And you have to set that value to the first 1,000 uh rows in the ID column. After you do that, you have to merge the DF_MOV and the DF ratings_2. This data comes from the first step, which is this one. And the type of joint you have to make is the exclusive right joint. And finally, you have to calculate the shape of all the data frames involved in this exercise. So we can draw some conclusions. And that's everything you need to do in this exercise. So now try to solve this on your own and after you finish you can continue watching this video to see my solution.

Okay. To solve this exercise I'm going to start by making a copy of the DF ratings data frame and I'm going to use the copy method. So I write df_ratings.copy() and with this we're going to create an independent copy. So now I'm going to set a name for this data frame and I'm going to name it DF ratings_2. Now we run and we created a copy of the DF ratings data frame. Then to set the first 10,000 values of the ID column as this ID that I created. Uh I'm going to use a for loop. So here I write for index in df_ratings_2.index: So we're going to loop through that df_r ratings_2 dataf frame. So now I write : and here um I'm going to write a condition. This condition will limit the changes we're about to make to only the first 1,000 values. So we write if index < 1000: then execute this line of code and in this line of code we're going to change the value of the ID column and here I'm going to use the lock method. So I write df_ratings_2.loc[ and here I use the index and the column. The index comes from the index that I created here for this for loop. And the name of the column is going to be this column id. So I write the imb title ID and then set it equal to this ID. So now I copy this ID and I paste it. So just a recap in case you didn't understand what I did so far. First, we loop through the indexes of DF ratings_2. I'm going to show you the indexes here so you understand much better. So, the indexes start with zero and ends with uh 18,855. So, we're going to loop through all these values, but we're going to limit the changes that we make with the lock method to the first 1,000 indexes. So, with this, only the first 1,000 indexes will have this new ID value. Okay, now let's run this code. And now we change the values. And now, let's see this data frame. So I print df_ratings_2. And now you see that the first 1,000 rows in the ID column have this TT 1 2 3 4 5 6 7 8 9 0 value.

So now I'm going to merge that two data frames. So df_mov and df ratings_2. So now I write df_mov.merge( then the second data frame and then the on parameter which is equal to the column in common in this case is the id column as you might remember from previous videos. Then the how parameter = 'outer' because we want to make uh this exclusive join. So we have to set it to outer and then we add the indicator = True. So now I'm going to make the query and to select only that data that is exclusive to the second data frame. I'm going to use this code that we previously created here. So this merge which is equal to right only allows me to pick only the data that is exclusive to the second data frame or to the data frame on the right. So now I copy this and after we do this we paste it inside the query. So now everything is ready we can run this code. So now we merge these two data frames and we see that we have some null values but that's fine because the second data frame doesn't have this title, years, genre and country columns. So it's fine. Now let's verify that we successfully made this exclusive right join by analyzing the shapes of the data frames. So now I'm going to set a name to this uh exclusive right join and I'm going to name it DF_exclusive join. So now I run again and now we created this data frame. So now let's examine the shape. So now I write this.shape. Now I paste twice and now I write the names of the other data frames. So DF ratings_2.shape and also df_movies.shape. So now it's ready. I'm going to put this first. And now let's print this. So we can see the shapes of the three data frames. So I print(). And now I run this code. and we see the shape of these three data frames. All right. Now, we see that the data frame we got after making the exclusive right join only has 1,000 rows. And this happens because only this 1,000 rows is exclusive to the DF ratings_2, which is our second data frame or let's say the data frame on the right. So, this data is exclusive to this second data frame. And for that reason, we only got 1,000 rows in this data frame. And that's it. In this video, we learn how to make the right join and exclusive right join in pandas.

All right. In this video, we're going to learn some meta characters used in regular expressions. So in regular expressions, meta characters are understood as characters with a special meaning like the \d, \w, and \s. And in this video, we're going to see the meaning of these characters and also see some examples to easily explain you how regular expressions work in Python. I have on the left a website called regx101. And this website allows me to test some regular expressions. And I have here a text that we're going to use as example. And on the right I have a list of meta characters with their meaning. So first I'm going to explain you the meaning of each meta character. And after that you can remember the meaning by looking at the right side. After we learn all these special characters in Python, we're going to go back to Jupyter Notebook and test some regular expressions using the Jupyter Notebook text editor.

Okay, now let's start with the first meta character. And this one is the \d or backslash D. And this meta character help us get the digits from a text. So the numbers from 0 to 9. So let's test it out. So the only thing I have to do here is to write the backslash followed by the D. And as you can see here on the left we have the numbers highlighted in blue which indicates that this regular expression \d gets all the digits from 0 to 9. Okay. Now this \d as most meta characters have their negation and their negation get the opposite values of each meta character. Okay. Now let's get the negation of \d. In this case the negation of \d is \D. So this one. So if we write \D, we're going to get not digits. So we're going to get from the text all the elements or all the characters that are not digits. So let's try it out. So here I write \D. And now you see that all the characters that are not digits were matched. And this includes even white spaces and also this uh hyphen. So anything that is not a digit will be matched if we use this \D. Okay.

Now let's see the next meta character. And now is the \w and this one means word character. So this includes letter from A to Z and then from A to Z but now in uppercase and also the digits from 0 to 9 and also the underscore. So keep this in mind. I used to forget about this underscore but this underscore is included as a word character. So now let's try it out. So here I'm going to test it out. Here I write \w. So as you can see all the word characters were matched and those that weren't matched aren't word characters. For example here the period is not a word character. Also these exclamation marks are not word characters. Also the hyphens are not word characters and the rest is a word character because we have here digits and also we have here letters either in uppercase or in lower case. And to get the negation of this \w we have to use the \W and now we get not a word character. So this \W is not a word character as you can see here on the right. So now let's write this \W and here we see that these spaces here this is an space this isn't a word character so it was matched also the period and the exclamation marks as we've seen before and well keep that in mind that this blank space counts as a character and in this case is not a word character because sometimes people used to forget that the blank space is there but yeah is there and is not a word character and now it's time for the \s and this \s means wide space and this includes a space also a tab and new line so if you don't know what I'm talking about well a space is only this is how I make a space then the top is this and the new line is only when we press enter and we get a new line. Well, it's as simple as that. So now let's try this out. So here I write \s. So here I have \s. And now we matched all the white spaces. So this white space between the words hello and world. Also this new line that is after this period. And then we have a lot of um blank spaces. And then uh again the here the new line and so on. Just keep in mind that this last line this one doesn't have a new line because here we don't have like a next line. We don't have a new line. But if I press enter here we automatically get this uh blue selection. So this is in blue because now we pressed enter and we created a new line. And if I press the delete key, we'll see that that new line disappeared and is not matched anymore. Okay, the negation of this \s is the \S. In this case, it's not y space. So this means like the opposite of whitespace as simple as that. So now I write \S. And as you might expect, we get everything that is not uh a space or a new line or a tap. So this is what we got.

Okay. Now let's see another meta character. And now it's time for the . So this period or dot sign matches any character except for new line. So when we write this dot sign, we're going to match any character that is not a new line. So now let's test it out. So here I write the . and here we matched all the characters except for the new line. So here you see that we match this uh space and also the . and also this exclamation marks but not the new line. So here this new line that is supposed to be here after the period is not matched. and also all the new lines that are at the end of each sentence, they are not matched. Okay. Now, there isn't such a thing like the negation of this . But if you want to ignore these special characters and not only this, but any special meta character, you can use only this \ to ignore any special character. So, let's try it out with this dot sign. So now this . sign is a meta character. So it's a special character in regular expression. But now if we add the \ here, we're going to see that we don't get any more any character except for new line. So as you might remember, this . sign gets any character except for new line. But now I added the \ and this isn't true anymore because this \ ignores any special character. So this means that we escaped that meta character and now this is only a regular character. So the only thing that matches is this regular . sign. So is the only thing that matched and the rest didn't match because this is only an average . sign. And that's how the \ works in regular expressions.

Okay. Now let's see some special symbols. And we have two special symbols that help us match elements at the beginning or at the end of a string. So first let's start with the ^ symbol which is this one that you see on the right. And this symbol matches the beginning of a string. So let's try it out here. Let's see how it works. So let's say that we want to match the word hello which is this one. So if we want to do that we only have to write hello and that's it. Yeah we can get the hello world. But now what happens if we have this. So we don't have one hello but we have two hellos. So here we have hello world and again hello. So if we only want to match the first hello, we have to use the ^ symbol to match only the beginning of a string. So here if I write a ^, we'll see that the only element that matches is the first hello because here is the beginning of the string. So it starts in this part matching elements and now it ends here. And since this hello isn't at the beginning of this sentence is not considered for this ^ symbol. But what if we want to match the second hello? So this one. Well, in this case we can use that $ symbol which is this one. And this matches the end of a string. So if we write here uh $ but now first delete the ^ and write the $ we'll see that well nothing matches because this first sentence doesn't end in hello but ends in hello with the period. So we have to add the period and now you see that we match this hello with the period that is at the end of the string because of this $ sign. Okay. Now let's delete this second hello. So we have the sentence as it was in the beginning. And now I want to show you that we can easily match the words hello and world by only writing these words. So for example, if we write hello, we match the word hello in the test string. And if we write the word world, we match the word world here in the test string. So that obviously makes sense. But here the power of the ^ and the $ sign comes when you have repetitive values in the string. So you only want to match either the words at the beginning or the words at the end but not the words in the middle. For example, you want to get only the last exclamation mark that is in this second sentence. So if we only write uh this !, we're going to get only the last exclamation mark and not the others. So if for some reason you want to get only the last element, that's when you can use this $ symbol or also the ^ symbol when you want to get it at the beginning. And something important I want to mention is that the regular expressions might change slightly their behavior based on some reg flags. So regular flax help us like edit the behavior of these regular expressions and here in this website you can see it in this site. So you only click here and you will see the regx options. So here I click and now we see all the rex flags. So here we have activated multi-line and this allows the ^ and $ symbol match start and the end of line. So this means that each line will be taken like a separate text which means that this is like one text and this is another text. So here we have one beginning of a string and here one end of a string and here in the second line we have another beginning of a string and now another end of a string and so on until the last sentence or the last line. Here also we have the beginning of a string and also the end of a string. So in total we'll have like 1 2 3 4 5 six beginnings and six ends of a string because we have six lines. And that happens because we turned on this multi-line. But if we turn it off. So I click here. Now we only have one beginning and one end of a string. So the beginning is here in hello and the end is here in four five six. So this happens because I turned off this multi-line. So we only have one beginning and one end of the string. And I think this is how Python behaves by default. We're going to see that later when we go to Jupyter notebook. But just keep that in mind. You can control these uh reg flags using a specific method in Python. And that's it. In this video, we learned some meta characters that we can use in regular expressions.

All right. In this video, we're going to see some quantifiers used in regular expressions. Some of the most common quantifiers in regular expressions are the *, the +, and the ?. So now, let's try them out. First with the *. So let's say we want to match the number of exclamation marks after the word python. So here we have the word python and we have five exclamation marks after the word python. So now let's match some of those exclamation marks using the quantifiers. So first with the *. So first here I write python! So now you see we have only the uh Python with !. So this match because it's exactly the same characters. And now if we use the *, we're going to see that here the * matches zero or more characters. And this * will have an effect on the closest character. So if we write here this *, this will count this ! zero or more times. So here as a result we got that the python and all the exclamation marks were matched and this happened because this * match zero or more characters. So in this case the character that was evaluated was this ! and we count them as many ! we wrote here in the text. So as a result we got the Python and the five !. So now let's see how the + will work. So this + matches one or more elements and I think this will behave similarly to the *. So let's find out. So here I write +. And now we see that it has the same behavior. So we got here the Python and the five !. So this + affected this ! and it capture one or more !. So here we have five ! and all of them were matched. So as a result we got Python and all !. And now let's see how the ? works. So in this case this ? matches zero or one character. So in this case if we write the ? we get only one because it says zero or one. And in this case there is one ! and that's why we got only the python and this first !. And now you might be wondering what does greedy and lazy mean? because here I wrote in parenthesis greedy for the * and for the + and lazy for the ?. Okay, first greedy means match the longest possible string. So it means match as many characters as you can. So for example here this * is greedy because it can match zero or more characters. But here if we write again the * we see that it matches the maximum number of characters. So it match the longest possible string because here it matched Python with five ! and the opposite

Of this is the question mark? Because if I write the question mark, it will match the shortest possible string to fulfill my requirement. So in this case, one is the minimum number of exclamation marks that will satisfy my regular expression. And also this plus sign is greedy because it behaves the same way the asterisk behaves. So if I write here the plus sign, we'll see that it will match as many characters as it can. Okay.

Now to understand this much better, we're going to see a different example. And in this case, we're going to use the dot sign. And as you might remember, this dot sign matches any character except for new lines. So here I'm going to write the word "hello". And as you can see, the word "hello" here in this string was matched. But I don't want to get that. But what I want to do is to get only the words between the H and the L. So to do that, I write here uh only the H and the L. And now I want to get everything that is in the middle. So here I can use the that dot sign and also I can use the plus sign because here if we use the plus sign, we're going to match one or more character. And if we use both the plus and the dot sign together, this means that we're going to match one or more characters except for the new line. So here if I write uh plus, we'll see that all the words between H and this L were matched. And this is kind of unexpected because here I got the words between this H and the L that belongs to the word "world". But I didn't want that. I only wanted to get the words between the H and this L that belongs to the word "hello".

So here we got this behavior because the plus sign is greedy. So it means it will match as many strings as it can. So we got the longest string, and in this case, the longest string is this uh from H to the last L. As a result, the regular expression we wrote here became a greedy regular expression because it matched the longest possible string. And if we want to make this regular expression lazy in order to get the shortest possible string, we have to convert this greedy expression into a lazy expression. And to do that, we only have to add this question mark. So this question mark converts any greedy regular expression into a lazy regular expression. So let's try this out. I'm going to add here the question mark after this plus sign. So here I write question mark. And now we see that we got the shortest possible string. So we didn't get from this H to this L. We didn't get from this H to this second L in the word "hello". But we only got the characters between the first H and the first L. And this is because we use the question mark. And this is a lazy operator, and it converted this greedy regular expression into a lazy regular expression. So anytime you want to get the shortest possible string, you only have to use a lazy operator like this question mark to convert a greedy regular expression into a lazy regular expression.

Okay, now that everything is clear about greedy and lazy matches, now we're going to continue with the quantifiers. So here the next quantifier we're going to see is this curly braces with a number inside. So this means exact number. So instead of getting zero or more or one or more or zero or one, now we can specify the exact number of characters that we want to match. So on the left I'm going to write again the "Python" with the exclamation mark. And now instead of writing asterisk or the plus sign or the question mark, now I write these curly braces. And now let's say we want to match three exclamation marks. So if that's the case, we only have to write the number three. And we got three exclamation marks. So "Python" and these three exclamation marks. And that's how you define the exact number of matches that you want for an specific character. So here you can modify. So now number two, and you get two exclamation marks or five, and you get five exclamation marks. So you can set the number you want, and these curly braces will match the exact number of characters that you're indicated inside the curly braces.

Okay. Now if you want to match more than n characters, so for example, if you want to match more than three characters, you only have to write the comma. So here let's go back to number three, and now here we match "Python" and three characters, but now let's say that we want more than three characters, so we're not sure about how many characters we want to get, we only know that is three or more, so if that's the is, you only write three and comma, and voila, you get five characters because you specify that you want it more than three. And this is kind of greedy. So it will match as many characters as it can. And with the next quantifier, you can specify a range of numbers. So the first element is going to be the minimum number of characters that you wish to match. And the last element is going to be the maximum number of characters that you wish to match. So let's say that we want to match between three and four characters. So if that's the case, we only write three, comma, four. And now we got four exclamation marks because it will always try to match as many characters as it can. And that's it. In this video, we learned how to use quantifiers in regular expressions.

Okay. Now let's see some meta characters that we also use often in Python. So first we have these parenthesis, and this represents groups. So to capture a specific group of characters, we only have to use this parenthesis. So now I'm going to show you an example so you understand much better how this works. Okay, for this example, let's create a regular expression that matches these numbers that we have here. So here we have nine numbers separated by this hyphen. So every three numbers we have one hyphen. So for this example, we're going to focus only on these three sentences. Okay. To do that, we have to use the \d because these are numbers. So we have to match digits, and the meta character that allows us to match digits is this uh \d. So here I write \d, and now we matched all the numbers. So this is fine. But what if we want to get the first group of three numbers? So in that case, we have to use a quantifier. So we have to use the plus operator. So let's write plus. And as you might remember, this plus allows us to match one or more character. So in this case, one or more digits, and that's why we got kind of the same result. But now if we add the hyphen, we're going to see that this behaves differently. So now we only got until the last hyphen. But now if we add this \d with a plus sign twice, we'll see something different. So here I paste this once. And now I add the hyphen. And now I paste this again. So here we have a digit then hyphen then digit and hyphen again and finally a group of digits. So each \d with a plus sign represents each group of three numbers separated by this hyphen. So we built this because we wanted to get only the first three numbers. So the first three that are here that I'm selecting right now, but I think it's not visible, but anyway, we want to get the first three numbers. So now if we want to get the first three numbers, now we have to add the parenthesis. So this parenthesis will help us like capture a group. So if I add parenthesis here and parentheses here, we'll see that now the first three numbers are in green and the rest are in blue. So that green means that we capture that group, and that's the first group that we capture. So let's capture the second group of numbers or the second uh group of digits that are between hyphens. So I add parenthesis and here again parenthesis. So now we see that the second capture group is in yellow. And if we add parenthesis to the third group, we'll see that this one is in purple. So here we have three groups, and we capture those groups by adding parenthesis. So here we'll see that it says in this uh box match two. But the important thing here is that below it says group one. So all the matches in green are group one because of the parenthesis. Then in yellow we have group two. And then in purple we have group three. So this is how the parenthesis meta character works. And this is an extremely useful meta character when you want to extract certain information from the whole text data. So let's say that you have a data frame and you have a column that has only information about the date. So if you match all the date, you can capture only the years. So if you capture only the years, you can extract those years and create a new column based on that year.

Okay. Now it's time to see the square brackets meta character. So this meta character matches characters in brackets. So unlike parentheses, square brackets don't capture an expression but only match anything inside it. So for example, we can write a square brackets and inside write the numbers 7, 8, and 9. And we see that all the 7, 8, and 9 digits were matched. So here we have a number seven. And here we have the number eight, and here the number nine. So this doesn't mean that we have the 7, 8, and 9 together, but it means that we match anything that is inside the square bracket. So this is like match 7 or 8 or 9, and instead of writing 7, 8, 9, we can write 7-9 and get the same result. So this hyphen means get a range from 7 to 9. So anything that is between 7 and 9 will be included. So if now we write from 5 to 9. So we're going to get 5, 6, 7, 8, and 9. So all those digits will be matched. And the same goes for letters. So if we write A to Z, we're going to get all the letters that are in lowercase and that are between the letter A and the letter Z in the alphabet. So here in this string, you see that all the letters in lowercase were matched. But those in uppercase were not matched. And if you want to match the letters in uppercase, you only have to specify here A in uppercase and then hyphen and then Z in uppercase. So here Z in uppercase. So this indicates that we want not only letters in lowercase but also letters in uppercase. And that's how the square brackets work. And now if you add a carrot in front of these square brackets. So at the beginning of the square brackets, this will modify the behavior of these square brackets. So when the carrot is at the beginning of the square brackets, this means match characters not in brackets. So this is the symbol. So it starts with square brackets, then carrot and then anything that you want to include and then square brackets. So if we add here for example the carrot symbol. So this means match anything that is between A to Z in lowercase and in uppercase. And as you can see here the spaces, the dot sign, the exclamation mark, and the digits were matched. So now if we write the carrot and the range from 0 to 9, we're going to get anything but digits. And this is the result. All the characters in blue are non-digits.

And as a final note, keep in mind that the expressions that are inside the square brackets can get really long, and sometimes people get scared because it's hard to understand what this regular expression means. But remember that it doesn't matter how long the regular expression gets because if it's within a square brackets, you can split them as groups. And when you split them as groups, you can think of each group that is separated with an or condition. So here for example, if I write 0 to 9 and then I write A to Z and then A to Z but in uppercase. So we can separate this um these characters with the or condition. So from 0 to 9 or from A to Z in lowercase or from A to Z in uppercase, and if you want we can add more and more characters, and it doesn't matter how long it gets, you can split them and add a or condition in the middle, and you can understand anything inside square brackets in that way.

And speaking of or conditions, let's review how this or condition works in regular expression. So now I delete this, and to make an or condition in regular expressions, we have to write this sign. So this is the equivalent of the or in Python, and it works the same way. So let's see it in action. So here I'm going to match a string that has 9 or 8 followed by two digits. So to do that, we have to write uh 9 and here the or and 8. So with this we match both 9 and 8. And now if we want to get two digits that are followed by this ninth or eight digits, we have to add here the uh backslash and then the d and here write this uh curly braces with the number two. But now we have to group this 9 or 8. So only this nine and eight digits are evaluated by this or condition and not nine with this whole text because right now this or operator is evaluating nine with this eight and this \d and we don't want that. So we have to group this. So we add parenthesis and here we group this nine or eight. So now you see that we got the correct match. So we wanted to match a string that has 9 or 8 followed by two digits, and we got that. So here we got this 9 or 8 followed by two digits. So the text that is highlighted in blue is the matched, and the text that is highlighted in green is the capture group because we use the parenthesis, and that's why we capture this group. And in case you don't want to capture a group, you have another alternative. So instead of using parenthesis, you can use the square brackets. So here I write square brackets. So this matches a string that has 9 or eight followed by two digits, but now without capturing the number nine or the number eight. So you see now that the text is highlighted in blue, and nothing is highlighted in green because we don't have any capture group. As a side note, I want to remind you that it is not necessary to add the or inside a square brackets because each character that is inside square brackets is split and then evaluated with an or operator. So if we delete this or operator, we get only this uh 9 and 8, apparently we got 98, but this in reality is 9 or 8.

Okay. Now let's see what this \b means in regular expression. So this \b symbol is another meta character that matches at the position between a word character. So the \w and a nonword character. So the \W. This is called a word boundary. And now let's see how it works. So here I write \b. And now we see that we have multiple characters matched in purple. So all of them are word boundaries. Okay. And among those purple characters that matched, we have three different positions that qualify as word boundaries. So the first position is the position that is before the first character in the string. So this one that is before the h in "hello". Okay. And the second position that qualifies as word boundary is after the last character in a string. If the last character is a word character, and the third position that qualifies as word boundary is between two characters in a string where one is a word character and the other is a nonword character. And a good example of this is the hyphen between the 987 and the 654. So this seven and this six are numbers. So those are word characters. But the hyphen is not a word character. And as a result, we got these two word boundaries that are in purple. Okay. Now, as you might expect, this \b has a negation and is a \B but now in uppercase. And this meta character will match no word boundary. So if I write \B, now we get all the elements that are not word boundaries.

Okay. Before finishing this video, let's see how back references work in regular expression. A back reference helps us reuse a capture group we already identified. And now to understand this much better, let's capture this pattern. We're going to capture this 1, 2, 3 with a hyphen to explain you much better how this back references work. So let's write here 1, 2, 3, and a hyphen. So you see here that we matched all the 1, 2, 3 hyphens. So here's the first match. Here's the second and here's the third. But we want to match only repetitive 1, 2, 3. So this means that we want to match this 1, 2, 3-1, 2, 3-. And all of these should be just one match. So this means that we don't want to match this 1, 2, 3 with hyphen. that is alone because that's not a repetitive 1, 2, 3 with hyphen. So if we want to do that, first we have to capture this 1, 2, 3. So we open parenthesis, and in this way we capture this 1, 2, 3. But this is not ready here. We have to use this backslash reference. So we have to add the backslash with the number one. And this indicates that we want to repeat that reference. And as a result, we got here the 1, 2, 3 and also this 1, 2, 3. And all of this is a single match. So what we did here is to repeat everything that is inside this parenthesis. So this is like copying and pasting 1, 2, 3- here instead of the \1. So if I do this, it will have the same result. So this is the same, but sometimes you don't want to repeat the same expression twice, but you want to use like a shortcut or you want to reduce a long capture group, and when you want to do that, you only have to use the \1. So that indicates that we want to use the reference that is inside the parenthesis. And now if we add another \1, nothing is matched because now we don't have a sequence of 1, 2, 3 with hyphen three times but only we have a sequence twice. So this and this. But now if I write here just to show you uh I create a new line and now 1, 2, 3 with hyphen. Now we see that we have this match, and we got this match because now we have a sequence of 1, 2, 3 with hyphen three times, and as a result, all of this represents a single match, and that's it. That's how the back references work in regular expression. And now if you have more than one capture group, you can also specify which group you want to reduce by writing its order. So let's say that you don't have one capture group but two. So instead of this 1, 2, 3- we have let's say 4, 5, 6-. So in this case, this is my first capture group, and this is my second capture group. So if now we want to reference the second capture group. So 4, 5, 6 with hyphen. We have to write backslash and the number two. So this indicates that we want to reduce this 4, 5, 6 with hyphen while the \1 still represents the 1, 2, 3-. So you only have to consider the position of the capture group in this regular expression.

Okay, that's it for this video. In the next video, we're going to see some examples and exercises so we can understand much better all the meta characters we learned so far.

All right. In this video, we're going to use all the meta characters that we learned so far, but now using Python code. So, when we want to use regular expressions in Python, we have to import the `re` module. This `re` module is a Python builtin package that is used to work with regular expressions. Okay. First we need to import `re` to see all the regular expression methods we're going to see in this video. So first we write `import re`, then we run, and now we imported this module. And now we're going to see two common methods that are used to match regular expressions. The first one is the `search` method. And here I have this `text` variable. And this has the same text that we used in the previous videos. So the same "hello world" and "I love Python" and all the same content. So this is the same text. And now let's say that we want to match only the digits. So here we want to return the first digit.

that is matched in this text. And we can use the search method to do that. So we only have to write research. Then we open parenthesis, and first we need to write the regular expression that matches the digits. So, as you might remember, to match digits we need to write the backslash and the d. So this matches digits in a string. And to specify that this is a regular expression and not average text, we have to open quotes. And here, before the quote, we have to write the letter R. So this letter R before the quotes indicates that what we have here is a regular expression. Okay.

After this, the second argument is going to be the text we want to evaluate. So in this case, we write the text variable. So I copy and paste it here. And that's everything we need. So let's review this uh method. First, we write re.search, and the first argument is the regular expression. We indicate that this is a regular expression by writing the r before the quotes. And the second argument is the text we want to evaluate. And that's everything you need. So now I run this. But first, I run the text variable. And now I run this. And now we got this that says re match object. And here we have a match equal to nine. So this means that we only got one matched that was this first number nine. So the search method only finds the first match in the whole text. So even if there are more digits in this text, it won't find all the digits but only the first digit that appears in the text. So if we check here in hello world, we don't have any text in I love Python neither, but in the third sentence we have many digits, and this one starts with nine, so the only digit that is going to be returned is the number nine, and here we got the number nth, and the rest was omitted. Great.

Now let's see how flags work when we write Python code. So before, when we were in this website, we saw that here in this option we got all these flags. So by default, we had the global and the multi-line flag turned on. And here we can control the flags using the flags parameter inside the search method. So let's say we want to get the word hello. So this word that says hello here. So let's use the search method. So we write re.search. Now open parenthesis. Now we write the r and here the quotes. So we indicate that what we have here is a regular expression. So now let's say that we want to match anything that starts with hello. But in this case, let's write hello in lowercase. So this is my first argument. And now the second argument is going to be the text because we want to evaluate this text variable. And in the next argument, we're going to consider the flags parameter. So we write flags. And now equal to. And now we have to specify which flag we want to turn on. So here we write re, and let's say we want to ignore the case of the text. So if we want to ignore the case of the text, we have to write here the I, which stands for insensitive. So with this, we won't consider the case of the text. So now if I run, we get that the match is hello. So here, even though this hello is with H in capital letter, we got the match because here we turn on the insensitive flag. However, if we didn't include this flag parameter, so if we only wrote these two arguments, we wouldn't get any match. So, as you can see, we didn't get any match because here this hello is in lower case and here this hello is in uppercase. And that's why we didn't get any match. But with the flax parameter, we got a match because we ignored the case of the text. And by the way, you can see the complete list of flags here in this website that we checked before. So, for example, I knew that the I meant insensitive because here the I is highlighted. And if we want to turn on the multi-line flag, we only have to write the M because here the M is highlighted. So here we could write the M, and that indicates that we want to turn on the multi-line. And you can check all the flags in this list here. Okay.

Now let's see how the find all method works. So unlike the search method, the find all method returns all the matches. So let's find all the digits inside this text. So I'm going to copy this uh re.arch. And now instead of writing research, now we're going to write re.find.findall. So the regular expression is going to be the same and also the variable. So now if we run this, we see that now we get all the digits inside that text variable. So 9 8 7 6 54 and so on. So as you can see here, these are the exact numbers that you see here, and they are listed in the order that appear here in this text variable. So it starts with 98 and it ends with 56. So let's check it out. 98 or knight and 8, and then it ends with five and six. So it's correct, and we got all the matches using the find all method. And that's it. In this video, we learn how the search and the find all method work.

Okay, now it's time to put everything we learned so far into practice by solving these three exercises. So here are the tasks. First, we have to match only punctuations from this text variable. Then we have to match the right date format. So in this case, the format is month, date, and year. And we have to match the right date format from this text variable. And the last exercise is to match the right username format. In this case, the right format is from four to 14 characters and only letters or digits. So here is the text, and you have to match the right username format. For these exercises, I will recommend you to use the find all method that we learned in the previous videos with all the method characters that we learned in the previous videos too. So try to solve these exercises yourself, and after that check my solution.

Okay, let's start by solving this first exercise, and we have to match only punctuation. So that means that we have to match only this dot sign, the here, the exclamation marks, and this hyphen, for example. So to do that, first we write refind all and then parenthesis. Here I already imported the re module. So you have to do the same. So here the first thing we have to do is to build the regular expression that matches only punctuation. And here we have to analyze what's the best approach. And actually, there are many approaches, and I don't think one is much better than other, but maybe one is shorter than other, but what I'm going to do here is a simple approach, and this consists in using the square brackets. So here I'm going to write it, and I write square brackets. So this matches any character that is inside. But now if we add the carrot, we don't match any character that is inside. So this means match characters not in brackets. So now what I'm going to write inside this square brackets are meta characters that don't represent punctuation. For example, the backslash w, this one represents word characters. So as we've seen before, this W represents uh word characters like the A to Z in lower case, then A to Z in uppercase, digits from 0 to 9, and also this underscore, and all of them are characters that are not punctuation. So we have a good number of nonpunctuation characters there. And another character that is not a punctuation character and also that is not included here in this back slashw is this backslash s. So this backslash s means a space. So here we have word characters, and here we have a space. So if we now insert this back slashw and also this back slash s inside these square brackets followed by this carrot, this means match any character that is not a word character and that is not an space, and what it remains is only punctuation because if you think anything that is not a word character and anything that is not an space is only the punctuation. So let's test it out. And here I copy this regular expression. So now I write the R and then open quotes, and inside I paste this. So this R means that we want to write a regular expression, and what is inside is what we created. So this is match anything that is not a word character and that is not an space, that is match only punctuation. So now the second argument is going to be the text. So this is the text variable that we want to evaluate. And now everything is ready. So we can run this cell. So now I run. And here we have a list of only punctuation signs. For example, the dot sign, the exclamation mark, the hyphen, and so on.

Okay. In this second exercise, we have to build a regular expression that matches the right format of a date. And in this case, the right format that I specified is here within parenthesis. So it should be month, date, and years. So here what we have to do first is to write re then find all then parenthesis then we write the r followed by the quotes, and inside we write the regular expression. So let's build this regular expression. So this date format should have these two digits followed by this forward slash and then this again two digits again forward slash and finally four digits. So we can create that using the back slashd that represents digits. So we write back slash d, and now this represents digits, but now we have to give it this shape. So first two digits. So to do that we open this curly braces and we write two. So this indicates that we want only two digits. And now I copy this and I paste it twice. So one and two. And the second is back slash D with number two inside curly braces. And that remains the same because we need two digits. But the last one should be four inside curly braces because the years need four numbers or four digits. So we write four. And now we have the format. And now I just realized I made a little mistake here because I wrote this forward slash, but I should have written here a hyphen. So instead of forward slash, we write here hyphen and here hyphen again. And we do that because here in the text we have hyphen between numbers. So it should have the same format. So the final format should be two digits then hyphen then two digits then hyphen and finally four digits. Okay. Now to finish this regular expression, we have to add the hyphen between the digits. So here one hyphen and here the other hyphen. And now this is ready. So I cut this and I paste it here. So here we have our regular expression. And now we just have to add the second argument which is the text. And now if we press Ctrl enter, we'll see that only the first row. So this one has the right date format because this one has two digits, two digits, and four digits, and the others have four digits at the beginning. So these two have wrong date format, and this one has the right date format, and this is why we got here this 13-4-2021.

Okay. Now let's solve the last exercise, and in this last exercise we have to match right username format, and the format indicates from four to 14 characters and only letters or digits. So here we have the text, and now we start by writing re.findall to find all the matches and now here are and quotes and inside the regular expression. Okay, we have to get only letters or digits, and that indicates that we have to get letters from A to Z or digits from 0 to 9. And to make both of them work together, we have to put it inside square brackets. So as you might remember, these square brackets matches characters inside the brackets. And any group that is inside these square brackets are split and then evaluated with an or condition. So this means that we'll get here characters from A to Z or characters from 0 to 9. And now if we want to get uh four or 14 characters, we only have to add the quantifier with this um curly braces. And now indicate four as a minimum and 14 as maximum. And we separate them with comma. And now this is ready. If you want, we can add also the A and Z in uppercase. So this is like more complete. So this is my regular expression. And now I'm going to copy and paste this one inside the R with a quotes. Okay. Now we only have to add the second argument which is the text. Now we run, and we got that the only username that matched was this username 10 because it has between four and 14 characters while the others have either three or two characters. And that's it. I hope you successfully solve these three exercises.

Okay, in this video we're going to see the data frame that we're going to work with in this project. So the data set is about Netflix movies and also Netflix TV shows. And here I'm going to read it. So first I import pandas as PD. And now to read this Netflix data set, I'm going to write PD.read_csv. And inside parenthesis, I'm going to write the name of the data frame, which is Netflix titles. So this one. So now this is the name of my CSV file. So I read it. Now I run this. And here we get the data frame. I chose this Netflix data set in purpose because it contains missing data and also it's going to help us apply some data cleaning techniques with pandas. So let's start with the first column, and here we can see the ID, and ids usually are unique, and here we can see the values, and then the next column here is the type, and here we can see movies and TV shows. Then we have the title of the movie or title of the TV show, the director, the cast, the country, the date, and the release year. And here we're going to use this column a lot to know whether an element of this data set is actually a movie or not. Then we have the categories of the elements and then the description of the movie or TV show. Okay. Now I'm going to name this uh data frame, and I'm going to name it DF_Netflix. So this is my data frame. So now I run this, and now let's see the data types of the columns of this Netflix data frame. So to do this, we have to use an attribute that we've seen before in this course, and this is the dtypes attribute. So this attribute allows us to know the data types of each column. So here I'm going to write DF_Netflix and then dtypes. And now I press Ctrl enter to know the data types of each column. And here we can see that the only integer column we have is the release_year. So this means that this is the only column that has numerical data. And that's kind of weird because here the duration and also the date added should have some kind of integer or date format. But here we can see that it has object as data type. So something weird is going on there, and we'll see that in detail in the following videos. But now let's check the shape of the data frame. And by shape, I mean the number of rows and the number of columns that the data frame has. And we can get this uh by using the shape attribute. So we only have to write df_netflix. So the name of the data frame followed by that shape. So this is the shape attribute. And now if I run this, we can see the number of rows which is the first uh value here and the number of columns which is the second value here. So if we print the data frame df_Netflix, we will see that uh below there is the number of rows and also the number of columns. So we can verify that this is correct. And that's it for now. In the following videos, we will see how to identify missing data and how to deal with missing data.

All right. In this video, we're going to see how to identify missing data. So, first let's have a look again at the Netflix data frame here. So, here I write DF_Netflix, and here we get the data set. And here it's the data set, and we can see that there are some nan values. So this one, so this nan represents null values, and this means that there is no value here, for example, for this cast in index zero and also for this director in index one. So there is no data, and what we can do here is to use the isnull method to recognize them because here we can easily recognize them because it says nan, but to check that in the thousands of rows that this data frame has is almost impossible. So we can do it easily with the isnull method. So now I'm going to show you how to do it here. So first we write df_netflix, and now to find the null values we only have to write that isnull. So here I write that isnull, then I open parenthesis, and if I run this, you will see that we get the same data frame, but in this case we get false and true values. So the true values are those nan values, so null values, and the false values are nonnull values. So here we can see that, for example, again cast index zero it's true, and this is true because before we saw that this was an nan value, so a null value, and the same goes for director with index one is the same is true because this one is a null value. So here we can recognize all the null values inside a column by using the isnull with the sum method. So here I write sum, and now what this is going to do is to sum all the values that are true. So here, for example, here we got some true values. So if we apply this sum method, this is going to sum all the true values inside this directory column. So here, here we have true the first true value and then again another true. So 1 + 1 = 2, then 2 + 1, in this case we have another true value. So 2 + 1 = 3. So far we have three null values. and then it will find true values that represent the null values we've seen before, and it's going to sum all of them. So in the end, we're going to get that total number of null values per column. So here let's have a look. I'm going to uh run this one, and now we see that, for example, the ID column doesn't have any null values. However, the directory column has 2,634 null values, and also the cast and country columns have some null values. And it's important to recognize how many null values a column has because this tells us how our data our data set is structured. So if a data set has many null values in a column, that means that some columns might be incomplete. So we should think about dropping those columns. So deleting those columns, and for this example, the director, cast, and counter columns are good candidates to be deleted. So here I'm going to show you a way to sort first these null values here because here is sorted according to the original data frame. But now I want to sort these values ascending. So the columns that have the higher number of null values will be on top. So here I'm going to use the sort_values method for that. And here inside parenthesis I write ascending. And now I set this equal to false. So now I run this. I get the column with the highest number of null values on top. So in this case is the director column.

Okay. As we said before, these three columns are good candidates to be deleted. But still, we don't have enough evidence or enough reasons to delete these three columns because here we have 2,000 null values. But maybe these 2,000 null values represent a small portion of the total number of rows that the data frame has. So we have to verify the percentage that these null values represent from that total data frame. And to do so, we have to use the mean method. So, I'm going to show you how to do it here. But first, I'm going to copy the name of the data frame. And now I paste it here. So now I'm going to write the name of any column I want to evaluate. In this case, let's try with the directory column. So here I put the name inside quotes. And now I'm going to use again the isnull method. So here if I run this, we get the true and false values in this column or in

this series. So here again, the true values represent null values. And now, to get the percentage, we have to use the mean method. So we write mean, and what this is going to do is to calculate the average number of true values in this series. So it's going to sum all the true values and then divide by the total number of rows. So, in this case, 8,86 rows. So if I run this, we get 0.29. And this represents that 29% of the director column are null values. So this is a good number of null values. And we have to analyze later if we want to delete or drop this column. But first, let's see the percentage of null values of each column. And to do that, we have to use a for loop.

So here I write for, then I write column in df_netetflix.columns. So here I'm using the columns attribute, and now I'm going to show you how this works in case you forgot. So here I paste it, and this columns attribute should get all the columns that this data frame has. So if I run this, we get a list of all the columns. So from the ID to the description. So all the columns this data frame has. And now this is a list. So we can loop through it. So here in my for loop, I'm looping through this list. So here I'm saying for each column in this list of columns, do this. So execute this line of code. So instead of writing here the directory column, we have to introduce the column variable here. So in this way, we're going to iterate through all the columns that this Netflix data frame has. So we're going to get the percentage of all these columns. So the percentage of null values.

So here now, to show you this, I'm going to uh here assign a name for this uh percentage. So I'm going to name it percentage. And now I'm going to print this one. So I write print. And now I want to show the percentage of each column. So here print percentage. Now I run this. And now we get the numbers, and the column that has the highest percentage of null values is this one with 29% of null values. But we don't know which column corresponds to this value because it doesn't say anything. So we have to print the column variable. So here I write column and then + to concatenate these two variables. And now, since this percentage variable is a number, we have first to convert this into a string. So we write here uh str. And now we convert this number or this integer into a string. So now I think everything is fine. So I'm going to run this one. And here we see the name of the column and the percentage. So I'm going to add a column in the middle so we can see clearly the name of the column and also the percentage. So here I open this quotes and now the column symbol. So here uh I do this and now run, and now we can see clearly the name of the columns and also the percentages of null values. So here the column that has 29% of null values is the director column, as you can see here. So we can now even customize these numbers. So we can write here in percentage, multiply uh here with 100 and then round this number, and we want two decimals. So I'm going to write two decimals here, here comma, and here two decimals. So if I run this, we get the same values but now in percentages and with two decimals. And that's it.

In this video, we'll learn how to identify missing data. And in the next video, we're going to decide whether these columns that have a good number of null values are going to be deleted. All right.

In this video, we're going to see how to deal with missing data. So, in the previous video, we identified some columns that have a good number of null values. For example, this director column has 29% of null data. And here we're going to decide whether we're going to delete or drop those columns, or maybe if it's a better idea to drop rows. So to drop those null values inside those rows. So let's see what's the best solution for this case. So here I'm going to use three different methods, and these are the drop, the dropna, and the isnull methods. Some of them we already seen in this course, but some of them we haven't seen so far, but don't worry because I'm going to explain you how they work from scratch.

So here first, let's say that we want to drop this director column because we consider it has a good amount of null data. So here, if we want to do that, we have to use that drop method, and we only have to write the name of the data frame. So df_netetflix, and then we have to write then we open parenthesis, and inside we have to specify the name of the column we want to drop. So in this case is the director column because it has too much null data. So here I write director, and now here, as you might remember, we have to specify the axis. So here we say axis=1, and we write =1 because this is a column. So column is 1, and row is 0. So here we're saying drop this director column. So now it's ready, and we can run this one. So I run, and now we get a copy of the data frame. In this case, this data frame doesn't have the director column. So we successfully deleted this director column. But here I'm not going to take this approach because the director column might have some important data that we might use later. So it's not a good approach to drop the whole column because it can contain some important data. So sometimes it's much better if you can get rid of that specific rows that contain that null data. And that's what we're going to do right now.

So here I'm going to write df_netflix. And now I'm going to find which rows are null. So here I'm going to write here director. So this is the director series. And now I use the isnull method. So here I'm going to get the null values inside the director series. And what we're going to do now is to filter this based on this condition. So here is my condition. So get the null values. And now I'm going to filter this. So with this, we get the data frame with null values. And if now I write .index, this .index is my attribute that gets the index of this data frame. And in this case, this data frame contains only null values. So this means that we're going to get the indexes that correspond to those null values. So here I'm going to run this one. So I comment this out and I run. And as you can see here, I got here a list of indexes, and it says 1, 3, 4, and so on. And these numbers represent the indexes that have null values inside the director column. So now let me show you here in the real data frame df_netflix. So now I run this one. And as you can see here in the director column, the null values are in row one or index one. Then row three or index three. Here we have nan. And then also in index four, nan again. So we should get 1, 3, and 4 because those indexes contain null values inside the director column. So now let's verify if that's true. So here we got 1, 3, 4, and so on. So all these elements inside this list are indexes that correspond to null values inside the director column. So here I'm going to name this as no_director, which means that the values inside this expression are null. So they don't have a directory value. So no_director. And now I'm going to drop these indexes because, as you might remember, these are a list of indexes that are right here. And now I'm going to drop those indexes. So what I'm going to do is write the name of the data frame as I did here. So actually I'm going to copy this one. So I have uh this to edit. So here instead of writing the name of the column director, I'm going to write the variable no_director. So here I paste it. And now instead of saying axis=1, I have to change this to axis=0 because 0 represents rows or indexes. And here we have a list of indexes. So now everything is ready. And now let's run this one. I'm going to comment this out. And I run. And here we got this data frame. And as you can see here in the indexes, we have 0. But then we don't have 1 because we removed or we dropped that index. Then we have 2. And then we have 5. But 3 and 4 are not here because those rows contain null values inside the director column. So that's why we dropped it. And now we can easily verify how many null values are in the director column using the same method we used in the previous video. So here what I can do is to write .isnull().sum(). So now let me show you here I run this, and here I go to that director column, and as you can see here it says 0, which means that this column right now doesn't have any null value, and this is because here we drop all the null indexes using the drop method and this no_director list. So now I'm going to get rid of this. And here we have the data frame without null values in the director column. And that's it. I consider this a better method to deal with missing data. So I will add here an inplace=True. So you can add this one. So you save all the changes we made with the drop method, and you update the values in the df_netetflix data frame. But I'm not going to do that because I need these null values to make the following examples. So I'm not doing that only for the sake of this video. But you should do it if you want because that's a better approach than dropping the whole column as we did with this line of code. So not this one, but we're going to take this approach. So this one is much better. But in my case, I'm not going to add the inplace parameter. So here I delete it. And now let's continue with the following example.

So here I want to show you how to do the same thing. So how to drop rows, but now only with the isnull method. So here before we use the isnull method with the drop method to drop these rows. But now I'm going to use the isnull method with the not operator. So let me show you how to do it here. So here I have to write first the name of the data frame, which is df_netetflix, and then we open square brackets, and actually we're going to do this filter is exactly the same as we did before. So I copy this one and I paste it here inside square brackets. So now instead of getting the index as we did before, we're going to use the not operator. So here I'm going to add parenthesis, and outside parenthesis I'm going to add the not operator. So this one. So here inside parenthesis I'm saying I want to get the null values inside the director column. But now when we use the not operator, we're saying hey, I don't want the null values, but I want the non-null values. So here by using the not operator, we get the non-null values, and after this we get the filter and we get the data frame with non-null values. So it's like we're dropping the null values in the rows. So it's like we were doing this part, but in this case we're just doing it with the not operator. So if I run this one, let me show you what happens. So I comment this out. And now I run. And here we can see that index 1 is not here. Index 3 and 4 is not here. And now if I use the here the .isnull().sum() to get the number of null rows, we're going to get that the director column doesn't have any null value. So now I'm going to delete this one. And as you can see here, the isnull with the not operator does a similar job as this isnull with the drop method. Okay.

Finally, I'm going to show you a third way to drop null values. But in this case, we're going to use the dropna method. And this is the simplest method. And I'm going to show you here. First we write the name of the data frame followed by the dropna method. So we open parenthesis, and now we have to specify the column in which we want to analyze whether there are null values or not. So here I write the name of the column. So in this case it's going to be the director column. And here I'm going to add these square brackets, and also add a parameter called subset. So here I write subset=, and this is the column that I want to analyze. So we do this because that's the syntax of the dropna method. So we have to write a subset parameter then = and then open this square brackets. So if this director column has null values, we're going to drop those null values by using this dropna method. So now let's check it out. I'm going to comment this out and these two out. And now I run this, and here we can see that we don't have this one, index 3 and 4 are gone too. So now fast I verify the number of rows that are null values. So here .isnull().sum(), and here we can see again that the director column doesn't have any null values. And that's it. In this video we learned different ways to remove a column or row with a drop, dropna, or isnull method, and also we learned best practices to deal with missing data in pandas.

In this video, we're going to see how to deal with missing data by using the fillna method. Okay, instead of dropping rows and columns as we did in the previous video, we can deal with missing data by using the fillna method, and we can replace the null values by the mean or the median or the mode. And I'm going to show you how to do it now. So the first thing you have to do is to calculate either the mean, the median, or the mode of the column that we want to replace its null values. So here let's say we want to evaluate the rating column. So this rating column has some null values. So we want to replace the null values inside this column for the median or the mean, for example. So what we have to do is to write df_netflix, which is the name of the data frame, open square brackets, and here we write the name of the column. So we write rating, this is the column that we want to evaluate. So now to calculate the mode, let's use the mode in this example. So to calculate it, we have to write .mode(), and then parenthesis. So this is a method, and it helps us calculate the mode of this rating column. So let's see what's the mode of this rating column. So here I run, and by the way, the mode is the most common value inside a group of elements, and in this case this TV-MA is the most common value inside the rating column. So here we can use this value to replace the null values, and this is a practice that sometimes people use because if this value is already common, is already popular, it's not going to affect the end result. There are already many values that have TV-MA. So if we replace the null values by the mode, at least it shouldn't affect so much the result. So here what I'm going to do now is to assign this value to the null values, and to do that I'm going to use the fillna method. So what I'm going to do here is to again write the name of the column, and now I'm going to write the fillna method. So I write fillna and then open parenthesis. In here the argument that we have to introduce is the mode. So this one. But now I want to show you something before we do this fillna method. This mode value, as you can see here, is not a string. It's not an integer; it's some kind of array or some series. I mean, if you see the data type, it says object. So if we add this value as it is to the fillna method, we're going to get an error because this fillna method expects some kind of integer or string. So what we have to do here is to convert this object, and actually I think this is a series because it has this format, the value and the index. So we have to convert this into a string. So we can add it to this fillna method. So as an argument, and then it's going to replace the null values here. So to do that, I'm going to use the .join() method, and this join method allows us to join strings. And in this case, I'm going to use it to convert this value into a string. So what I'm going to do here is to add parenthesis. And now add here .join(). And this join method is going to join all the elements inside parenthesis. So here since we only have one element, we're going to get the same element but in string format. So the final format of this is going to be a string. And now we only need to add the separator. So I only open quotes. So in this way, this value inside the join method is going to be the same as the one we introduced. So this TV-MA is going to remain the same, but now the data type will be a string. So now let's try this out. I'm going to assign this to a variable which I'm going to name mode. So this is my mode variable. And now let's see the data type of this mode. So here I'm going to use the type function. And now use parenthesis. Run this one. And as you can see, it says str, which means that this now is a string. And now we can add it to the fillna method. So we now copy this mode and we paste it here. So now we have the mode. And now we can replace this null values by the mode. So now I'm going to add the inplace parameter, and I'm going to set it to True to update the values in this df_Netflix data frame. So now before I run this code, I want to show you fast how this rating column looks. So here this is the data frame, and the rating is this one. So let's see how many null values this rating column has. And to find the null values, we have to use the isnull method. But first we write the name of this column and then we use isnull() with parenthesis. And now to filter this, I'm going to insert this inside square brackets. So now I run. And now we get the rows that have null values and the rating column. So this one, uh these three, and this one. So we only have four rows with null values in the rating column. And what we're going to do is to replace this nan values by the mode. So now I don't need this, and let's do this here. So now everything is ready, and I'm going to run this one. So here I run. And now let's see how many null values this rating column has. So as you might remember, we only need to use the isnull() and the sum() methods. So first I delete this. Now I write .isnull().sum(). So now I run this, and here we get the columns and also the null values here on the right. So now we look for the column named rating, and it's here, and we get 0. So it means that this column doesn't have any null values anymore. So we successfully replace the null values by the mode. Now I delete this. And by the way, you can use the mean or the median. But in this case, I chose the mode because this rating column, I'm going to show you here, this rating column has text. So here all these values are text. So this is known as categorical data. And when we have categorical data, we have to replace null values by the mode. But when we have numeric data, like the column, for example, here the release_year, that's numbers. So that's numeric data. There we have to replace null values by the mean or the median. So in this case, for example, let's say this rating column has numbers, and we will use either the mean or the median, but in this case this one is categorical.

So here we use the mode. So just keep that in mind. Okay.

Now I'm going to show you how to replace null values by an arbitrary number. So sometimes, instead of replacing null values by the mean or the median, we use an arbitrary number and we replace null values with this arbitrary number. Usually, this number won't affect the final result. But it helps us work with the column as if all the values inside the column were non-null. And this is useful because when sometimes you have null values in a column with numeric data, you won't be able to do some operations because those null values won't allow you.

So now I'm going to show you how to do this. And as usual, we write first the name of the column. In this case, I'm going to use a column with numeric data. And this one is going to be the duration. So here I write duration. And before we apply the fillna method, I'm going to show you how this one looks. So here we got this column and you can see that it has a mix of numbers and text. So we can see that this column isn't numeric at all, but it has some numbers. And I'm going to add the number zero to this column because in the next videos, we're going to use some operations to split this data. So we get in one column only the numbers and in the other column only the text. And I'm going to show you how to do this, but in the next videos. But now I need the number zero because if I have null data in this column I won't be able to do this split. So that's why I'm going to replace null values by the number zero.

So now I write here .fillna(open parenthesis and the first thing we have to write is the arbitrary number, so in this case zero. I want to add number zero, and now I just write inplace=True, and that's it. With this, we replace the null values by this number zero, so I run this one now. Okay.

Now I'm going to show you a parameter that allows us to fill null values backwards and forwards. And first, we have to write this one again. Actually, I'm going to write—I'm going to copy all of this. And now I'm going to paste it here. So now I'm going to delete this—what is inside? And now, instead of using the mean, the median, an arbitrary number, or any other thing we've seen so far, we're going to use the forward or the backward non-null value to replace any null value. So to do that, I'm going to add the method parameter. So here, this method parameter has two options. So the first option that we're going to see is the backward option. So if I add this one, the null value will be replaced by the non-null value that follows it—so the one that is in the next position. So I'm going to show you here with an example. So I'm going to print the df_netflix. And here we have the data frame, and let's find the directory column. And here we see one null value. So if we apply the fillna method with the backward option, what we're going to get is that this director Julian L cleric will replace this null value. So it's going to replace backwards the non-null value Julian Llur will replace the null value backwards. But now if we use the same method, but now in the second option, forward. So here I write it. What we're going to get is that the first director here, Kristen Johnson, is going to replace this null value forward. So this non-null value will replace this null value forward. Okay.

And now that this is clear, I'm going to show you the true names of this method. So instead of writing backward, this one is called bfill. So backward fill, and this one is called ffill. So forward fill, and these are the names. And now I'm going to apply this to the whole data frame. So not only to the directory column but to the whole data frame. So now I delete this one and also I delete this one. So with this, all the columns will be affected by this bfill or ffill. So let's try first with the backward field. So now I run this one, and we see that this director Julian L clerk replaced this null value or this former null value because now it has the value of Julian L clerk. And now let's try with the method fill. So it's going to be the forward field. And now I run. And now it's different. So we see that this former null value is now Kirsten Johnson. So this director here replaced this former null value, and it was a forward field. This is usually used when we have a sequence of values in a column. So, for example, if we have the data of the weather every minute in a city, and for some reason we don't have data of one specific minute, we can either forward fill or backward fill because the weather shouldn't change so much in one minute. So in this case, it makes sense a lot to backward fill or forward fill. But in some other cases, it wouldn't make so much sense. So it depends on the data. So in some cases, maybe it's much better to fill with the median or the mode or the mean or just drop the column or the row as we've seen in the previous video. And that's it. In this video, we learned different ways to deal with missing data. Remember that each data set is different. So you have to analyze your data set and take the approach that fits best your data.

All right. In this video, we're going to learn how to extract data from a column. And in this case, we're going to extract data from the duration column using the split and extract methods. So first, I'm going to select the duration column. So I write the name of the data frame, df_Netflix, and then the name of the column. So duration. Now I run this, and now we can see that this data has the duration of movies and TV shows. So here we can see that durations of movies is in minutes, but the duration of TV shows is in seasons. So we have two seasons and three seasons and so on. So here we're going to analyze only the movies because we only want to compare in the following videos with minutes. So our unit of measure will be minutes, and we only need movies for this purpose. So what we're going to do now is to filter out TV shows. So we do this with the methods we learned in previous videos. So we write first df_Netflix. Then we write here the type of this uh show. So if it's a movie, it's going to be in this data frame. But if it's a TV show, it's not going to be in this data frame. So we're going to filter out TV shows. So here I write the name of this column, which is type. So here type, then equal to, and now write only movie. So here copy and paste movie. So now we only want movie, and this is our condition. And here we put it inside square brackets. And now I'm going to set this as a new data frame. And I'm going to name it df_movie. So this is my new data frame. And now I'm going to choose this data frame to make the selection. So now I paste this. And now let's select this duration column from the df_movie data frame. So now I run this. And as you can see here, we only get the duration of movies. And here we got the unit of measure, which is minutes. And this is great because now we can make a better comparison because all of them are in minutes. But this still isn't complete because here we have numbers and text. So to compare, we only need numbers because we compare one number with another number. But we cannot say that one element is greater or less than order with text. So here we have to extract only the numbers. And this is what we're going to do using the split and the extract methods.

So first, let's have a look at the split method. And to use the split method, we have first to get access to the str attribute. So as you might remember, this str stands for string, and this helps to work with string data inside this column. Okay. Now that we have the string of this duration column because we use the str attribute, we now can use the split method. So I write .split(), and by default the split method splits this data every time it encounters a blank space. So here, between the number and the text, we have a blank space. So every time it finds a blank space, it's going to split this whole text or this data. So now I'm going to run this one, and you're going to see the results. So here we get a list, and inside this list we have all the elements that were split. So here we see that we have the numbers on the left, and we have the text on the right. So here the text is minutes, and this is not so relevant. So we're going to select only the numbers—so the first element in the list. And one parameter that is used frequently in this split method is the expand parameter. So I write it here, and this parameter allows us to expand this list into columns. So if we have here two elements, if we write here expand=True, we're going to get two columns, and each column will contain an element of this list. So column number one will contain only the numbers, and column number two will contain only this text that says minutes. So here I run this, and as you can see, we have this data frame with 0 and 1 as columns, and here in the column 0 it says 90, 91, and only numbers, and here in the column 1 it only says minute, and that's what we wanted. Now we can select only the zero column by using square brackets and writing the name of the column. So now I write 0. And now if I run this, we get only the column with the numbers. So here we have the duration in minutes, but now only the numbers. Great. So far so good. But here there is a little detail. We see that all the data here are numbers. But here below, we see that the data type here in this dtype attribute is object. And this is a bit weird because all of the data that we see here are numbers. So we have to convert this data in this series into integers. So what we have to do here is to use the astype method. So we write astype, and this method allows us to convert from one data type to another. So here inside I write int, and this stands for integer. So now I'm going to convert this data, which is object, into an integer. So now I run this, and let's see the results. And I got an error. So let's see what happened. Now I scroll down, and I'm going to read the error, and it says cannot convert float NaN to integer. So this means that there are null values inside this column. So that's why it cannot convert to integer, and that's a bit weird because in the previous videos we replaced the null values inside this duration column by an arbitrary number, and if I'm not wrong, that number was zero. Yeah, it's here. So we replaced all the null values inside the duration column by the number zero here. And then we updated with this inplace=True. So what's going on? So there is a simple explanation to this here. When we use the fillna method to replace null values inside the duration column by this zero number, we set this zero number, which is an integer. But here below, you can see that here when we call this column, this duration column, we're getting access to the string data type inside this column, and here there is a conflict because here we're expecting strings, but here we added integers. So this is why when we use the split method here, it doesn't find any string when we get to the rows that have these zero values. So when it gets to those rows, it will detect some null values. So here when it makes the split, we get null values as a result too. So that's why when we use the astype method here to convert this data into integers, we get this error because it says that there are null values, and of course we cannot convert null values into integers. So here a simple solution is just to modify here what we added. So instead of adding this zero as an integer, we add this zero as a string. And to do that, we only have to add these quotes. And that's it. Now this zero is a string. And when we get access to this string attribute, this time we're going to get the zero because now zero is a string. And then we're going to split this successfully. So now let's try this out. I'm going to update this change. And to do that, I'm going to click in this option that says "Cell" and then click on the option that says "Run all above." And I want to do this because I want to start from scratch. So all the changes are properly executed. So now I click on "Run all above." Okay. After running all the cells above, I'm going to run this cell. So before I run the cells from the beginning to this one. So now I'm going to run this one to update this changes too. So I run this, and now we should get this zero integer instead of the null values. So now I scroll down, and now let's run this code and let's see if everything is fine now. So now I run, and as you can see here, I don't get any error. So now we got this series, and now this data type is integer. So everything is fine now. Okay. In the following videos, we're going to use this column. So I'm going to give it a name. So here I'm going to write df_movie. And now I'm going to give it a name. So I'm going to name it minute. And probably if we assign it this way, we're going to get a warning message, and it's going to be fine. So I'm going to show you here. Here you see that we got this SettingWithCopyWarning. And warnings are not errors. They just let us know that something is going on and that we should check it out. But it doesn't mean that we did something wrong. It's just a warning. So you can even turn off these warnings, but I don't recommend you because sometimes they are really useful. By the way, we could avoid this warning by using the assign method. But in this case, I wanted to simplify this and make this simple assignment, and it's fine. We don't have to worry about the warning. So now let's continue, and I'm going to show you how this data frame looks. So here I print this df_movie, and now the last column should be this minute column that you see here. And now we only have numeric data inside this column. So it's great. Now if we check the data type, we see that the minute column is integer. So it's great. That's what we wanted. So as I said before, we're going to use this minute column to do some analysis in the following videos. But now I want to continue with the split method because it has some other parameters that could be really useful when working with data. So now in this case, let's work with a different column. So now let's work with this date_added column. And this date_added column has some patterns. As you can see here, it has the name of the month. Then it has the number of the day and the year. So here we could split this date by the comma. So here if I write df_movie and then we write the name of the column here and then .str to get to the string data and now .split, we can use a separator. So here before we didn't add a separator because we want it to separate by the default blank space, but here we want to separate by a comma. So we write here .split and now I write comma inside this quotes. So if we do this, let's see what happens. So now I run, and as you can see here, we got a list, and inside the list we have two elements, and the two elements were separated by the comma. So we have the year here, and we have the month and the day, and we can even add the expand parameter to create two columns here. So I write expand=True. And now we see that we have two columns. And now if I select column number one, we get only the years. Great.

Now I'm going to show you how the extract method works. And this is a similar method, but it has some differences. So here I'm going to uh delete this. And now to use the extract method, first we have to get access to the string. So here we write .str. And after this we write .extract. So here .extract(then parenthesis, and inside we have to write the element that we want to extract. So, for example, if you want to extract only the commas, you have to write quotes and then write the comma, and here the only thing that you have to add is parenthesis because when we use the method, we have to add uh capture groups, and the capture groups are represented with parenthesis. So here I add parenthesis, and now we can extract only the commas. So here you can see that only the commas were extracted. But the extract method is really powerful when we use it with regular expressions. If you never heard of regular expressions, they are expressions that help us extract data from a text based on some patterns. So here I'm going to show you a pattern that extracts only the years, and the years only has four digits, right? So we can write the digit symbol that is represented by this uh backslash and the d. So this in regular expression represents the digit symbol. So now if we want four digit symbols like zeros have, we have to add these curly braces and write the number four, and this in regular expression means that we want four digits together, and that's what the years represent because the years are four numbers that are together. So I'm going to show you here, and here we have it again. A year is made of four numbers that are together. And this is different from the date because a date has only two numbers. So here I'm going to do this. I'm going to get these four digits together that represent the year. And before I run this code, I'm going to add this parenthesis because, as I told you before, when we use the extract method, we need to capture groups with parenthesis. So now I run this. And here I got a column with only the years. So 2021, 2016, 2019, and so on. And we got here this because we use this regular expression that represents four numbers together. And by the way, you can even use regular expressions in this split method. You only have to add it here inside quotes. And that's it. In this video, we learned how to extract data using the split and extract methods. Remember that in the following videos, we're going to use this minute column that we created in this video to identify outliers.

Okay, in this video, we're going to learn how to identify outliers. And we're going to use histograms to identify outliers within numerical data. So here first we're going to make a histogram from the minute column we created in the previous video. So I'm going to show you this column. So I write df_movie. And now I write the name of this column we created before. So it's this one. So this column has this numerical data that represents the duration of a movie. And now we can identify outliers inside this column. And in case you don't know what's an outlier, outliers are uncommon elements that you will see in a group of data. So, for example, in a movie, movies usually last between 1 hour and 2 hours. And if you find a movie that has like 30 minutes of duration or a movie that has like 4 hours of duration, that's extremely uncommon. So those types of movies will be considered outliers, and the criteria to select outliers will depend a lot on your data set. Okay. Now let's find these outliers inside this minute column by using histograms. So we're going to use the plot method that we learned in previous videos. So this plot method helps us make visualizations in pandas. So here we need to add the kind parameter, and here we have to set it equal to hist to make a histogram. Now I'm going to add the bins parameter to specify how many bins this histogram will have. And in this case, I'm going to set it to 10 bins. So

Now it's ready. So I run this, and here we get this histogram.

So here it's easy to recognize that the outliers are here in the first bar and in these tiny bars that you can barely see from 200 to 300. You can say that those are outliers because those bars have almost no values inside. So only a small proportion of the data are in those ranges. So if there are little data in those ranges, we can say that the data that is there are outliers. Okay.

So we know that here in the first bar and in the last two bars, we have outliers. But now it'll be great if we can recognize the ranges. So here in the first bar, we know that it starts with zero. But we don't know what's the value that this range ends because it doesn't say anything here in the x-axis. So we can get these values here that should be here with the value_counts method. So let me show you here.

So apart from getting this nice visualization and recognizing where the outliers are, we can get also the ranges. So here if I write df_movie and now uh here minute and now I use the value_counts, we can see the values of the ranges. So we only need to add the bins. So here bins, and this one has to be exactly the same as the number of bins we added here in the histogram. So here in the histogram I added 10 bins. So here in the value_counts I also have to add 10 bins, and after this I run, and here you can see that we get the ranges here in the indexes, but it's not properly sorted. So here I'm going to use the sort_index method to sort the indexes. So here I run this, and now we sort this ascending.

So here we got the first range which represents this first bar that I'm pointing now, and here the last two ranges represent these tiny bars that you can barely see here. So we can say that in this data set outliers are movies with duration less than 31 minutes or with more than 249 minutes.

However, if you start analyzing this data set a bit more, you will notice that you can expand the range of the outliers. For example, here we can say that this range from 218 to 249 can be also considered outliers because here we only have six, and it's only six movies from thousands of movies in this data set. So, it's a tiny proportion. So again, this depends a lot on your criteria and also on the data set you're working with. Okay.

After you decide which ranges are considered outliers for you, you can filter them out. So we only have to use conditions as we learned in previous lessons. So here, let me show you. So let's say that the first range is an outlier and also the last three ranges. So to filter them out, we only have to write this df_netflix. Then we write here square brackets and now minute. So here I'm going to compare this uh column with a value 31. So I write 31 here. And now this is my first condition. So now my second condition is going to be this less than 218. So I write it here. And now I have my second condition. So here I add parenthesis to condition number one and also to condition number two. And now I add this and operator. And to filter these two conditions I add the name of the data frame and also square brackets.

So here if I run this, and here I got an error because I wrote DF_Netflix but here it should be df_ov. So my bad here. So I just replace df_netflix and write df_movie. And now I run this. And now everything is fine. So here we can see that this data frame only contains movies with duration greater than 31 minutes and less than 218 minutes. So now let's check the last column. And here we can see 90 minutes, 91, 125. And yeah, it's fine. So we filter the outliers out.

And now in case you want to see the outliers, I can use the not operator here with parenthesis. And now the not operator to print the data frame with the outlier. So now I run. And here we have this data frame with only outliers. So let's have a look at the last column. And here we have a movie with 23 minutes, 13 minutes, 229 minutes and so on. So these are the outliers and before we filter them out. Okay.

Now we're going to identify outliers with box plots. So another way to identify outliers within numeric data is using box plots. And a box plot let us see how the values in a data frame are spread out, and we can use it to recognize outliers. So now I'm going to show you how to do it. So first we have to write the name of the data frame and then we write the name of the column. So minute, then we have to write the plot method to make this box plot, and after this we write the kind parameter and write here equal to and here box, and this is how we make the box plot. So now I run this, and as you can see here we have here the box plot, and all those dots that you see here are the outliers, and all the values that is inside these whiskers that this box plot has is the range where the data will be usually located.

So here now I'm going to customize this box plot and I'm going to add some parameters. So here to see this box plot much better, I'm going to add the vert parameter and set this equal to false. So this one will allow me to put this box plot in vertical position. Now I'm going to add another parameter, in this case the color parameter. And I'm going to set this box plot to the blue color. And finally, I add the figsize parameter just to make this plot bigger. So I set this equal to 10, 5, and that's it. Now I run, and we can see this box plot much better. So here on the x-axis we have the minutes. So this is the duration of the movies, and here these values that are dots and that are outside the whiskers of the box plot are outliers. So this box plot is telling us which values are outliers.

And by looking at this box plot, we can see that the outliers are less than 48 approximately and also the outliers on the right are greater than 155. But that's only a guess because I don't know exactly what are those values. I just know that it's close to 50 on the left and on the right is close to 150. But to find these values, we can use some formulas that I have here.

So here we can use some formulas to calculate the range where the outliers are located. So here first we have the IQR, and this IQR is the Q3 minus the Q1. And as you might remember from previous lessons, this Q3 is the top of this box and the Q1 is the bottom of this box. So here we can use the describe method to get this Q3 and Q1 values. And with this we can calculate the IQR. And with the IQR we can use these two formulas I have here. And this will help me calculate the ranges where the outliers are. So the first formula represents the minimum value of the whiskers. So this one that is in blue. And the second formula represents the maximum value of the whiskers. So this one that is here. And both can help me find what are the ranges of the outliers.

So here I'm going to find the Q3 and Q1 of this box plot. And to do that I only have to use the describe method. So here I write the name of the data frame then the name of the column I want to analyze. So minute and now describe. We've seen this method before, and this method gives some descriptive statistics and we can see it here. So here the 25% represents the Q1 and the 75% represents the Q3. So here Q1 is 87 and Q3 is 114. So with these values now we can replace in this first formula to calculate the minimum value of the whiskers. So here instead of Q1 we can write 87 here and then minus 1.5 then uh multiply with this IQR and the IQR is Q3 minus Q1. So Q3 is 114 and Q1 here again 87. So here we have this value and I'm going to name it this uh min_box_plot. So this is the minimum value of the whiskers or the box plot. And now I'm going to copy this one and paste it right here. So now I'm going to replace this uh with the second formula. So Q3 is 114 and then plus 1.5 times and this 114 minus 87 which represents the IQR. So here we have the first and the second formula but here I have to change the name. So in this case max_box_plot.

All right, with this we calculate the min and the max values of this box plot. So now I'm going to show these values. So here I comment this out and now I print both values. So here the first one and now the second one. So max_box_plot. So if I run this we get the minimum and the maximum value of the whiskers. So here the first is 46.5 and this should be this minimum value of the whisker here and the maximum is 154. So this one now that we have these values we can say that in this data set the outliers are movies that have a duration of 46 minutes or less and movies that have a duration of 154 minutes or more. And such movies are represented by the dots that you can see here outside the whiskers of this box plot. And now we can filter out these outliers as we did previously with the histogram. So what we have to do is to copy this condition. And now just modify the values. So here I paste it here and now I write instead of 31 46 and here 154. So these are my new limits for the outliers. And as I told you before, this criteria to select outliers depends a lot on your data set and also on the project that you're working with. So it depends a lot on the goals of your project. So in this case, if I want to filter outliers using the data that I saw in the box plot, I will write these values 46 and 154. So now I run this and here we get the outliers because I'm using here the not operator and here we see 161 and 166. So these movies are outliers because these values are uncommon and that's how you use a box plot to identify outliers within numeric data. Okay.

Now we're going to see how to use bars to identify outliers within categorical data. So here when we have to find outliers within categorical data. So data that has many categories, we cannot use histograms or box plots because that's for numeric data. But we have to use bar plots. And here I'm going to show you how uh categorical data looks. And here I'm going to print the column uh rating. So here this rating column has uh categorical data here. And here we can see that we don't have numbers but we have categories. And you can see all the categories by using the value_counts method. So here I write value_counts parenthesis. And here we get all the categories. So for example, the TV-MA rating is the category that has the highest number of values. And here these three last categories 84, main, 66, and 74 are probably outliers because we only have one value in these categories. So we can say that these three last categories are uncommon and for that reason outliers. But now we can see this much better in a plot. So I'm going to use the plot method. So I write plot now parenthesis. So I add the kind parameter and here bar. Then I add another parameter to make the plot bigger. This case figsize. And I'm going to set this to 10, 5. Then I run this. And now it's evident that the last six categories in this bar plot are outliers because we can barely see the bars that these six categories have. And that indicates that those categories are uncommon. In contrast, this TV-MA category has the highest number of values. And we can see this because it has the tallest bar. And now that we recognize these outliers, we can filter them out using the conditions. So in this case, we could use a condition that says that ratings is different from a list and that list is made of these six categories. And by doing that, we filter them out of the df_movie data frame. In this case, I'm not going to do that because it's the same as we did in the histogram and in the box plot, but you can do it as an exercise. And that's it. In this video, we learn how to identify outliers using bar plots, box plots, and histograms. And also, we learn how to deal with these outliers in pandas. All right.

In this video, we're going to learn how to deal with inconsistent capitalization using the lower, upper, and title methods. This will help us standardize the letters that are in one column. So we can set all the letters to uppercase, to lowercase, or to title case. So let's try this with the title column, and I'm going to show you this one. So here I write df_movie then open square brackets and here title. Here we can see the titles of the Netflix movies. And these titles by default have this title capitalization. So here we can modify this capitalization using the methods. But first we have to get access to the str attribute. So here we have to write str. And now let's say that we want to make these letters into lower case. So we have to use the lower method. So we write lower, and with this we change the case of the text to lowercase. So I run this, and here we can see that now we have all the words in lower case. So this is great. But now let's say that we want that case of text in uppercase. So we have to write this again. So I paste this one. And now instead of lower we write upper. So with this we make these words inside the title column into uppercase. So I run this one now. And here we can see that all the words now of these movies are in uppercase. And finally the last case of text is the title case. So I'm going to show you here. So instead of upper we write title. And here we got the same title we got before in the original data frame. And here is the title case. So now let's say we want to update the data frame with text in uppercase. So here I uncomment this. And now I paste it here. And here I only override the values inside the title column. So here I write this. And now we run this one and we'll see the data frame here. I'm going to print it. So here df_movie. And now I run this. Here we get this warning. But it's fine. Nothing wrong happened. Just a warning. And here we see that the title column has words in uppercase. So all of them now are in uppercase. But this looks a bit weird because all the letters here in the other columns are in title case, but this is the only column in uppercase. So I'm going to change it now to title case, but I'm going to use a different method. And in this case I'm going to use the apply method. So here instead of getting access to the str attribute, I'm going to use the apply with the lambda function and then use the title method. So here I'm going to show you. So I write df_movie then the title column and now I write apply. So this is the apply method and we've seen this method a lot in this course. So now I write lambda. So I'm going to create a lambda function and here's the input uh random variable. So I name it x now colon and now the output. And now I want to convert this x variable into title case. So I write x.title. And as you might remember this x represents an element of this title column inside the df_movie. And that's why we use the title method without getting access to the str attribute. And before we run this code, I'm going to print this df_movie below this line of code. And now I comment this out. And here let's update the values inside this title column. So here I'm going to override the values. So I write this equal to the same column but now with apply method. So now I update and run this one. We get the warning message again, but everything is fine. And here in the data frame we have the words with title case. And this means that the first letter in every word is going to be in uppercase and the rest is going to be in lower case. And that's it. In this video, we learn how to deal with inconsistent capitalization using the lower, upper, and title methods. All right.

In this video, we're going to learn how to remove blank spaces using the strip, lstrip, and rstrip methods. First, I'm going to show you how the strip method works. So, here I'm going to create a variable named movie_title. And here I'm going to assign this to a value named Titanic. So here I write Titanic. And I'm going to add these blank spaces at the beginning and at the end of this word. And I'm going to do this on purpose for the sake of this video. Here we have these blank spaces. And here another blank space. So here I'm going to show you how the lstrip works. And here I write movie_title. And now I write that lstrip. And what this is going to do is it's going to remove the blank spaces at the beginning of this string. So now I print this. And here we can see that we don't have this blank space at the beginning of this text anymore. But we remove this blank space. And now let's try with the rstrip method. So what this is going to do is to remove the blank spaces at the end of this string. So here I print it, and now we can see that the last blank spaces are not anymore in this string. And now if we want to remove the leading and trailing spaces, we have to use the strip method. So instead of only removing the blank spaces at the beginning with the lstrip or the blank spaces at the end with the rstrip, now we're going to remove both with the strip method. So here I copy and now I paste this one. And here I delete the R. And now I'm going to apply the strip method. So here I run. And now we see that we only have the Titanic text and we don't have any blank space. And that's how you remove blank spaces with the strip method. And this is a really useful technique because sometimes when you have a data set, you don't know if that data set has some blank spaces in a column. And if you don't remove the blank spaces, you could get some unexpected behavior. And that's why it's always recommended to remove blank spaces from the text you're working with. And this is what we're going to do with the title column here in our data frame df_movie. Now I write the name of the column. So title now I print this, and in the result that we got it's not so obvious that we have blank spaces, but this column could have blank spaces. So we have to use the strip method in case this column has blank spaces. So here now I write str because we need to get access to the str attribute before we use the strip method and only now we can use the strip method. So we write strip and parenthesis. So with this we can remove leading and trailing spaces from these columns. So now I run this and here we got this column but now without blank spaces and here to update the changes we made we override the title column. So now I reassign it to this title column and now I'm going to show the data frame. So df_movie I get this warning message but it's fine. And here we see that the title column doesn't have any blank spaces. Great. Now I'm going to show you a different way to get rid of leading and trailing spaces. But now using the apply method. So here I'm going to write df_movie and select this title column. But now I'm going to use the apply method. So here that apply and here inside parenthesis I use the lambda function. So here lambda then the input then colon and then the method I want to apply. So I write x.strip with parenthesis and here this x variable is an element from this title column and that's why we can use the strip method without getting access to the str attribute. So now I comment this out and I run this and we

got the same result. And that's it.

In this video we learn how to remove blank spaces with the strip, lstrip, and rstrip methods. Okay.

In this video we're going to learn how to replace strings using the replace and sub methods. And as a first example, we're going to remove punctuation signs using the replace method and also regular expressions. But first, I'm going to show you the column we're going to work with, which is the title column. So here I write title. Now I run this, and here we have this data. So some of the movie titles that are listed here have some punctuations. For example, this second movie has the colon sign, and this is a punctuation sign. And sometimes when we work with text data, it's not a good idea to have numbers or some punctuation signs. And sometimes it's recommended to remove everything but letters. So that's what we're going to do. We're going to remove punctuation signs. And here I'm going to use a regular expression to do that. And I'm going to use it with the replace method. But first, I'm going to show you the regular expression we're going to use for this example. So, we have to create a regular expression that matches punctuation signs. So, here first I'm going to show you some popular symbols in regular expressions. And here I have two of the most popular symbols in regular expressions. So, the first one, \w, represents letters from the alphabet, so a to z, also digits from 0 to 9, and also this underscore symbol. The second symbol, \s, represents wide spaces. And with these two symbols, I'm going to create a regular expression that matches punctuation signs. So to do that, first I write the \w. So with this we get these elements, and now I write the \s. And with this we get white spaces and letters from a to z and digits from 0 to 9 and also the underscore. And something you need to know is that all the symbols that don't fall in these two categories are considered punctuation signs. So here we can use a not operator to get the opposite of these two symbols and as a result, get punctuation signs. And in regular expressions, the not operator is represented with the square brackets and the caret symbol in the beginning of this square brackets. So here I'm going to show you much better with a code cell. So I type Y, and here we have the code cell, and now you can see much better this symbol. So here in regular expressions, we get the not operator by writing this caret at the beginning of square brackets, and this means match anything that is not letters or digits or underscore and also not wide space. So the remaining is only punctuation signs. So this is the regular expression we're going to use here in the replace method.

So now that we have the regular expression, I'm going to use the replace method. So here I write that, and we need this str attribute to get access to the string that is inside this title column. And now I can write that replace, open parenthesis, and as the first argument, we have to write the text we want to replace. So in this case, we have to write this regular expression. So I write quotes and now this regular expression. So this means that we want to replace anything that matches with this regular expression. Then as a second argument, we have to write the text we want to insert instead of this first argument. So here in this case, we want to remove punctuation signs. So I only write quotes and nothing inside. And as a final argument, I have to write this regex parameter, and I'll set it equal to true. So this means that we are using regular expressions in our first argument. If we write regex equal to false here, the first parameter will be considered as a standard string and not a regular expression.

So now let's try this out. I'm going to run this code. But first, I want you to remember this second movie because this one has punctuation, which is this colon. So, after running this line of code, we shouldn't get this colon in this second title. So, let's try this out. I run this. And now we see the results. And here in this second movie, we don't see the colon sign anymore. So we successfully remove punctuation signs using this regular expression with the replace method. Okay.

Now I want to show you how to do this but only with the replace method. So this one looks similar to what we built right now, but now we don't need the str attribute. So I can delete it, and we can use only the replace method. And the replace method works in a different way than the str.replace, because this second replace method can replace not only strings but also other data types like integers. However, this str.replace method only can replace strings because this str attribute limits the capabilities of this replace method. So we only can replace strings with the str.replace method. And that's the big difference between these two methods.

So now to show you this much better, I'm going to use the title column again. And now I'm going to try to replace numbers with the two methods. So first with replace. So write replace. Now one, two. And here regex equal to false because this is not a regular expression. And now I do the same but with the str.replace. So here I write str.replace. So now let's see how it works with the first line of code. But first I comment this out. And now let's see how it works with the replace method. So I run. And as you can see here, it replaces all the one numbers with this two number. And now let's try with the second method. Let's see if it works with the str.replace. replace. So now I run, and as you can see here, I got an error, and here it says that it must be a string. So this str.replace only accepts strings in the first argument, and that's one of the advantages of the replace method over the str.replace. Okay.

Now I'm going to show you an alternative of the replace method. And in this case, we're going to use the sub function. And this sub function belongs to the re module. And this is the regular expression module. So first let's import it. So I write import re. After that we can use this function. But first we select the title column. So I copy and paste it here. And now we use the apply method to apply. Then parenthesis. Now the lambda function, the input x. And now the output here will be the re.sub. So this is the function, and now I open parenthesis, and the first argument is going to be the same regular expression here. So I copy and paste it. Now the second is the element we want to insert instead of this regular expression. And here I write this empty quotes to remove this punctuation. And as a third argument, I have to write the variable I want to evaluate. So in this case, this x that I had it as input. And now this is ready. So I'm going to comment this out. So I don't have this anymore. So I comment this out and also this too. Now I run this, and let's see the results. So here we see that this second movie doesn't have the colon sign. And with this we verified that we successfully remove all the punctuation signs in this column. And that's it.

In this video, we learn how to replace strings using the replace method and the sub function.

In this video, we're going to see the data set we're going to use in this section. So, in this section, we're going to use a data set that contains Boston house prices. And we're going to use this data set to predict the values of the houses in Boston. So let's start by importing pandas as pd, and then let's read the CSV file. So pd.read_csv, and then the name of the file is Boston house prices.csv. So now you run this, and here we got a data frame that contains three columns. The first column is the rooms column. This will contain the number of rooms that the house has. So usually if the house has more rooms, the price of the house will be more expensive. And if the house has less rooms, the price of the house will be less expensive. Then we have the distance column. This one contains the weighted distance to five Boston employment centers. And in this case, houses that are closer to employment centers will be more expensive. And finally, we have the value column. This one contains the median value of the owner-occupied homes in thousands of dollars. So what we're going to do later is to predict the price of the houses in Boston. So we're going to predict the value column. And to predict this, we're going to use two columns as inputs. So we're going to use the rooms column and also the distance column. Okay.

Now let's name this data frame. So now I write equal to and now df_Boston. So this is the name of the data frame. And now I run this, and this is my new data frame. So now I show it again. And now let's see some statistical values of this data frame. So now I'm going to use the describe method. So I write describe with parenthesis. And now we can see the number of rows this data frame has, also the mean and the minimum and maximum value. So for example, in the rooms, we can see that the mean number of rooms in these houses is six. Also the minimum number of rooms is three, and the maximum is eight. So now is your time to familiarize with this data frame so you can understand much better the following lessons.

In this video, we'll see one of the most basic machine learning algorithms. This is the linear regression. The linear regression is an approach for modeling the relationship between two or more variables. When we have only two variables, we're dealing with a simple linear regression. But when we have more variables, we're dealing with a multiple linear regression. In a simple linear regression, one variable is considered the predictor, also known as the independent variable, while the other variable is known as the outcome, also known as the dependent variable. This is the linear regression equation. Here y is the dependent variable, also known as the target value, and x1, x2 until xn is the independent variable. Here also b0 is the intercept, while b1, b2 until bn are the coefficients, and n is the number of observations. This equation is represented by this graph. So in this graph we can see a linear relationship. So if one independent variable increases or decreases, that dependent variable will also increase or decrease. Linear regression can be used to make simple predictions such as predicting exam scores based on the number of hours studied. Also the salary of an employee based on years of experience, and so on. Okay.

As I mentioned before, the equation behind this graph is this one that now you see. So the first one is the general equation or the complete equation, and the second one is the simplified equation. This one represents a simple linear regression because we only have one dependent variable and one independent variable. Some people use this notation, but some others use y = a * x + b. So these are two popular notations, and you have to keep in mind that people use both. Okay.

Now let's see what are the dependent and independent variables of our data set that we've seen in the previous video. So in the previous video, we read a Boston house prices CSV file, and this CSV file had three columns. This data set has information about the number of rooms, also the weighted distance to five Boston employment centers, and also the value of a house. So here first we're going to see that simple linear regression. And in a simple linear regression, we have one predictor and one target. And in this case, our predictor is going to be the number of rooms. And our target value or target variable is going to be the value of a house. So we're going to predict the value of a house based on the number of rooms. So we can say that the number of rooms is the independent variable, and the value of a house is the dependent variable. So this is for our simple linear regression. But now for the multiple linear regression, we're going to have two independent variables. So the number of rooms and also the distance, and the target variable is going to be the same. So the value of a house. So for this multiple linear regression, we have two predictors and one target. All right, that's it for this video, and in the following videos, we'll implement all of these using Python.

Okay.

In this video we're going to implement linear regression in Python. And there are two ways to implement linear regression in Python. And two simple ways to implement linear regression in Python is using the statsmodels and the scikit-learn libraries. Both are popular libraries used in machine learning. And we're going to learn how to do it with both. And in this video, we're going to use statsmodels. So to use statsmodels, we have to import this by writing the following. So we write import statsmodels.api as, and here we write sm. So this sm represents this statsmodels API. So now I press Ctrl+Enter to import this statsmodels library. And now we imported this, and statsmodels is a module that helps us conduct a statistical test and estimate models. And some of the benefits of statsmodels over scikit-learn is that it provides an extensive list of results for each estimator. And now before we create this linear regression, we have to read the data set we've seen before. So this Boston house prices. So now I run these two cells, and here we have the data set. So now let's create this linear regression. So we'll start with a simple linear regression. And the first thing we have to do is to define a dependent and independent variable. This is the first step you have to do always when you want to create a simple linear regression. In the previous videos, we discussed what are the dependent and independent variables. So the dependent variable is the value we want to predict, and it's also known as the target value. On the other hand, the independent variable is the predictor in our data set. So this df_Boston data frame, we have two predictors. So we can use any of them in our simple linear regression. In this case, I'll choose the rooms column as the independent variable. And the dependent variable is going to be the value column. So now let's define the dependent and independent variable, but now in Python. So here I'm going to write y = df_boston, and then I'm going to select this column, the value column. So this is my dependent variable, and now I'm selecting this column. So now I set this equal to y, and y is my dependent variable. So now I'm going to do the same but now with the independent variable. So now my independent variable is going to be named x, and I'm going to select the rooms column. So with this we have the dependent and independent variables. Okay.

And before we create our simple linear regression, let's explore our data set. A common way to explore a data set is using scatter plots. So a scatter plot allows us to see how the data is distributed. So to make this a scatter plot, I'm going to use the plot method that comes with pandas. So I'm going to write df_boston.plot. And now I write the kind parameter equal to scatter. Then I add the x and also the y parameter. So now x and y. And now I'm going to set the x equal to these rooms and the y equal to value. So with this we can plot this data frame, and now I press Ctrl+Enter, and here we have our scatter plot. So in the x-axis we have the rooms, and in the y-axis we have the value, and we see that most of the data is located here in the middle. So what we're going to do in the next step is to create a simple linear regression that fits best these data points. So the line that fits best these data points is the one we're going to choose. So let's create this line and let's write the code here. And here to make a regression, we have to add a constant and fit the model. First we need to add a constant because a statsmodels by default doesn't add the constant that we have in the equation of the simple linear regression. So if you remember, there was a b0 that represented the constant, and by default statsmodels doesn't add this constant, but we can add it manually. So here I'm going to write the code that helps me add this constant. So I write sm.add_constant. So with this we can add a constant. So now I have to write the independent variable, in this case the number of rooms, which is x, and with this we add a constant. So now I'm going to write x = this, and this x represents the independent variable but now with a constant. So now that we added the constant, we can fit the model. So to fit the model, we only have to use the OLS method. So I write sm.OLS and then parenthesis. And here first we have to write the dependent variable, which is y, and then the independent variable, which is x. Then to fit this model, we use the fit method. So I write fit then parenthesis, and with this we fit our model. And by the way, fitting the model means finding the optimal values in the equation of the linear regression. So we obtain a line that best fits the data points. A model that is well fitted produces more accurate outcomes. So only after fitting the model we can predict the target values using the predictors. Okay.

Before I predict the values, I'm going to name this. So I'm going to name it lm, which stands for linear model. So I run this. And here I got an error because I didn't run the previous cell. So this one in which we define the dependent and independent variable. I didn't run this cell. So now I press Ctrl+Enter. And with this I define the dependent and independent variables. So now I can run this cell too. And now we added the constant and fitted the model. So now we can predict the values of these houses. So I'm going to use the lm, which is the linear model we created. And now I use the predict method. So I write predict and now I introduce the independent variable, which is x. So this x represents the number of rooms. So with this x we're going to predict the values. And here we got the values of the houses. So we predicted these values based on the number of rooms. Okay.

Now let's see how this linear model performs. And to do that, we're going to use the regression table. This table provides an extensive list of results that reveal how good or bad is our model. So to get this table, we only have to use the summary method. So we write lm.summary and then parenthesis. With this we get this table, and this table has two or three parts. So the first one is the OLS regression results, and OLS it stands for ordinary least squares, and this is the most common method to estimate linear regression. Then in the second and third tables, we'll see more values. So now let's see which are the most important values in these tables. So let's start with this dep. variable, which means dependent variable. So we can see that our dependent variable is the value. So this is the value of the houses. Then we have the R-squared, and this R-squared takes values from 0 to 1. The R-squared values close to zero correspond to a regression that explains none of the variability of the data, while values close to one correspond to a regression that explains the entire variability of the data. So the R-squared we got here is telling us that the number of rooms explains 48.4% of the variability in house values. Then we have the coef column, and here we can see the coefficient of the linear regression equation. So the first one is the constant, and the second one is the coefficient that corresponds to the rooms variable. Then we have the std err column, and this one represents the accuracy of the prediction. The lower the standard error, the better prediction. Finally, these t and P>|t| values are used for hypothesis tests. Okay.

Now let's use these coefficients to create the line that best fits the data points that we've seen before in the scatter plot. I already have the coefficient. So 9 for the rooms variable and -34 for the constant. So here the linear equation is a * x + b. And we're going to replace the values of the coefficients a and b in this equation. So now I create a y_pred, and this is equal to 9.1021 multiplied by the x variable, which is the number of rooms. So this one is x, and here rooms. So I'm selecting the rooms column.

And now, plus b, and in this case, b is minus 34. So I only write this minus 34. And with this, we have this equation that corresponds to the line that best fits the data points we've seen before.

And now let's plot this line. And to plot this line, I'm going to use advanced data visualization libraries such as seaborn and matplotlib. So here I have the code to plot this line with the scatter plot. And I'm using, in this case, Seaborn and Matplotlib because these two libraries allow me to put the scatter plot and the line plot in one graph. So you don't have to memorize this code. The most important part is this line. And here we indicate that we want to make the line plot. And as a y parameter, we write the y_predict. And here this is the equation we created before here. And also in the X parameter, we set it this equal to the rooms column in the data frame. And that's everything you need to know. So now I run this, and we're going to see the scatter plot. But now we also see the line plot, which is in red. So we can say that this red line is the line that best fits the blue data points. By the way, you don't need to create this plot. I just made it to show you how this regression line looks.

And that's it for this video. In the following video, we'll see how to make a multiple linear regression with Python.

All right. In this video, we're going to see how to create a multiple linear regression with statsmodels. So, first we have to read the same data frame df_boston. And now we have to import statsmodels. So we write import statsmodels.api as sm. So with this, we can start this, and here I just need to add the s, and with this I imported the statsmodels.

And now let's define the dependent and independent variable. So in this case, it's going to be independent variables because we're going to have more than one variable. So I'm going to show you here, and uh, the df_boston is this one, and here we have the dependent variable, which is the value column, and the independent variable is going to be, in this case, the rooms and the distance columns. So now let's define this using pandas. So the dependent variable is df_boston[‘value’]. So I'm going to set this equal to y, and with this we have the dependent variable. So now I can copy this one, and here I forgot to write _boston. Okay. So now I copy this one, and with this we can create also the independent variable. So I write here x, and here instead of value, I'm going to write rooms and distance, but here we have to write double square brackets because we want to select two columns. So here I write rooms and distance. So I copy and now I paste it right here. So now I write comma and add the quotes. So with this, we have the dependent and independent variables. So I run this, and now it's time to make the regression. So let's add a constant and fit the model. So first to add the constant, we have to follow the same steps we did in the simple linear regression. So we write sm.add_constant, and then inside parenthesis, we have to write that independent variable. So in this case, is x, and now we set x equal to this. So with this, we're going to get the independent variables, but now with a constant. So here I'm going to fit the model, and to fit the model, we used sm.OLS. So I write sm.OLS, and here first we have to write the dependent variable, which is y, and then the independent variable, which is x. Then we fit the model using the fit method. And I'm going to set this equal to lm, which stands for linear model. So with this we run, and we created our multiple linear regression. So now we can see how this model performs by using the summary method. So we only have to write lm.summary(), then parenthesis, and we get the same table we got in the simple linear regression. So the table is similar, but in this second table, you're going to see that there is an extra row, and this extra row corresponds to the independent variable that we added. So before we only had the rooms variable, but now we have also the distance variable. Here also we can see that the R-squared increased a little bit, and overall, the analysis of this table that we got after creating the multiple linear regression is similar to the one we obtained after creating the simple linear regression.

And that's it for this video. In this video, we learn how to implement the multiple linear regression in Python.

All right. In this video, we're going to see how to create a linear regression using sklearn. Sklearn, also known as scikit-learn, is the standard machine learning library in Python. And it can also help us make either a simple linear regression or a multiple linear regression. In this case, we're going to see how to make a multiple linear regression with sklearn, but the steps to make a simple linear regression are going to be the same. So now let's start by importing linear_model from sklearn. So we write from sklearn.linear_model import LinearRegression, and this linear_model has everything we need to create our linear regression. So now I run this, and after we imported this, we have to define the dependent and independent variables. So now this is going to be exactly the same as we did previously with statsmodels. So I'm going to write it again. So first, the dependent variable is df_boston[‘value’]. And here I'm going to select the value, which is the target. And now I copy and paste it. So I create the independent variables. And here I'm going to write x =, and here I'm going to select two columns. So I write double square brackets to select two or more columns. And now I copy and paste this rooms and distance. So with this, we create that independent variable. So after we do this, we're going to fit the model. So now to fit the model, we only have to use the fit method. And by the way, here with scikit-learn, we don't need to add a constant as we did with the statsmodels library because scikit-learn adds this constant by default. So we only have to fit the model. So now I write lm = LinearRegression().fit(x, y). And now I use the linear regression method. So I write LinearRegression(), now parenthesis. And now I set it equal to lm, which is stand for linear model. So now to fit the model, I use the fit method. And here I write x, y. So keep in mind that here the order is different from statsmodels because in statsmodels, first you write y and then you write x. But in scikit-learn, first we have to write the independent variable and then the dependent variable. So keep that in mind. Okay. Now let's run this cell, and with this we fit the model. So now let's predict the values. So we write again lm and we use the predict method. So predict, and inside parenthesis, I write the independent variable, which is x. So now I run, and with this we predict the values of the houses using the independent variables. So here we got an array, and this represents all the predicted values. And now if you want to select only the first five predicted values, you only have to write square brackets and then write colon 5. And with this, we select only the first five values in this array. Then if we want to get the values in the summary table that we got using statsmodels, we have to use individual methods because with scikit-learn, there is not a single method that can help us get the same summary table that we obtain with the statsmodels library. So in this case, for example, if we want to get the R-squared score, we have to use the score method. So I only have to write lm.score(x, y). So the independent and then the dependent variable. So now I run this, and with this we got the R-squared, which is the same value to the one we got with statsmodels. Then we can also get the coefficients. We only have to write lm.coef_ and run this. And we got the same coefficients we got with statsmodels. And we can also get the intercept by using the intercept_ attribute. So we only write lm.intercept_ and _. So now I run, and we can see now that the intercept is minus 34, which is the same value we got in the summary table using statsmodels. In this case, I'm not going to explain the meaning of these values because they are the same values we got before. So the analysis is the same. And if you want, you can compare the values to verify that we got the same result. In the following videos, we're going to use the scikit-learn library because it's the standard machine learning library used in Python and is more powerful than statsmodels.

All right, in this video, we're going to see the data set we're going to work with in this section. So first let's import pandas and then let's read this data set. So import pandas as pd, and then we use the read_csv method to read this CSV file. So read_csv, then the name is imdb_data_set. So here data_set.csv. So this is the name of our data frame. Now we, uh, read okay, the data set that we have now is an IMDb data set that contains 50,000 movie reviews, and this data set contains two columns. The first column is the review column, and in this, uh, review column, you will find what the users, uh, think about the movie. So that's what they think if that movie was good for them or terrible. So based on that review, we will get a positive or negative sentiment, and you will find that data in the sentiment column. So the sentiment column contains the sentiment of the review. So if it's a good review, you will have a positive sentiment. So if the review has words like great, awesome, good movie, the sentiment will be positive. But if the review has words like bad, terrible, the sentiment will be negative. So our sentiment column only has positive and negative values. I chose this data set to create a basic machine learning model. So the model that we're going to create will predict whether a review is positive or negative, and we're going to use this data set as input to fit our model. But first, we need to transform this data frame in order to build the model. So we're going to try different machine learning models in this section. And our final goal is to find which machine learning model is best suited to predict sentiment given a movie review. So the sentiment is going to be the output of our model, and the movie review is going to be the input of our model. Remember that because that's the foundation of this project. So the sentiment is going to be the output, and the movie review is going to be the input.

All right. As you can see, this data frame has 50,000 rows. However, to train our model faster, we're going to take a smaller sample of 10,000 rows. This small sample will contain 9,000 positive and 1,000 negative reviews to make the data set balanced. So, I can teach you under-sampling and over-sampling techniques in the following videos.

All right, we're going to create this small sample with this code here. So here below, we're going to write the code that would allow me to take a sample of 10,000 rows to make processing faster and get balanced data.

All right, before we create this small sample, let's give a name to this data frame. So now I'm going to set this data frame equal to df_reviews. So I write df_reviews, and then I run this code. So here I run, and now we have our data frame. So now let's take a sample of 10,000 rows to make processing faster and also get balanced data. So now I write df_reviews, and then I, I'm going to, uh, make a condition. So I select only the data I want. So here I write, uh, sentiment, because that's the column we want. So I'm going to show you here again the data frame. I'm going to select the column sentiment, which is this one. And now we write double equal sign to select only the positive, uh, sentiment. So positive, and now I'm going to select this by writing the name of the data frame and then putting this inside square brackets. So this is the selection we make, and we learned this in the previous videos how to filter a data set. And now I'm going to run this, and let's see the output. So here we have the data frame, and we see that all the data inside the sentiment column is positive. So we only have positive sentiment. So now we have here 25,000 rows. But as I mentioned before, we only want 9,000 rows that correspond to the positive sentiment. So here I'm going to select only 9,000. So I write colon and then 9000. And with this, we're going to get only the first 9,000 positive sentiment.

All right. Now let's do the same, but now with the negative sentiment. So now I write negative. And now I select only the first 1,000 rows.

All right, with this, we have 9,000 positive reviews and 1,000 negative reviews. So now let's give a name to these two data frames. So here I'm going to name it df_positive. And the other is going to be, uh, df_negative. So now df_negative. And here we have our two data frames. So here, uh, we have 9,000 df_positive and 1,000, uh, negative. So now I write it here. And now let's run this code. So to create our two data frames. So now we have our two data frames. Now let's concatenate these two data frames. So we get only a single data frame as we had before here in this df_reviews. So here let's concatenate vertically. So I write pd.concat(), then parenthesis. Now I open these square brackets and now I write the name of the two data frames we created. So first df_positive and then df_negative. So these are my two data frames. And now let's run this. And here we concatenated these two data frames vertically. So now we have, uh, this data frame, and this data frame contains positive and negative reviews and 9,000 positive and 1,000 negative reviews. So now let's give a name to this data frame. I'm going to name it df_reviews_imb. This IMB stands for imbalanced, and we can say that we have imbalanced data when we have a large amount of data for one class and much fewer observations for other class. So this is known as imbalanced data because the number of observations per class is not equally distributed. So here we have 9,000 positive reviews, and we have 1,000 negative reviews. So 9,000 is way bigger than 1,000. So we can say that we have imbalanced data because the sentiment of the review is not equally distributed.

Okay. Now this df_reviews_imb represents our imbalanced data. So now I run this, and now let's see how this, uh, data frame looks. So now I paste this, and this is the data frame we got after concatenating the positive and negative reviews. Now let's see how is distributed. So now I use the value_counts method. So I write value_counts, and now we write the name of the sentiment column. So here I open quotes and write sentiment. So now I run this, and here we have that there are 9,000 positive and 1,000 negative, uh, reviews. So we verified that this data is indeed imbalanced. And in the following videos, we're going to see how to deal with imbalanced data.

And that's it for this video. In this video, we had a look at the IMDb data set, and also we, uh, selected only a few positive and few negative reviews. So we made that data imbalanced, and we created this df_reviews_imb data frame. In the next video, we're going to see how to deal with imbalanced data.

In this video, we're going to see the under-sampling and over-sampling techniques. Oversampling and under-sampling in data analysis are techniques used to adjust the class distribution of a data set. Both techniques help us deal with imbalanced data. In some cases, we should avoid imbalanced classification because it can influence the performance on our machine learning algorithms. That said, both oversampling and under-sampling involve introducing a bias to select more samples from one class than from another to compensate for an imbalance that is either already present in the data or likely to develop if a purely random sample were taken.

Okay, let's see how both under-sampling and over-sampling work. Okay, first let's see under-sampling. And under-sampling is just deleting samples from the majority class. So here we have two classes. The blue class and the yellow class. So the blue class is the minority class, and the yellow class is the majority class. Okay. Now let's imagine that we have movie reviews, and the minority is represented by the negative reviews and the majority is represented by the positive reviews. So if we want to balance the data set, we have to under-sample the majority class, that is deleting samples from the majority class. So in this case, we will delete samples from the positive reviews. So from the majority class, we do this because we want the same number of observations in both categories. So we want the same number of positive and negative reviews. So we reduce the number of positive reviews. So we get the same number of reviews.

Okay. Now let's see the oversampling. And oversampling is just duplicating samples from the minority class. So here again, the majority class is the positive reviews, and the minority class is the negative reviews. So now if we want to balance this data by oversampling, we need to increase the number of observations of the minority class. So we have to duplicate some samples in this minority class. And we do this, and in the end, we'll have the same number of observations between the minority class and the majority class. So this means that we will have the same number of positive and negative reviews.

So now you might be wondering which one is better, oversampling or under-sampling. So the answer depends a lot on the data that you're working with and also depends a lot on your project's goal. So sometimes it'll be better to oversample your data, but in some other cases, it'll be much better to under-sample your data.

All right. Now let's implement the oversampling and under-sampling techniques to our project. So let's go to Jupyter Notebook, and here we're going to start by making a bar plot to show how the data is distributed. So now I'm going to, uh, write the name of the data frame that we created before, which was, uh, df_reviews_imb. So this imb stands for imbalanced, and this is our imbalanced data. And we're going to use the plot method to plot this, uh, data frame. And now I'm going to use the kind parameter and set it equal to bar. Then I'm going to use the value_counts method to only get the number of elements the sentiment column has. So here I copy sentiment. And now I write value_counts(). And now I write sentiment. So I'm going to plot this output. And now this is ready. So I press Ctrl+Enter. And now we have this bar plot. And this bar plot is useful to show how the data is distributed. For this particular example, it might look a bit unnecessary, but when you have many categories, this becomes really useful.

Okay. In this plot, we can see that the majority class is the positive reviews because we have 9,000 reviews, and the minority class is the negative reviews because we only have 1,000. So here we have two options. We can under-sample the positive reviews, or we can over-sample the negative reviews. For this example, I'm going to under-sample the positive reviews. And now I'm going to show you two different ways to under-sample, uh, class. So now let's scroll down, and here let's sample the positive reviews with the sample method. And to do this, first we have to calculate the length of the negative reviews. And to do that, first we write df_reviews_imb. Then we select only the negative reviews. So here I write sentiment == ‘negative’. And now we filter this with these square brackets. So now we only have negative reviews. And now to calculate the length of these negative reviews, we use the len function. So now I'm going to set this equal to length_negative. And with this, we can see the length of this, uh, negative reviews. So here I'm going to print it; n is 1000. Okay. Now we're going to extract a sample from this positive reviews, and this sample is going to be equal to the length_negative. So we get in the end the same number of observations. So I write first df_reviews_imb. And now I'm going to select only positive reviews. So now I

Write sentiment again, then equal to, and here positive. Now I filter this with square brackets, and here I'm going to use the sample method. So I write that sample and inside parenthesis I write the end parameter and set it equal to length negative. So, with this we're going to extract only a sample of 1,000 from this positive reviews.

So now let's try this out. And here I have the output. And we see that in the sentiment column there are only positive reviews. And here we see that we have only 1,000 rows. So we got 1,000 positive reviews. So this is the output we wanted. So here I'm going to set this equal to DF_reviews_positive. So this is my new data frame.

Okay. Now, let's extract only the negative reviews. And then we're going to concatenate the positive and negative reviews. So, I'm going to use this code because this code already gets the negative reviews. So, is this one I run and as you can see, we have in the sentiment column only negative reviews. And we have 1,000 rows. So I'm going to set this equal to DF_reviews_negative. So now I run and I created this data frame.

Now let's concatenate these two data frames, the positive and the negative. Okay. Now I write PD.concat, then open parenthesis and then open square brackets. Then I write the name of this positive data frame and then comma and then I add the second data frame. So the negative reviews data frame. So now we concatenate these two data frames and let's run. So here we have a data frame with positive and negative reviews. And we see that now we have 2,000 rows. And if we check the distribution of this uh sentiment column, we will see that now the data is equally distributed. So now we have 1,000 negative reviews and 1,000 positive reviews. So this is what we wanted. Now the data is balanced.

So now I'm going to set a name to this data frame, and the name is going to be DF_reviews_bal. And this bal stands for balance. So now I run this and we created this data frame. So now let's check again because I think the indexes are not properly assigned. So here first I'm going to delete this value_counts. So we get the concatenation. So now I run again. And now I run this DF_reviews_bal. And now uh here we see that the indexes are not sorted ascending. So let's reset the index. So it starts with zero and then one, two, three and so on. So now I write reset_index and here in the parenthesis I write drop equal to true. And then I add the in_place parameter and set it equal to true to update this data frame. So now I run this and here we have the result. So now I'm going to print this data frame and see the index. And now we see that the index starts now with zero. And as you might remember, if we use the value_counts we can verify again that the sentiment is equally distributed. So now we have 1,000 positive and 1,000 negatives. So this is the first way to balance data using the sample method.

And now I'm going to show you an easier way to balance data. And the second way to balance data is using the RandomUnderSampler. And to use this RandomUnderSampler, first we need to install the imblearn library. And to install this library, we have to write uh exclamation mark and then pip install imbalanced-learn. So now we run this and wait a couple of seconds until the installation is done. So now the installation is finished and now we can import the library. So I'm going to delete this. And now to import the RandomUnderSampler we have to write the following: from imblearn.under_sampling import RandomUnderSampler. So here I'm going to just copy and paste this to write it faster. So here is import RandomUnderSampler. So now I run this and now we successfully imported this RandomUnderSampler.

So now to sample the positive reviews, first we have to create an instance of this RandomUnderSampler class. So here I write rus equal to RandomUnderSampler and here in parenthesis I write the random_state. This is just a way to control the randomness of the RandomUnderSampler. But it's not necessary. But here I set it equal to zero. So I write equal to zero. And now we create this rs that stands for RandomUnderSampler.

Okay. Now to resample the imbalanced data DF_review_imb, we have to fit the rus variable that we created here and we have to use the fit_resample method. So here the x represents the data which have to be sampled and the y corresponds the labels for each sample in x. In this case, X is the review column and Y is the sentiment column. So now let's resample the imbalanced data. So here instead of X, I'm going to write DF_review_imb. This is the data set that is imbalanced. And now I'm going to select only the review column. So now I add double square brackets. And now here I select the review column. And here in the Y argument I write the sentiment column. So here I copy the name of the data frame. And now I select only the sentiment column. So sentiment. And now keep in mind that the first argument should be a data frame while the second argument should be a series. So this is a 2D and this is a 1D. This is how the fit_resample works. And you can also see that I used here double square brackets for the data frame because if I use double square brackets, I get a data frame as output. But if I use only a single pair of square brackets, I get a series as output. So keep that in mind.

So now I'm going to run this and let's see the result. And here I got an error because I didn't write the name of the data frame correctly. So I add the S. And now here the S2 and now I run. And this is the output of the last line of code that we wrote. So this is the review column and this is the negative column. So now let's assign this to a new data frame and this data frame is going to be the balance data frame. So here I write DF_review_bal and something I almost forget to tell you is that this output that we got after using the fit_resample method has two elements. So the output has two elements and the first one is this data frame. So this is a data frame and this data frame has only one column and the second output is a series. So this one is a series. So here apparently is only one element but actually we have here two elements. So this is the first one and this is the second one. And the first one is a data frame because here we introduce a data frame in our first argument. So makes sense that we input a data frame and we output a data frame in the first element. And here in the second element we insert a series. So again we input a series and we output a series too. So this second element is a series.

So now let's um continue here. So first this is my data frame and the name of the data frame is DF_review_bal and this represents the first element. So the data frame so this one and now I'm going to write the series that corresponds to this second element. So the sentiment column. So now here I write uh DF_review_bal but now I select only a column and it's going to be sentiment. Since this DF_review_bal data frame is new, what we're doing here is to create a new column for this data frame. So this column is going to be sentiment and actually we're adding this column to the DF_review_bal that I'm creating right here. So now let's set all of this equal to the expression that we wrote here. And anyway you can see here that we resample the reviews and also the sentiment. So now let's run this and see the result. So now I run this code and now let's see the data frame we just created. So here we have the DF_review_bal and now we have this data frame. So here we have the review and the sentiment column and we see that now we have only 2,000 rows and apparently we undersampled the data. But now let's verify if that's true. So let's use the value_counts method to count the elements inside the sentiment column. So now I copy the name of the column. I open parenthesis and now I run this code. So now let's verify. And here we have 1,000 negative reviews and also 1,000 positive reviews. So we successfully undersampled the data using the RandomUnderSampler. By the way, in this example, I use the RandomUnderSampler because I chose to under sample the positive reviews. But in case you chose to over sample the negative reviews, you only have to use the RandomOverSampler. And you can do this by importing the RandomOverSampler from the imbalanced-learn. And that's it. In this video, we learned the oversampling and under sampling techniques.

Okay, in this video we're going to see how to split data into train and test. Before we build our model, we have to split the data into a train and test set. The train data set will be used to fit the model while the test data set will be used to provide an unbiased evaluation of a final model fit on the training data set. Remember that we always have to split our data into train and test set before building our model.

All right. Now to split the data into train and test set, we're going to use the sklearn library. And this library comes with Anaconda. So we already have this library installed. So you don't have to install it. So let's use the sklearn library. So I write from sklearn.model_selection import train_test_split. So this train_test_split will help us split data into train and test. So now the only thing we have to do is to use this train_test_split and now open parenthesis. So after we do this we have to introduce the name of our data set in this case is DF_reviews_bal. So this is the balance data set that we created before. And now we have to split the data and we have to set the size of the split. So here I write the test_size parameter and set it equal to 0.33. So this 0.33 indicates that we want to set the 30% of the data frame DF_review_bal to the test data and the remaining 67% will be assigned to the train data. So now I'm going to assign this to train, test. So this train_test_split will return two values. And these two values are the train and the test. So the 67% and the 33%. And before running this code, I'm going to add an extra parameter that you don't need to add, but I'm just going to do it to control the random state. So I add random_state and this set it equal to 42. This helps me set the same random values no matter how many times we run this code. So with this I control the randomness. So here now I can run this. So now I press Ctrl enter and here I got an error because I didn't write correctly the name of this model_select actually is model_selection. So I write selection and now I run. And now we successfully created this train and test data frames. So now I'm going to show you the trained and the test data frame. So this is the train data frame. And here we have uh approximately 67% of the data and in the test data frame we have 33% of the data.

And another good practice that we have to follow before building the model is to create an X and Y variables. So the X represent the independent variables and the Y represent the dependent variables. So for example, we should split that train into train_x and train_y. So in the train_x, we should have only the predictors also known as independent variables and in the train_y we should have only the dependent variables also known as the target label. In case you don't remember the definitions of independent and dependent variables, I'm going to make a recap. So the dependent variable is the value we want to predict and is also known as the target value and the independent variable is the predictor. In our example, the dependent variable is the sentiment because this is the value we want to predict and our independent variable is the review because this is our predictor. So with the review we predict the sentiment.

Okay. Okay, now let's set the independent and dependent variables within our train and test set. So here I'm going to write train and then I'm going to select only the review column and then I copy this. I write comma and I write train and here I'm going to select only the sentiment column. So the review is my predictor. So this is my independent variable. So here I set this equal to train_x. So x is the independent variable also known as the predictor. And here the sentiment is the dependent variable. So I'm going to set this equal to train_y. So the y is the dependent variable. So x is independent and y is the dependent variable. And I'm going to do the same for the test data set. So instead of writing train, I'm going to write test. But here I'm going to show you a shortcut. So here if I press the key F, we're going to see this find and replace. And here I can write the word train. So it matches four words that have the word train. And now I can replace it for test. So if I write test here, we can replace the word train with the word test. So now I click in replace all and now voila, we have test instead of train. So now I'm going to copy and paste this one here again. And now I'm going to run this code and we'll see the result.

Okay. And as I mentioned before the train data set. So these two will be used to fit the model. So now let's see how these two data sets look. So first that train_x so this one contains only the review column and now the train_y should contain the sentiment column. So yeah we have this and now let's count the number of positive and negative reviews. So I use value_counts now parenthesis and we got 675 positive reviews and 665 negative reviews. And that's it. In the following videos, we'll use this train_x and train_y we just created.

In this video, we're going to learn what is bag of words. The bag of words model is a representation that turns text into fixed-length vectors. This helps us represent text into numbers. So, we can use it for machine learning models. The model doesn't care about the word order, but it's only concerned with the frequency of words in the text. The typical bag of words workflow involves cleaning raw text, tokenization, building a vocabulary, and generating vectors. We can do all of this in Python using a tool called CountVectorizer that takes care of most of the bag of words workflow. And to understand how the CountVectorizer works, let's see an example. And in this example, I have two reviews that show the opinion that people have on Python and on Java.

So to implement the bag of words through the CountVectorizer, we only have to count the number of times each word appears in both sentences. So we can build a table and this table could show the number of times each word appears in the sentences. So let's have a look. This is a table I built and as you can see here I have the sentence, the two sentences or two reviews and also in the table I have how many times each word appeared in each sentence or each review. So for example the word code appeared two times in the first review. So in the first review here we can see that the word code appears after the word writing and also after the word Python. So yeah it appears twice in this first review. So this is how you build this table and this table represents a document-term matrix that we're going to see later. And we'll tame this document-term matrix by using the CountVectorizer in Python. By the way, keep in mind that words with two letters or fewer are not taken into account by the CountVectorizer.

All right. Now let's see how to implement the CountVectorizer in Python. So now we go to Jupyter notebook and here let's import the CountVectorizer. So we write from sklearn.feature_extraction.text import CountVectorizer. So I write count and then vectorizer. So now I'm gonna paste the text that we had before. Okay, I just pasted that text we had before. And now I'm going to create a data frame. So I use the pd.DataFrame. And now I open parenthesis. And now I create this data frame with a dictionary. So I open curly braces. Now I create the first column which is going to be review. So the first column is review and then I create the elements. So the first element is going to be review one. And the second element is going to be review two. And now I create a second column and it's going to be named text. So this is the name of my second column and the value that is going to have is the text uh list that I have here. So this is my list and now I just paste the variable. So now I'm going to create an instance of the CountVectorizer. So here I copy and paste CountVectorizer then open parenthesis. And that's how I create an instance of this CountVectorizer. So I'm going to name it this cv. So cv that stands for CountVectorizer. Now we can add a parameter here. So I want to add the stop_words parameter. And this one will help me filter out words. After this I'm going to use this cv I created. And now I'm going to feed this CountVectorizer. So I use the fit_transform and inside parenthesis I have to write the name of the data that I want to fit to this CountVectorizer. So in this case I want to fit this text list. So I'm going to put the name of the text column. So I write df then open square brackets and then text. So this uh column has the data that I want to fit to this cv that is the CountVectorizer. And by the way the fit_transform method works in this way. First the fit will find the best parameters that fit to this data and then the transform will transform our data using these optimal parameters.

Okay. So we can run this to see the results we have so far. So now I run and as you can see the result after feeding and transforming this data is only this sparse matrix that we cannot actually see. But now if I create another data frame we can see this sparse matrix into this data frame. So I'm going to do this now and I'm going to create this data frame and I'm going to name it DF_DTM and this DTM stands for document-term matrix. This is what we're going to build. So now I'm going to use here that pd.DataFrame again. So I just copy and paste this one because we're going to build a data frame. And now I open parenthesis. And here the first argument we're going to introduce is the data. So the data that we want to show in this data frame is this sparse matrix. So I'm going to name this sparse matrix and the name is going to be cv_matrix. So this is the name. Now I run. And now we add this sparse matrix to the first argument. So we show this sparse matrix in the data frame. Now the second argument is going to be the index and this index is equal to df['review'].values. So here we're going to get the review one and review two in the indexes. By the way here I almost forget to add that toarray because here now this is a sparse matrix but I want to get an array. So I use that toarray. So I only get the the values inside. And also here we want the values. Right now we have the series. So it's the review series. But I only want the values. So I use the values attribute. And with this we get only the values. Next I'm going to add the columns parameter and set it equal to cv.get_feature_names_out(). So this one. Now I open parenthesis and with this we're setting each word that appears in these two sentences in the column of this DF_DTM data frame. So with this our data frame is ready. So now I run this and now I show you the result. So here this is the result. So this is the same uh table we had in the slides and we build it with Python. So in this case it might seem unnecessary to make this little table. But now

Imagine that you have many, many sentences, or you have a movie script with a lot of text. In that case, you will need to use a tool like the CountVectorizer to show the frequency of the thousands of words that that movie has. And that's it for the CountVectorizer.

Now let's see another way to implement the bag of words in Python. So now we're going to see the TF-IDF. TF-IDF stands for term frequency inverse document frequency. So the first one, the term frequency, is the TF, and the last one is the IDF. Unlike the CountVectorizer, the TF computes weights that represent how relevant a word is to a document in a collection of documents, also known as a corpus. The TF value increases proportionally to the number of times a word appears in the document and is offset by the number of documents in the corpus that contain the word.

The TF-IDF has applications in information retrieval, like search engines that aim to deliver results that are most relevant to what you're searching for. So, for example, Google, when you search a keyword, Google returns the most relevant information to you. So that's some kind of basic TF-IDF application there.

Okay. Before we see the Python implementation of this TF-IDF, let's see an example, so you have a good idea of how the TF and IDF are calculated. The TF-IDF is calculated by multiplying the TF and the IDF.

And now let's see how to calculate the TF. So the TF is that term frequency. And this one looks like the CountVectorizer because here we only count the number of times a word appears in a text. So, as an example, let's take the same two sentences that we used in the CountVectorizer. So here we have similar output, but in this case, the two columns are TF1 and TF2. So the term frequency one and term frequency 2. And in the table, we see that, for example, the word "loaf" appeared two times in the TF1, but it didn't appear in the TF2. So TF1 represents the first sentence, and TF2 represents the second sentence. So with this, we calculated the TF. This is exactly the same as the CountVectorizer. But keep in mind that some people have a different definition of the term frequency. But in most cases, the term frequency is only considered as the number of times a word appears in a text. So exactly what the CountVectorizer does.

Okay. Now that we have the term frequency, we're going to see how to find the IDF. So the inverse document frequency. So the IDF is calculated with a formula you see on screen. This may look intimidating, but when we implement this with Python, we don't have to make any calculation but only use the sklearn library, and with that, we can easily calculate all of this. By the way, sklearn assumes natural logarithm instead of the log that you see in this formula. So now let's calculate the IDF values for each word as sklearn will do it. So using the natural logarithm.

Okay. In this table, I replaced the values of the formula using the data that I had in the two sentences that I showed you before. And as you can see, the numerator in all the calculations is three. So you see three in all of them. And this happens because here the numerator is the total number of sentences plus one. And the total number of sentences remains the same because we have the same number of sentences. So we have two sentences, and 2 + 1 is three. And that's why we have three in all the numerators. However, the values in the denominator change. So we have here two, and then three, and then two, and so on. And this happens because here the number of sentences containing a specific word in this list will change. So, for example, we'll see that the word "writing" will appear more times than the word "love," and that will change depending on each word and also on the sentence.

Okay, now that we have all the values of the IDF. So here it's IDF1. I didn't write it correctly, but this is IDF1, and here is IDF2. So now that we have IDF1 and IDF2, we can calculate the TF-IDF1 and the TF-IDF2. So we only multiply the TF1 with the IDF1 and the TF2 with the IDF2. And here we have the values. And in this table, the higher the TF-IDF score, the more unique the term. So, for example, here we have the highest score for "love" in TF1 and also "Python." So these two words are more unique to this first sentence. While in the second sentence, the words "hate" and the words "Java" are more unique to this second sentence because they have the highest score. So we can say that if the TF-IDF score is high, that term will be more unique or more valuable. In contrast, if the score is low, that means that that word is not unique. So probably that word belongs to more sentences or more documents. For example, here the words "writing" and "code" appear in both sentences, and this is why we got a low TF score.

Okay, now let's see how to implement this TF-IDF in Python. So let's go back to Jupyter Notebook. And now I'm going to copy this code that helped me create the CountVectorizer. So this text variable. And now here I paste this code. And now I'm going to import a library, which is the TF-IDF Vectorizer. So I write from sklearn.feature_extraction.text import, and then TF-IDF Vectorizer. So this one. So now I'm going to create an instance of this TF. So I write TF-IDF =, and now I copy and paste this TF-IDF Vectorizer. Then I open parenthesis, and here I add this stop_words parameter. So I get rid of stop words in English. So now it's here. And now I'm going to create a data frame so I can show you this table with these values that we calculated manually before, but now we're going to do it with sklearn. So here I write df, and I'm going to create the same data frame. So here I just copy this because it's going to be the same. So now I paste it here. Now let's fit and transform this TF. So I write tf.fit_transform. So this one, and now I open parenthesis. And now I introduce that text data that we want. So df[text]. So now I'm going to run this to see the result. So now I run, and again we got this sparse matrix, and now I'm going to name this result. So I'm going to name it TF_matrix. So here _matrix, and now we run this, and we created this new variable. So with this, I'm going to create a new data frame so I can show the TF score in this new data frame. By the way, here the fit_transform works the same way that in this CountVectorizer. So the fit finds the best parameters, and then the transform transforms the data that we have using these optimal parameters. So now let's create the data frame. So now I write pd.DataFrame. Then I open parenthesis and write the name of this TF_matrix. Then we transform this to an array with parenthesis. Now the index, going to be, actually it's going to be the same here. So I'm just going to copy so I save some time. So I copy this and now I paste it right here. So this is the same index and the same columns because we're using the same text variable here. It's the same. We're just changing the approach. In this case, we're using the TF-IDF Vectorizer and not the CountVectorizer. But here we have to make a slightly change. So instead of cv here, we write tf. And now this is ready. So now I can show you this one. And now we run this. And by the way, the TF-IDF Vectorizer. So this one uses L2 normalization. So this is why we got these numbers. And to get the same result I got here in these slides using the formulas I showed you before, we have to add a parameter to this TF Vectorizer. So this parameter is norm, and we have to set it equal to None. So by default we use the L2 normalization, but with the norm = None, we'll be using the same formulas I showed you before, and we're going to get the same result we got in the slides. So let's try this out first. I run this, and now I run this one, and as you can see, we have now the same numbers that we got in the slides. For example, review one love is 2.88, and here review one TF-IDF1 love is 2.88. So it's the same result. So with this, we successfully built our TF-IDF Vectorizer, and in the next video, we're going to see how to implement this in our project.

All right. Now it's time to put into practice the concepts we learned in the previous videos to turn data into numerical vectors. So in the previous videos, we balanced our data. So we have imbalanced data, and then we use the RandomUnderSampler to balance this data. Actually, we saw two ways to balance the data. So first we use the sample method, and then we use the RandomUnderSampler. But for the analysis that we're going to make in this and in the following videos, we're using the data we got with the RandomUnderSampler. So in case you only used the first method, the sample method, there is nothing wrong; the result is going to be almost the same, but I'm going to choose this one, and the output will slightly change, but it won't affect so much the result. So in case you want to have the same output that in this video, you should use the RandomUnderSampler too. So I'm going to run this, and after this, we split the data into train and test. So here I run this again, and now we're going to use this train_x, and we're going to turn text data into numerical vectors using that TF-IDF that we learned in the previous video. So I'm choosing the TF-IDF because I want to identify unique and representative words for positive and negative reviews. And the TF-IDF gives a higher score to unique words within sentences.

Okay. Now to do this, let's first import TF Vectorizer as we did in the previous video. So we write from, then sklearn, then feature_extraction.text import, but before I have to add here the text. So now I can write TF-IDF Vectorizer. So TF-IDF Vectorizer. So now it's ready. So we press Ctrl+Enter, and we imported this TF-IDF Vectorizer. Now let's initiate an instance of this TF-IDF Vectorizer. So I write this, then parenthesis, I add the stop_words parameter. So stop_words = English. Then I'm going to set this equal to TF-IDF. And now let's feed this TF-IDF with the data that is in the train_x. So we created before this train_x, and this variable has the reviews that we're going to fit into that tf. So now we write tf, and now I write the fit_transform method. So fit_transform, and then I copy and paste that train_x. So with this, we're going to fit and transform this train_x. Now I'm going to assign a name to this, and I'm going to name it train_x_vector. Now I run this, and with this, we have this train_x_vector. So now I'm going to print it here, and this is a sparse matrix, and a sparse matrix is a type of matrix in which most of the elements are zero. So if I use the toarray method, we can transform this to an array. So here I run, and as you can see, most of the values are zero. So if you see here in the output that we got here, you see that this is a matrix of 1340 rows and 20625 columns. If you multiply these values, you're going to get a lot of elements in this matrix. However, only 118,000 are nonzero values. So the percentage of nonzero values is little. And we can see that this sparse matrix has a lot of elements that are equal to zero. And here we can also create a data frame from this train_x_vector, but it's not necessary, but it can help you visualize how this matrix or this vector looks. I'm going to create a data frame as I did in this example. But in this case, we're going to have a lot of columns and also a lot of rows. And here I pasted a code that helps me create this data frame. And it's similar to the code we wrote before in this example. But in this case, I'm not transforming this matrix to array, but I'm using the sparse attribute that this data frame has. And also the from_sparse method. And with this, I don't need to transform the train_x_vector into an array as I did in the previous example. So I run this. And now you can see the data frame. And as you can see, it has a lot of zeros. And in the columns, we have the vocabulary used in the reviews. So all the words that were mentioned in the reviews in this train_x that I'm going to show you right now. So in case you forgot about it here, all the vocabulary used in these reviews are here in the columns. So every word that has more than two letters are taken into account in the columns, and these indexes represent the number of the review.

Okay. Now before we finish the video, we're going to also fit and transform the test_x because the test_x will help us evaluate the model that we're going to build. So now I'm going to add here a cell. And now I'm going to copy this one and paste it here. So now instead of using the fit_transform on the train_x, I'm going to use it on the test_x. And actually, we don't need to use the fit_transform method because we already got the best parameters when we fit and transform the train_x. And those parameters are here in the TF-IDF. So here I can only use the transform method. This one is also a method. Actually, there is a fit method and there is a transform method, and also a fit_transform that does both. But in this case, we only need the transform method because the TF already knows what are the optimal parameters. So with this, we transform the test_x, and we name this test_x_vector. So here test_x, and now I run this, and with this, we're going to get a sparse matrix again. So let me show you, and this is the result.

All right. In the following videos, we're going to use the train_x_vector to build the model, and we're going to use the test_x_vector to evaluate the model. In this video, we're going to see the machine learning algorithms that we can use in our project. But first, let's see the difference between supervised and unsupervised learning. Machine learning algorithms are divided between supervised and unsupervised learning. In supervised learning, models are trained using labeled data, while in unsupervised learning, patterns are inferred from the unlabeled input data. In our project, the input and the output are clearly identified. So the input is the movies reviews, and the output is the sentiment. So either positive or negative. So we can say that we have labeled input and output data. Therefore, we're dealing with supervised learning. Two common types of supervised learning algorithms are regression and classification. Regression is used to predict continuous values such as the price, the salary, and the age. Then classification is used to predict categories such as male, female, spam, not a spam, or positive and negative. So in this case, it can be more than two categories. So it can be two positive, positive, negative, and two negative. So it doesn't need to be only two categories.

So now let's make a recap. And now let's see the types of algorithms we should use in our project. So first, we're dealing with supervised learning because we have labeled input and output data. Then we have to use classification because we want to predict the category of our review. So we want to know if the review is either positive or negative. So this is why we choose classification. And some of the popular algorithms used in classification are SVM, decision tree, naive Bayes, and logistic regression. And we're going to see the concept behind each of those algorithms and also how to implement them in Python. And we're going to see that in the next video.

In this video, we're going to see the support vector machine algorithm. The support vector machine algorithm, also known as SVM, is a supervised learning algorithm that is mostly used in classification problems. We only need to feed the SVM model with labeled training data in order to categorize new text. This algorithm is a good option for text classification problems because it has a high speed and good performance with a limited number of samples. And we usually work with a data set that has a few thousand of tagged samples in text classification. To understand much better how SVM works, let's see an example. So in this graph, we have two tags, green and yellow. Also, we have two features, X and Y. In a data set, a feature is simply a column that represents a measurable piece of data that can be used for analysis. And in this example, we want to build a classifier that finds whether our text data is either green or yellow. So we plot each observation, also known as data point, in an n-dimensional space, where n is the number of features used. In our example, we only have two features, x and y. So the observations are plotted in a two-dimensional space. The SVM takes the data points and makes a hyperplane that best separates the classes. Since the observations are plotted in two-dimensional space, the hyperplane is simply a line. So you can see this line now, and this red line represents the hyperplane that is plotted in a two-dimensional space. This red line is also known as the decision boundary. The decision boundary determines whether a data point belongs to one class or to another class. In our example, if the data points fall on the left side, it will be classified as green. And if it falls on the right side, it will be classified as yellow. And now you might be wondering what's the best hyperplane? Well, it's the one that maximizes the margins from both classes. So you can draw many hyperplanes, but the one that is the best is the one that maximizes the margin from both classes. So in this case, the red line is the one that maximizes the margins from both classes.

So now let's see how to implement the SVM algorithm in Python. And in Python, we can use the sklearn library to import the SVM algorithm. So we only have to write from sklearn.svm import SVC. So this SVC belongs to the SVM algorithm. So now let's create an instance of the SVC, that I should have written in uppercase. So now it's in uppercase. And now I initiate an instance by writing SVC( ). And now we add the kernel parameter and set it equal to linear. So now we set this equal to svc, and then we fit this model. So we write svc.fit(, and inside parenthesis we have to write the input and the output. So in this case, train_x_vector, and the output is train_y. So this train_x_vector is the vector we got here when we turned our text data into numerical vectors using the tf vectorizer. Here we got the train_x_vector that we're using right now. And later we're going to use that test_x_vector to evaluate the models we're going to build. So now let's go back, and here I'm going to fit this model by running the cell. So I run, and now I wait, and now it's done. So here I use a linear kernel, but you can change that type of kernel if you want. And now we can use this svc to predict sentiment. So we only write svc.predict(, and inside we use that tf we created before. So this is going to transform the text that I'm about to write. So let's write a text that says "a good movie." So this is my movie review, and I write "a good movie." So with the TF-IDF that I initiated before here in the previous video, with this TF-IDF, I'm transforming this text into vectors, and then with the SVC, I'm predicting the sentiment based on these vectors. So now I run this, and now let's see the result. But first, I'm going to write this square brackets. So this is a list, and now I run, and we got that the sentiment is positive. So probably the SVC detected this "good" as a positive adjective, and this is why it output this positive sentiment. So now let's try with two more sentences.

Okay, I just came up with two more movie reviews, and this one says "an excellent movie," and the last one is "I didn't like that movie at all. I gave this movie away." So let's see what the SVC will predict. So now let's run this. And we got that the first is positive. So

a good movie. That's a positive review. So it was correct. The second uh is positive too. So an excellent movie. So it's a positive review. So it's correct. And the last one is I didn't like the movie and more text. And in this one it predicted that is negative. So here it successfully predicted these three reviews, but with more complex sentences maybe it can predict wrong. So we have to build the other algorithms and compare with each other which is the best. And that's it for this video.

In this video we learn the concepts behind the SVM algorithm and how to implement it in Python. All right.

In this video we're going to see how the decision tree algorithm works. The decision tree algorithm is a supervised learning algorithm that can be used for solving both regression and classification problems. We use a decision tree to build a model that can predict the class or value of the target variable by learning decision tree rules inferred from the training data. To predict a class label for a record, we start from the root of the tree. So the root of the tree is this first node that you see here. Each node in the decision tree will evaluate the record following a specific rule. Then we follow the branch that corresponds to the result of the comparison, and then we jump to the next node.

Let's understand much better how the decision tree works with an example. In this example, we decide whether a customer will turn or not based on some rules. Turning is defined as customers who leave a company. And in this case, we're going to see whether a customer turns or not based on the monthly charges. So here in my root, I have monthly charges less than 40. And this is kind of a question because if the answer is yes, then we'll follow this branch on the right. And if the answer is no, we'll follow the branch on the left. So let's say the monthly charges is 30, and now we evaluate this node. So let's imagine this is a telco company. If a customer has been more than 3 years with the company, we say that the tenure is more than 3 years. So if that's the case, if the customer has been like four years with the company, then we follow the branch on the right. So we set churn equal to no. So if this is an old customer, probably this customer won't leave the company. But if the tenure is less than 3 years, so the customer has been with the company only two years or one year or only a few months, this customer probably will leave the company. So if the answer of this question is no, then we follow this branch and then churn is equal to yes, because if the customer has a few months or a few years with the company, most likely this customer will leave the company if the monthly charges has been less than $40. So with this we finish the analysis on the right branch, and now let's say that the monthly charges are greater than 40. So let's say that this customer charges $100 per month. So we'll follow the branch on the left, and then we'll ask a new question, and this question is: customer has more than one product? And if the answer is yes, then churn is equal to no. So if the customer has many dependencies, most likely this customer won't churn because usually companies give promotions and some benefits to customers who have more than one product. However, if the answer of this question is no, then we'll start asking more and more questions. And this is basically how a decision tree works.

So now let's see how to implement the decision tree in Python. So let's go back to Jupyter notebook and we're going to use here again the sklearn library. So first we write from sklearn.tree import DecisionTreeClassifier. So this one, then we create an instance. So we initiate an instance from this DecisionTreeClassifier, open parenthesis. And I'm not going to add any parameter. I'm going to leave it with that default values, and I'm going to assign a name to this decision tree classifier and it's going to be dc_3. So now I copy and paste, and now I fit this model. So I write fit, then parenthesis and use again the x and y variable. So now train_x vector and then train_y. So now I press Ctrl enter to feed this model, and we successfully build this model. And that's it for this video.

In this video we learn the core concept behind the decision tree algorithm and also how to implement it in Python. All right.

In this video we're going to see how the naive Bayes algorithm works. Naive Bayes is a supervised learning algorithm that uses conditional probability to predict a class. It assumes that every feature is independent of each other, which is not always the case. So we should always analyze our data before choosing the algorithm. The naive Bayes algorithm is based on the Bayes theorem. And the Bayes theorem is the one you see now on screen. So the first element, this one, represents the probability of event A given event B has already occurred. Then this one represents the probability of event A. This one represents the probability of event B given event A has already occurred. And finally, this one in the denominator represents the probability of event B. The assumption that features are independent of each other makes this algorithm fast compared to more complex algorithms. However, this assumption also makes the naive Bayes algorithm less accurate.

So now let's see an example of this naive Bayes algorithm. In this example, we have this data that shows how many times players played a match under certain weather conditions. So we have to find whether players will play if weather is sunny. So this is our task. And here we have the formula. We replace the formula, but now with our task. So we have to find whether players will play if weather is sunny, and this is represented here. So we want to know the probability of event yes given event sunny has already occurred, and this is the formula. So now let's calculate each element to replace it and finally calculate this probability that we want. So now let's calculate the probability in each row and column. But first we have to calculate the total sum in each row and column. So here we have the total sum of each row and column and also the sum of the total values. So this sum of total values is nine. And based on this data we can calculate the probability of each row and column. So here we have the probability of each row and column. And as you can see the denominator is always nine because that's the total number of values. So now let's see the probability of yes and the probability of sunny given yes and the probability of sunny. So the first one, the probability of yes is this first column. So the probability of yes is four divided by nine. So four represents the number of times is yes and nine the total values. So we got 0.44 for this first probability. And now let's calculate the second one. In the second one we have to calculate the probability of sunny given yes has already occurred. And in this case we have to divide three, which is the number of times players played in a sunny day. And we have to divide this by four because this is the number of times that players played in general. So three divided by four is 0.75. And finally we have to calculate the probability of sunny, and this is this row. So sunny, five divided by the total values. So five divided by nine and in this case is 0.55. So now we replace these values in this formula and then we calculate the final probability which is 0.6.

So now let's see how to implement this in Python. So we're going to use sklearn again and we write from sklearn.naive_bayes import GaussianNB. So this one. So now we initiate an instance of this GaussianNB. I open parenthesis and I'm going to name this one gnb that stands for GaussianNB. So now I'm going to fit this model. So I write fit, parenthesis and inside train_x vector and also the train_y, and here there is a little detail you have to know: unlike the previous algorithms we've seen here we have to include that train_x vector as an array. So we have to use that toarray method. So with this we transform this train_x vector to an array, and if you don't do that it's going to throw you an error. So now let's run this and with this we fit this model and that's it.

In this video we learn the concepts behind the naive Bayes algorithm and also how to implement it in Python.

Okay, in this video we're going to see how the logistic regression algorithm works. Logistic regression is a supervised learning algorithm that is commonly used for binary classification problems. We can use logistic regression to predict whether a customer will churn or not, a mail is a spam or not, a sentiment is positive or negative and more. The logistic regression is based on the logistic function, also known as the sigmoid function, which takes in a value and assigns a probability between zero and one. Then we get this S-shaped graph of logistic regression that you see on the screen. To understand much better how logistic regression works, consider a scenario where we need to classify whether an email is a spam or not, in the graph. If this Z goes to infinity, our target value, so Y will become one. So the email is spam. And if Z goes to negative infinity, Y will become zero. Which means that the email is not spam. As I mentioned before, the output value is a probability. So if we obtain a value of 0.7, this means that there is a 70% chance that an email will be spam.

Okay. Now let's see how to implement this algorithm in Python. So we go back to Jupyter notebook and here I'm going to use sklearn one more time and I write from sklearn.linear_model import LogisticRegression. So I write LogisticRegression and now I initiate an instance of this LogisticRegression. I open parenthesis and now I assign this to log_reg. So this is the variable I'm creating. And now I'm going to fit this with the train_x vector and with the train_y. So I write fit, then train_x vector and then train_y. So now with this I'm going to feed this model. And now I just fitted. And that's it for this video.

In the next video we're going to compare the four models we built in this project. All right.

In this and in the following videos, we'll see traditional metrics that are used to evaluate models. And the first metric we're going to see is the confusion matrix. A confusion matrix is a table that allows us to visualize the performance of an algorithm. This table typically has two rows and two columns that report the number of false positive, false negatives, true positive, and true negative.

Okay, to understand what these four values mean, let's see the following example. Suppose that we have a computer program that recognizes dogs in photographs, and we give it a task to recognize the five photos you see now on the screen. So here we have three photos of dogs and two photos of cats. So our computer program should recognize which photos belong to dogs. So this computer program recognizes that these three photos are photos of dogs. However, we see that two photos are photos of dogs, but one of them is not a dog, it's a cat. So the two photos of dogs are the true positive because we wanted to obtain dogs and we obtain dogs. But the photo of the cat that was incorrectly recognized by this computer program is called a false positive because according to this program, the photo is a photo of a dog. But we know that this is the photo of a cat. So this is a false positive because it was selected by the program, but we know it is false because it's not a dog but a cat. Then the values that were not selected by our program are either true negative or false negative. In this case the cat that was not selected is a true negative because we asked to recognize the photos of dogs. And this is actually not a photo of a dog, but is the photo of a cat. So this is a true negative. In contrast, the photo on the right is the photo of a dog, but our program didn't select this photo for some reason. And this photo should have been selected, and this is known as a false negative.

Okay, now that we know the concept of the values in this confusion matrix, let's see the convention for axes. So the confusion matrix that I showed you before and that is not on the screen is the traditional confusion matrix that you will see in books. So usually confusion matrices follow this convention for axes. So the true positive is in position number one, false positive is in position number two, false negative is in position number three, and true negative in position number four. However, sklearn doesn't follow this same convention. So sklearn has a different convention for axes, and we should keep that in mind. So now let's see how the confusion matrix looks by default in sklearn. So by default here we have the true negative in the first position, then false positive here, and false negative here, and true positive here. So this is the default order that sklearn follows, and as you can see it's a bit different from the traditional confusion matrix. However, we can customize the position of this confusion matrix a little bit by using the labels parameter. So now we're seeing the sklearn, but with the labels parameter that I'm going to show you later when we go to Jupyter notebook. But if we set the labels parameter to the values positive, comma, negative, or also to the values 1, comma, 0, we're going to get this order. And this one is slightly similar to the traditional confusion matrix. And we're going to use this confusion matrix in this project.

Okay. Now let's see how to implement the confusion matrix in Python. So we go back to Jupyter notebook and now we're going to calculate the confusion matrix of only the SVC. So this one that we created before. So as you might remember the SVC was the first model we built, and this one came from the support vector machines, which is this one. So we're going to create the confusion matrix of this SVC, and then you can create the confusion matrix of either the decision tree, the naive Bayes or the logistic regression. So it's going to be the same. So now to do this we're going to import confusion_matrix and we write from sklearn.metrics import confusion_matrix. So now we use this confusion_matrix and we open parenthesis and the first argument we're going to introduce is the test_y. So we write test_y and this is also known as the true labels. So now the second argument is going to be the prediction. So we have to predict using this svc model. So we write svc.predict, predict and then we introduce the test_x vector. So I write test_x vector and now with this we predict the values that correspond to this test_x vector, and here we're going to get the prediction of our svc model. So we have the true labels and these are the predicted labels. So remember that order when you build your confusion matrix. So now let's add that third parameter which is the labels parameter and with this we're going to customize the axes of the confusion matrix. So now I'm going to set it equal to positive, comma, negative. So you can also write 1, comma, 0, and with this we're going to get the confusion matrix in this order that we've seen before. So now I'm going to run this and here we got an array. So we got four numbers and this one is the true positive. This one is the true negative. This one is the false positive and then false negative. So 60 false positive and 45 false negative. And that's it.

In the next video, we're going to use these four values to evaluate the models we built.

In this video, we're going to see the first metric to evaluate our models. And this one is the mean accuracy. The model accuracy is defined as the number of classifications a model correctly predicts divided by the total number of predictions made. That is the true positive plus the true negatives divided by the total number of predictions made. So in the previous video we calculated the four values using the confusion matrix. And now let's see this confusion matrix. So these are the four values and we calculated it using sklearn. And by the way this is the confusion matrix of only the SVC model. So there are confusion matrices of other models too. But here just to simplify things I'm just calculating the confusion matrix of the SVC model. So now let's see the accuracy of the SVC model. So now we replace the values in the formula and with this we can calculate the accuracy of the SVC model. So this one is 0.84. And now let's see how to implement this accuracy in Python. So in Python we have to use the score method. So we only have to write the name of the model, in this case the SVC, then use the score method and then inside parenthesis we write the input and the output. So now let's go to Jupyter notebook to see how to do it.

So here in Jupyter notebook I'm going to start by calculating the accuracy of the SVC. So first I write svc, then score, then open parenthesis, and here we use the test_x vector and also the test_y. So in this case we use the test and not the train because the test data set provides an unbiased evaluation of a final model fit on the training data set. So we cannot use the training data set again, but we should use the test data set now to evaluate the model. So here I'm going to copy this test_x vector and test_y. And now I paste it here. And actually I can calculate the accuracy of all the models. I only have to write the names. So here I can write dtree which was the name of the decision tree model. And now the gnb which stands for GaussianNB and then the log_reg that stands for logistic regression. So now to show all the values, I'm going to print the four of them and then we'll compare which one has a better accuracy. So now I print this and here I got an error because I forgot to transform this test_x vector to an array. So here we have to use the toarray method and this is only required for the GaussianNB as we've seen before. So now we run and we got the accuracy of the four models and we see that the model with highest accuracy is the SVC model with 0.84 and this is the same value we obtain by calculating this manually. That said, this is not the only way to assess the performance of a model. So in the next video, we're going to see a different metric to evaluate the performance of a model.

Okay. In this video, we're going to see another metric that will help us evaluate our models. And in this case, we're going to see the F1 score. So the F1 score is the weighted average of precision and recall. Accuracy is usually used when the true positives and the true negatives are more important while the F1 score is usually used when the false negatives and false positives are crucial. Also, F1 takes into account how the data is distributed. So it's useful when you have data with imbalanced classes. The F1 score is calculated with this formula. So we have to multiply precision and recall and then sum precision and recall. Then we divide these two values and multiply it by two. So now you might be wondering what's precision and what's recall. So to explain you the meaning of these two values, let's see this graph. And in this graph we see the elements of the confusion matrix. So true positive, false positive, true negatives and false negatives. So let's say again that we have a computer program and let's say that our computer program selected this circle area with green and pink sides. So this circle is the selected elements and this left rectangle is the relevant elements. So these relevant elements represented by this left rectangle cover all the elements that we're interested in. This is why the selected elements that are relevant are called true positives and the relevant elements that were not selected are called false negatives.

Okay, with that in mind, to get the precision we should ask ourselves the following question: How many selected items are relevant? So we see that the selected elements are in this circle and those that are relevant are in this green area. So the result of the precision will be the true positive divided by the true positive plus the false positive, that is the green area divided by the whole circle. So if we replace the values we got in the confusion matrix we'll get 290 divided by 350. So this will be 0.828, and this is the precision of the SVC.

model. Now, to calculate the recall, we have to ask ourselves another question. In this case, the question is how many relevant items are selected. So, the items that were selected and are relevant are inside this green area that you see now, and all the relevant elements are inside this left rectangle. So, this recall will be the true positive divided by the true positive plus the false negatives. And if we use the values we calculated for the confusion matrix of the SVC model, we'll get 290 divided by 335, which is 0.865.

So now, if we replace the precision and recall in this formula, we're going to get the value of 0.845. And that's how you calculate the F1 score.

So now, let's implement this F1 score in Python. So we don't have to make all these operations manually, but now with sklearn. So we go back to Jupyter notebook, and here I'm going to import F1 score, and I write from sklearn, and then we write import f1_score. So now we use this f1_score; we open parenthesis and again we use the test data set. So we write test_Y, then the prediction, which is SVC because we're going to get the F1 score of the SV model. So SVC.predict, then parenthesis, and we use the test_x vector. So here I write this, and here we have the true labels and the predicted labels. So now we add the labels. So labels equal to, and then positive and comma negative. So with this, we can calculate this F1 score. But here I'm going to add another parameter, which is the average, and I'm going to set it equal to none. Okay. Now I press Ctrl enter to calculate this F1 score. And we see that this first value is similar to the one we obtain manually here. And that's it. In this video, we'll learn the concepts behind the F1 score and how to implement it in Python.

Okay. Now, let's see how to create a classification report. A classification report shows the main classification metrics we've seen so far. For example, the precision, the recall, and the F1 score. And now let's see how to create a classification report in Python. So we go back to Jupyter notebook, and here we create a classification report. So we use sklearn and we write from sklearn.metrics import, and here uh we import classification_report. So this one, and now we use this classification_report, and now we open parenthesis, write the test_y, then we predict the values using the svc model, so we write svc.predict, then we predict based on the test_x vectors, so I write this one, and now I add the labels, so the labels are going to be uh positive and negative. So I write positive and then negative. So, as we can see, the arguments are in the same order that we seen in the matrix before. So now I press control enter to run this. And now we get this classification report. But it's not properly printed. So I'm going to use the print function. So I write print. Now I open parenthesis. And now I run this. And we get this classification report properly printed. And here we got the metrics we calculated so far. So the precision, recall, and the F1 score. If you want, you can compare these values with those we calculated before, and they should be the same. And that's it. In this video, we learn how to create the classification report in Python.

Okay, now it's time to see how to maximize our model's performance. Based on the metrics we've seen so far, we got that the SVC model performs better than the other models, but we can still maximize our model's performance. And we can use the grid search CV to make an exhaustive search on a specified parameters in order to obtain the optimum values of hyperparameters. And to do this, we have to import grid_search_CV from sklearn. And after this, we have to set some parameters to find which is the optimum parameter. So here I write parameters equal to, and here I open a dictionary, and the key is going to be C, and here I write a list of values. So first one, then four, then eight, then 16, then 32, and then I write the kernel key. So I write kernel, and then I open a list and write two types. So linear and also the RBF. Then we initiate an instance of the SVC algorithm. So we write SVC with parenthesis, and then here equal to SVC. Then we have to use the grid_search_CV. So I write it here. And in the first argument we write SVC. Then a second argument we write the parameters. And now I'm going to add another parameter, which is CV. And I'm going to set it equal to five. Now I'm going to name this SVC_grid because this one comes from the grid search CV. And now I'm going to use this variable, and I'm going to fit this with a train_x vector and also with the train_y. So now I write that fit, and here I write the train_x and then the train_y. So, as you can see, this code is similar to the code we wrote to fit the SV model before, but now we specify some parameters to obtain the optimum model. So now let's fit the model. I press Ctrl enter. And now this might take some time. So now I'm going to cut the video and come back when it finishes. So now the execution is finished. And before I show you how to get the best parameters, here you can add more uh parameters to this grid search CV by pressing shift tab. So here there is a list of more advanced parameters. So, for example, you can set the number of jobs and customize this even more with this parameters. So now let's see how to get the optimal parameters of this SVC model. So here I copy svc_grid, and then we have to write the following attribute. So best_params_, and we run this, and here we got the optimal parameters of this svc model. We can also get the best estimators of this model by using the best_estimator_ attribute. So now I copy and paste this, and instead of params I'm going to write estimator. So with this we get the best estimator of the SVC model. So now I run, and here we see the best estimator. And that's it. In this video, we learn how to find the optimal parameters of a model using the grid search CV.