Transcription
Hello!
Today, I will talk about the library for working with tabular data, Pandas. It is one of the most popular libraries for data analysis in Python. If you plan to engage in machine learning or work as a data analyst, you need to know this library.
In today's video, based on a real dataset, we will explore how to read data and write it in Pandas, the two main data structures in this library, how to filter data by rows, how to modify data, how to join multiple tables, how to calculate various analytical functions like mean, median, or count of objects, and how to visualize data using Pandas, meaning how to create different graphs.
Before we start coding, I want to remind you that if you want to learn not just Pandas but a comprehensive set of skills for Data Science, I create an individual training program tailored to your goals with mentorship.
Now, let's move on to coding. By the way, if you want to run today's tutorial yourself, I will upload it to GitHub and leave a link in the description of this video.
As I mentioned, Pandas is a library for working with tabular data, and we will use one of the most well-known datasets for this purpose: the Titanic dataset. Let's download it and read the data from it. The dataset is in CSV format and is located on GitHub, so we will use the `read_csv` function to read it.
In fact, in Pandas, you can read data in various formats. To see all the functions for reading data, you can type `read` and then press the tab key. A prompt will appear with all the functions, showing that you can read data from CSV, Excel files, JSON, parquet, and so on. You can also read from sources that support SQL. For example, if your data is stored in a Postgres database, you can create a connection to it and read data directly into Pandas.
Let's see what kind of object we created using the `read_csv` function. It is a DataFrame class, which is used to represent two-dimensional data, meaning data that has rows and columns.
Let's take a look at the data we have. The first row contains all the columns we have. In this case, we have various data about passengers, such as their name, gender, age, and a boolean flag indicating whether they survived the Titanic disaster.
You can create a DataFrame not only by reading data from files or SQL tables but also from Python objects, such as dictionaries. How can we do this? We need to call the `from_dict` method of the DataFrame. This method takes a dictionary where the key is the column name and the value is the list of all values for that column.
Thus, if we pass a dictionary with two keys, A and B, where A has values 1 and 2, and B has values 3 and 4, we get a DataFrame like this.
Now, I will show you how to write data from a DataFrame back to files or Python objects. After that, we will move on to working with the DataFrame itself.
To write data, you need to use the mirror function. Just as there is a `read` function for reading, there is a function for writing. For example, if we want to write data back to a CSV file, we call the `to_csv` function and specify where we want to save our DataFrame. I saved it in a local directory as a file named `tmp.csv`, and here it is. You can open it and see that it is a CSV file containing all the data from the DataFrame.
Now, let's move on to analyzing the DataFrame itself. The first function you can call on a DataFrame is the `info` function. It provides general information about the DataFrame, such as the number of records or rows, a description of all columns, their names, and data types as determined by Pandas. In this case, it identifies integers as `int`, floats as `float`, and strings as `object`. It also describes how many non-null values we have, meaning some columns may have values equal to NaN, which indicates missing values.
To simply get the size of the DataFrame or to know which columns we have, there are other functions. For example, the `shape` function returns the size, meaning it returns a pair of the number of rows and the number of columns. The `columns` function returns the names of all columns in the DataFrame.
We have many elements, almost a thousand, and if we want to see just the first few elements or the last ones, there are also functions for that: `head` and `tail`. The `head` function will return the first three records, and if we want, say, the last five records, we can use the `tail` function.
Another useful function for getting all data types in the DataFrame is the `dtypes` function. It returns the types of all columns in the DataFrame.
Now, let's see how to filter data, meaning how to select certain columns or specific rows. For example, if we want to take only the `name` column, we write `name` in square brackets and get the `name` column. If we call the `type` function on this expression, we will see that it is no longer a DataFrame; it is a Series.
A Series is a data structure in Pandas used for storing one-dimensional data. It is essentially like an array, but the index can be non-numeric and does not have to be sequential. In this case, we have a numeric index, and it looks like an array, but it can also have string indices, and the elements do not have to be sequential.
If we call the `shape` function on this Series, we will see that it is one-dimensional, meaning it returns only the number of rows, and there are no columns.
You can apply row and column selection together. For example, if we want to take the `name` and `age` columns and the first three records, we can do it like this. In this case, if we need to get more than one column, we should pass them inside an array.
If I specify them in reverse order, they will return in that order. If I want to select by both rows and columns, I can use the `loc` function. For example, if I want to take the fifth, tenth, and fifteenth rows and only the `name` and `age` columns, I can write it like this.
Now we have returned a DataFrame where we took three rows with indices 5, 10, and 15 and two columns: `name`. In fact, this is not the fifth position but the sixth, as indexing starts from zero, but I think the meaning is clear.
The `loc` function works with column names and index values. If I want to filter elements not by column names and not by index values but by the order of columns and the order of index values, I should use the `iloc` function instead of `loc`.
For example, if I want to take the same rows as in the previous example but take the first and second columns, I should write it like this. The columns are numbered starting from zero, just like arrays in Python.
Now we have identified the first and second columns. Let's check if this is true. If I call the `columns` function, we can see that these are indeed the first and second columns.
In addition to specifying row and column numbers, you can use Python slices to indicate ranges. For example, if I want to get rows where the index goes from five to ten and get the first three columns, I can do it like this.
Now I have obtained all rows where the index goes from 5 to 10 and the first three columns.
Next, we move on to another cool feature of Pandas: the ability to filter data using a boolean mask. Instead of explicitly stating that we want to take, say, the fifth row or from the fifth to the tenth, for example, if we want to get all adult users, we can easily do this by saying we want all users whose age is greater than 18.
Inside the square brackets, we specify the boolean mask by which we filter the data. Let's check what this is. It is visible that this is an object of type Series. If I write it like this, it will be a Series object where we have boolean values, indicating whether it is true or false for each row.
We pass this Series into the square brackets of the DataFrame, and thanks to this, we filter the data based on this mask.
In addition to standard comparison operations like greater than, less than, and equal to, Pandas has an `isin` function that allows you to compare with a set of objects. For example, if we want to get users whose age is either 20 or 30, we can do it like this.
Now we have obtained users whose age matches only the values we provided. You can also pass not just such values but a column from another DataFrame.
Pandas also supports boolean operations. If we need to perform several boolean checks simultaneously and then combine them using `and` or `or` operations, we can do that, but the syntax will be slightly different.
For example, if we want to check if the age is either 20 or 30, we can write it like this. Each boolean operation should be enclosed in parentheses; otherwise, it will not work.
Here, we use the Pandas `or` operator, which is represented by the pipe symbol (`|`). If we want to write an `and` operation, we use the ampersand (`&`). In this case, it will return zero values because a user cannot be both 5 and 10 years old at the same time.
So, we returned an empty DataFrame. But if we write back `or`, we will get users whose age is either 20 or 30.
An important function to know that returns a boolean mask is the `notna` function. As you can see, it also returns a boolean mask for each row and checks if there is an element in that position.
For what might this be useful? For example, if we want to filter data where we have missing values, we can call `notna` for each column or call it for the entire DataFrame. After that, we can pass this mask into the filtering construct.
We can write it like this, and we will have a DataFrame without missing values for age. In addition to the `notna` function, there is the `isna` function, which, as the name suggests, probably returns the opposite, meaning whether the element is missing.
We can also call the `sum` function to count the number of missing values for age. It is visible that for 177 people, we do not know the age.
The last thing I wanted to show regarding filtering is the ability to use the `loc` function and pass a boolean mask to it. For example, if we want to output the names of people for whom we know the age, how can we do this?
First, we pass the mask, and then we write the column we want to output. To use this method, meaning to pass a boolean value for filtering and specify column names, we need to use the `loc` function. It will return the names of all people for whom we know the age.
Now, let's move on to sorting data. For example, if we want to sort people by age, we use the `sort_values` function and pass it the column by which we want to sort. After that, I displayed the first 10 records to see what we have.
It is visible that we have younger ages, even below one year. These are indeed infants who are just a few months old.
There is also the possibility to sort by multiple columns simultaneously and in different orders. For example, if we want to sort by age in reverse order, meaning to show the oldest people first, and then if people have the same age, sort them by name in alphabetical order, we can do that.
To do this, we pass an array of columns by which we will sort. First, we sort by the first column, and then for all elements with the same value, we sort by the second column, which is the name.
In the `ascending` parameter, we also pass a list of True or False for each column. False means to sort in descending order, and True means in ascending order. So, we will sort in descending order for age and in ascending order for name.
Now, let's move on to how to combine multiple DataFrames. For this, I will create a temporary second DataFrame, copying values from the first one. To copy, I use the `copy` function and pass the parameter `deep=True`. This means that a true copy occurs, meaning not just a reference is copied, but all elements inside are copied.
If I change the value in the second DataFrame, the first one will not change. The first way to combine multiple DataFrames is concatenation, meaning concatenation by rows or by columns. For this, we use the `concat` function.
Let's concatenate them and check the size of the original DataFrame. It is visible that the size has increased. Let's check the concatenated DataFrame. The number of records is twice as much, meaning we simply appended the records from the second DataFrame.
In addition to concatenating by rows, you can concatenate by columns. For this, the parameter is set to 1, meaning the DataFrames should be concatenated by columns. Now we have twice as many columns, while the number of rows remains the same.
If we display this DataFrame, we can see that we simply copied it. Since they are the same, we now have all columns duplicated. For example, the `class` column appears twice, the `name` column appears twice, and so on.
Besides regular concatenation, a very useful function is to join DataFrames. This is essentially the same join used in SQL databases. By the way, if you do not know SQL and do not know what a join is, you can check out my video on SQL.
To demonstrate what a join is, I will now create a temporary new DataFrame that we will join with our original DataFrame. First, I will create an empty DataFrame and set its index to be the same as our original DataFrame so that they have the same number of rows.
After that, I will create a `Passenger ID` column, which is the column we will use for the join. Then, I will create a column that represents some useful information from another DataFrame, similar to another table in the database that we want to join.
For example, this will just be a boolean flag indicating whether the ID is even or odd. I will name it `even ID`, and then I will use the `apply` function to apply a lambda function to each value, checking whether it is even or not.
Now we have created this temporary DataFrame. Let's take a look. It is visible that we have a passenger ID and a boolean flag indicating whether it is even or odd.
Now we want to join it with our main DataFrame. How can we do this? We can use the `merge` function. What does this function take? It takes two DataFrames that we are going to merge, as well as the method by which we are merging them, just like in SQL databases.
There are several types of joins, but since we are actually using the DataFrame, it is clear that all IDs will match. In fact, it does not matter which join we use; any will return the same result.
Now, let's look at the resulting DataFrame. It is visible that we have all the columns that were in our original DataFrame, and the `even ID` column has been added, which shows whether our ID is even or odd.
Now we move on to the analytical functions available in Pandas, which allow us to calculate statistics over rows or columns. The first and probably simplest function we can use is to count the number of non-null elements.
We can count them for the entire DataFrame, for example, by writing `df.count()`, and it will count the number of non-null elements for each column. It is visible that for some columns, there are no non-null elements, while for others, like age, we have quite a few missing values.
We can also call all these functions on a Series. For example, if we want to count the number of non-null elements for just one column, like age, we can do it like this.
We can also calculate the mean and median. For example, if we want to do this for age, we can do it like this for the mean and similarly for the median. We see that the average age is 29, while the median is 28.
To get a detailed breakdown, we can call the `describe` function, which will return several statistics at once. If I call the `describe` function on age, it will return several statistics, some of which we have already seen, such as the mean and count.
It also returns quantiles, such as the 25th percentile, the 50th percentile (which is the median we already saw), the 75th percentile, as well as the minimum and maximum values.
Another very useful function in Pandas is the `groupby` function. It allows you to group data by one or more columns and then call various functions for each group. For example, if we want to calculate the average age separately for men and women, we can do it like this.
We provide the column by which we are grouping because we do not want to calculate the mean across all columns. After that, we call the function for what we want to do, which is to calculate the mean.
We can also call the `describe` function, and it will return all statistics, including the median. We see that the median age for women is lower, meaning men are generally older here.
As I mentioned, `groupby` allows you to group by multiple columns. For example, if we want to group by gender and whether the person survived, and calculate the mean age, we can do that too.
We pass a list of columns by which we are grouping. First, we group by gender, and then we group by whether the person survived or not. We also take the age column and call the `agg` function, where we can pass a list of aggregate functions we want to calculate.
This way, we can calculate not just the mean or median but also other statistics.
Another useful function in Pandas for discrete data, especially for data with a few unique values, is the `value_counts` function. It allows you to count the number of elements in each group. Essentially, it is like doing a `groupby` followed by a `count`.
For example, if I want to count the number of men and women, I can do it like this. It will show me how many men and women I have. This is the same as writing a more complex expression and getting the same result.
The last analytical function I want to show you is the correlation function. There is a `corr` function in Pandas that returns pairwise correlations between all columns in the DataFrame. This can be used to understand how certain data points depend on each other.
Now, let's move on to data visualization, meaning creating graphs. We will start with the age column. For example, if we want to create a histogram of the age distribution of passengers, we can use the `plot` function.
We take the Series from the age column and call the `plot` function, specifying the type of graph, which in this case is a histogram. Now we have our histogram drawn.
However, if we feel that the number of bins in the histogram is too small, we can increase it, say to 20. It is visible that most people are between 20 and 40 years old, and there are also quite a few infants.
Now, let's try to draw another graph. Instead of a histogram, we can approximate the age distribution using a KDE plot. We write almost the same thing, just changing the type of graph to `kde`.
Now we have our KDE plot. However, it is clear that ages below zero do not make sense, and probably ages above 100 do not either, as the maximum age is around 80. We can specify limits for the x-axis that we want to use, for example, from 0 to 100.
Now we have this distribution plotted. What if we also want to examine this distribution separately for men and women? We can also use the `groupby` function as we did for analytical functions.
Now we have drawn two graphs, but it is unclear which is which. To clarify, we use the `legend` parameter and set it to True. This means that we will label each graph, and now we can see the distribution for men and women separately.
We have learned how to filter data, sort it, select the values we need, create graphs, and calculate analytical functions. However, we have not yet learned how to modify data in DataFrames and Series.
Let's see how to do this. I will create a temporary DataFrame that will be the same as our original one, and now we will experiment with it.
The first way to change the value of a column is to simply assign it a constant value. For this, we take the `class` column and set it to 1 for everyone. Now, if we call `value_counts` on the `class` column, we will see that the class value is 1, while our original DataFrame still has three classes.
The second option for creating a new column or modifying the current one is to assign it a Series or an array. For example, if we want to create a column that indicates whether a person is an adult, we can do it like this.
As you probably remember, this will be a boolean mask Series containing True or False for each row. This Series will be assigned to this new column. Let's print the first 10 values of this DataFrame.
It is visible that for people older than 18, the `is_adult` column will be True, while for records where the age is not specified, it will be False, and for people younger than 18, it will also be False.
We can also change values not for the entire column but for only certain records. For this, we can use the `loc` function, which we already used to get data.
For example, we want to create a column that will contain a boolean flag indicating whether a person survived and is an adult. How can we do this?
First, we assign the value indicating whether a person is an adult. After that, we use the `loc` function to change the values that are incorrect.
Now, what I did here is first assign boolean values for whether a person is an adult. After that, we need to set the value to False for those situations where a person is an adult but did not survive.
So, I take the `loc` function, get all values where the user did not survive, and for our new `is_adult` column, I set it to False.
Let's see what we get. We will print more values to check, for example, 20 records. Here, we see that for the first person, `is_adult` is True, while `survived` is False because they did not survive.
For the next person, they are an adult and survived, so `survived` is True.
In addition to renaming the values themselves, you can also rename the column names. For this, you can use the `rename` function, where you can pass a dictionary where the key is the old column name and the value is the new one.
For example, if we want to rename `is_adult` to just `survived`, we will output one element just to check the columns. I forgot to specify `columns`, but now we can see that our column is named `is_adult` instead of `survived`.
Instead of using a dictionary and passing values for each column separately, you can use functions. For example, if we want to rename all columns to lowercase, we can do it like this.
Now we can see that all columns are written in lowercase, while before they were not.
The last modification method I want to show is using a lambda function. For this, we can use the `apply` function. For example, if we want to convert names to lowercase, we can do it like this.
We specify a lambda function that will be applied to each name value. This will be a string, and we apply the function to convert it to lowercase.
If we look at the values now, we will see that they are in lowercase. However, for strings in Pandas, there is an even faster solution. There are vectorized functions for strings.
If we want to do the same thing without using the `apply` function, we can do it like this. We create another column and write it like this.
Now we will see that this is the same, meaning the same names converted to lowercase.
Thank you for watching this video until the end. We covered how to read and write data in Pandas, how to create graphs, how to calculate analytical functions, how to select the values we need, and how to write new ones.
If you liked this video, please give it a thumbs up and subscribe. Let me know in the comments if you found something new.
See you in the next video! Goodbye!