Python for Data Science and Data Analytics Full Hands On Learning with Projects Full Course

If you're an aspiring data analyst, data scientist, or you are aspiring to get into the field of machine learning or AI, then mastering the basics of data and analytics is your starting point. In this comprehensive 6+ hour course, we are going to start with the Python implementation of data analytics. We are going to look into the data analytics basics when it comes to putting that in Python programming. After this, we are going to get into the A/B testing theory, which is fundamental for any data analyst or data scientist who wants to drive experimentation changes in the product, like UX design to the algorithms using the data.

This course will consist of three parts. The first part will be dedicated to the Python data analytics course. In this Python data analytics course, we are going to cover the basics of performing data analytics, including data visualization and data preprocessing in Python. After this, we're going to get into the second part of the course, which consists of the theory of data analytics and data-driven experimentation. This is fundamental for any data analytics and data science professional. Here you are going to learn the A/B testing theory, starting from the hypotheses, the business problem, up to the point of conducting a data analysis on that collected data to make a data-driven decision for different sorts of online problems.

Then we are going to get into the third part of the course. In this third part of the course, we are going to conduct two end-to-end portfolio projects. The first one will be related to A/B testing, where we will conduct an end-to-end A/B testing online analytics-related project which you can also put on your resume. And in this 1.5-hour section, we're going to cover from the basics of A/B testing results and analysis in Python to the actual implementation of it and conducting your data analytics in Python programming. Finally, the third part of the course will consist of two separate end-to-end full data analytics projects. The first one will be about online testing where we will use the data analytics as well as Python programming to derive the landing page UX design decision on the landing page of Lunch. And then the second portfolio project will be another one-half hour end-to-end data analytics project where we are going to look into the data analytics for the Super Store project. So those two projects, in total of 3 hours, will be a great way to implement the theory into practice in an actual business real-life setting.

That data scientist and AI professional, and I've been in this field for more than 5 years. I'm co-founder of Lunar Tech, where we are making data science and AI accessible to everyone: individuals, businesses, and institutions. So here is what we are going to cover as part of this full data analytics course. In the first part of the course, we are going to cover the data analytics in Python programming. So, it is expected for you to know some basics in Python, but not more. We're going to learn how to load data in Python using pandas, how to do data wrangling and data preprocessing using libraries such as NumPy and scikit-learn. Then we are going to look into the data preprocessing techniques, how to do sourcing, filtering, as well as data aggregation, how to join data using different joins including inner join, left join, left anti join, right join, how to do different statistics-related tasks, including calculation of data, um, descriptive statistics for our data using Python. Then we are going to do data sampling in Python. We are going to learn different data sampling techniques, and we are also going to look into data visualization in Python, which is really important as a data analytics professional when it comes to bringing the theory of data analytics into practice.

So after this, once we are done with the practical programming section for the data analytics in Python, we're going to get into the second part of this course, which is about A/B testing. Here we are going to look into a quick high-level theory behind A/B testing, and then we are going to dive deep into it. We are going to learn this idea of A/B testing, online experimentation, and how data analytics is relevant for A/B testing. And we are here; we are going to learn this entire cycle of A/B testing from the design up to the data analytics or the final results. Be prepared to learn here the concepts like primary metric, the design of the test, how you can design a proper A/B test, including choosing the right parameters for your test, calculation of the minimum sample size. So as a prerequisite for this part of the course, it requires for you to know some fundamentals in statistics. So understanding the basics behind probability, probability theory, this concept behind normal distribution, how you can use a sample to derive insights about your entire population. And once we are done with this theory behind A/B testing and we have also learned how you can conduct the data analytics and final data analysis for your A/B test, we are ready to go into the third and final part for our data analytics full course.

In this third part of the course, we are going to conduct two end-to-end case studies. In the first one, we are going to conduct data-driven decision-making for Lul Text landing page, where we are going to use data analytics, data visualization, as well as A/B testing to understand whether we need to replace our current button. So here, expect to use Python in this 1.5-hour course. We are going to conduct data wrangling, data preprocessing, also data visualization, and then we are going to analyze our results and make a decision by using the theory that we learn as part of the second part of the course as well as the programming that we learn as part of the first part of this course. Then, once we are done with this first end-to-end project, we are then ready to go on to the second project as part of this final part of the course, which will be about a pure data analytics case study. In this case study, we are going to conduct the data analytics for our Super Store. So here we are going to start with the overview of this analysis, and then we are going to analyze Superstore customers. Then we are going to see what techniques we can use and how we can conduct Superstore customer segmentation analysis in Python. Then we are going to analyze the revenue of the Super Store by customer segment. After this, we are going to explore the customer loyalty at the Super Store, and then we are going to finish off with the insights that we derived based on this analysis for our customers from the sales and just in general. So we are going to analyze the sales of these customers by segment, and then we are going to conclude at the end of this course. Expect to learn all the essentials for your data analytics journey. So, without further ado, let's get started.

Hi there, and welcome back. In this demo, we are going to talk about how to load data and view this data in order to obtain more information about a certain data that is provided to us. We're going to learn how to load CSV files, how to load TXT files, how to load Excel files, as well as JSON files, and also how to load SQL databases. For this one, we are going to use predominantly the pandas library, a library that we spoke about in the previous demo, but we are also going to use some other libraries. All right, so without further ado, let's actually go ahead and learn how to load CSV files in Python. So for that, the first thing that I'm going to do is to import the pandas library: `import pandas as pd`. And then the next thing that I'm going to do is to pick the name of the CSV file. So you might notice that in the left-hand side, in our PyCharm, you can see that we have a file called `percent_bachelors_degrees_for_usa.csv`. This is a CSV file containing the following data. So you can see we have information about the year, agriculture, architecture, art and performance, etc. So you might have already guessed that we are dealing with the data that describes the percentage of the females that have completed a B degree in the corresponding fields and the corresponding year of it. Well, let's go ahead and load the data in Python. For that, what I'm going to do is to use this pandas library, and as an acronym name, we usually always write a name of `pd` for the pandas. So let's go ahead and name our data frame, which we will call, let's say, `data_csv`, and this will be equal to, and here we need to take the name of the library we are going to use, which is `pd`, which stands for pandas, dot, and then `read_csv`. As you can see, we are getting already many options. So we have `read_csv`, then we have `read_excel`, we have `read_html`, `read_json`, `read_parquet`, `read_pickle`, `read_sas`. So here you can see all sorts of data formats that you can, file formats that you can import, and we are going to learn a few of them, and those are the most popular file formats that you can expect whenever you are entering the data science field. So, as we have a CSV file, we are going to use the `read_csv` option, option, and within the parenthesis, we always need to specify the name of the file we are dealing with. So you always need to put the name of the file as it's a string. So within the quotation mark and here, so let's go ahead and actually print our data frame to see what is actually going on. Here we go. So you can see we are getting our data nicely. So we see that the headers are recognized, so we see the column names, we see also here the indices corresponding to our observations, and this is really a great way to look into your data for the first time. In the same way, by using exactly the same function, so `read_csv`, we can also load a TXT file. So TXT and CSV files are pretty similar to each other. So in case of CSV files, which stands for comma-separated values, we do not usually specify that the separator is a comma. So, as you can see in here, this is a CSV file, and the values corresponding to each of the columns are separated by a comma. And if we're dealing with a TXT file, in the TXT file, we don't really know what the separator can be. Sometimes it's the comma, sometimes it's the space, sometimes it can be an entirely different character. So it's really up to the data that is provided to you, but one simple way to load a TXT file by using exactly the same function, so the `read_csv` is the following. So here we have two different TXT files. In here, we have the `student_grades.txt` and the `student_schools.txt`. Go ahead and use them. So we have `data_txt` and it's equal to your `pd.read_csv`, and then here within the parentheses, we have, um, let's say `student` and then `schools.txt`. So this is the path, but before moving on towards the other arguments, let's actually go ahead and click on this to see how it looks like. So you can see we are dealing with a TXT file where we do have the header. So we have the name, school ID, and country, which all represent the name of the corresponding columns. As you can see here, we have the name here, we have the school IDs, and here we have the countries. And another thing that we can notice is that we are dealing with a separator in the form of commas. So therefore, what we need to do is to provide, in here, by the way, instead of `txt`, we would just use `csv` for simplicity. And here we will mention that the `header` is equal to zero, so the first row corresponds to the header, which means that that row should not be counted as data. And then the next argument that we will use, just for knowing how to use it, is the separator. So here we will mention that the separator being used to separate each column's value in the row is the comma. But if you were to be supplied with the data in a more difficult format where you had a different separator, so let's say the separator is a percent symbol, then here you need to specify that your separator is this, is this symbol. So whatever the symbol is used, the character is used to separate your data, that's exactly what you need to put in here such that Python can understand when it needs to cut and needs to take that value and say that this value corresponds to that specific column. And then the same holds also for the header. If your header is not present, then you need to specify that in your argument `header`. All right, so let's go ahead and load this data and see what is underneath. `print(data_txt)`. Here we go. So as you can see, we nicely get all seven rows. So the first name is Tina, the last name is Anna, and then the country, Canada, and last one, Armenia. So let's go ahead and check it in here. So I always recommend to check the first and the last rows of the database to make sure that you correctly have loaded your data and you are not missing any information from your database.

The next thing we are going to learn, and you have the name of your first sheet equal to sheet one, and then the second one corresponding to another name, and you have multiple of those pages in your Excel file, is how to load Excel files. So let's say you have an Excel file and you want to load only the first page. Well, for that, what you can do in here, I'm not going to look into a specific Excel file; feel free to search for an Excel file or maybe one that you can create yourself and then create your own pages within your Excel file and try to load that in Python. But for now, let's assume that we do have that Excel file in our PyCharm environment, and we are going to load that. So `data_excel` will be the name of the data frame that we will store our data, and then the function we can use is `pd.read_excel`, and then here we have `file_xlsx`, which is a common extension of the Excel file. `file.xlsx` is the name of your Excel file, and here I'm making an assumption that your Excel file is within this Python for Data Science or your own folder, the one that you are currently using in PyCharm. So here, then the next thing we need to specify is the exact spreadsheet we are looking into because otherwise, you will get an error in PyCharm, and PyCharm will not recognize where exactly it needs to look for the data. Therefore, we need to use this argument called `sheet_name`, and here you need to specify the name of your exact spreadsheet you are looking for. It can be that it is the default acronym usually used in Excel, but in case you have renamed it or someone has renamed it, then you need to specify that specific name. So it can be, for instance, `first_spreadsheet`, let's say if that's the name of your first spreadsheet, or it can be `Sheet1`, which is usually the common convention used in Excel whenever you are not changing the name of your spreadsheet. And this is how you can read an Excel file. I won't be running this code because we do not have the `file.xlsx` in our folder, but this is something that you can experiment with yourself. And another common file format that you can expect is the JSON format. Here, once again, we are following the same ideas in case of Excel files. So feel free to go ahead and look for a JSON file online, download it, and try to load that into your PyCharm environment, but this is the way that you can load JSON type of data. `data_json = pd.read_json(file_name)` So this will be the name of your file. We can also make this more convenient, so let's make it `file_name`. Here also `file_name`, and this is the only thing that you need to specify. So this will be the name of your file, and here once again, I'm assuming that your `file_name.json` is actually in this folder that you are currently running; otherwise, you need to specify the exact path of the file that you have. So once you write this, then you should be able to successfully load your JSON type of data in your PyCharm. And then finally, we will look into a way to load SQL databases. SQL databases are common database formats whenever you are working with big data. This is very common in the field of data analytics, but I think it's still worth to know at least the commands and the library you can use in Python in order to load this type of data. So let's actually go ahead and import the corresponding library we can use to load SQL databases, and the library is called `sqlite3`. So for that, we will do `import sqlite3`. Then what we need to do first is to make a connection with this SQL database, and that's exactly what we can do by using this: `connection_db = sqlite3.connect(database_name + ".db")`. And in this way, you will make a connection with the corresponding database. The way SQL works is that in SQL we are creating databases, and within each database, we can have multiple tables, and each table has its own name, and then within each table, when we are loading that table, we can run queries. I won't go too much into details about what is SQL, how you can use databases, how you can create tables, and how you can run queries because that's outside of the scope of this course, but I would highly suggest you to at least learn the basics of SQL. It's not necessary to enter the field; therefore, it's also not included as part of this course, but it's good to know at least what SQL is and also how you can use it and what the functionality of SQL is in the entire world of data science. So I will include some resources about SQL and the usage of it in the resources section, but just know that in order to be a technical data scientist, at least to enter the field of data science, you do not need to know SQL. It's something that I would highly suggest you to learn as you grow your career, but it's not a must-know. So once we have made the connection with our database called `database_name.db`, then we can specify the exact query that we want to run, and in this specific scenario, what we mean by the query is that we will use the commands commonly used in SQL in order to select all the rows within a specific table. So in our database, we can have multiple tables, and here I will assume that we have a specific table from which we want to import, let's say, the first column only. For that, what we need to do is to do a query, let's call it a query, let's say it's our first query, `query_1`, it's equal to, and then here we have a quotation mark, and then `SELECT`. This is a common way of specifying that we want to select specific variables from our table, and here we can specify the name of the column that we want to import, let's say `col_one`, and then we need to specify `FROM`, and here we will specify the `table_name`. And in this way, the `query_1` will then go ahead and select the first column from the table with the `table_name`. So this is one way of running a query and selecting just one variable. We could also write a query, we will call `query_2`, and this will select all variables from a `table_name`: `SELECT * FROM table_name`. In SQL, whenever we are saying `SELECT *`, it will go ahead and select all the columns included in that data. Something that we usually prefer instead of...

Uh, just selecting one variable. So you will select the certain variables only if you are specifically looking for those features, but otherwise, I would highly suggest you to include all the Cs. All right, by the name of the data print, we want to load this data in, and then we read_csv. As you can see, we already see something. So here you can see that we have three different functions we can use. We can use reor SQL; we can use reor SQL uncore cury; or we can use a read uncore SQL uncore table. They do defer in the way they import the data. So for one, for instance, you can specify; also another one, you can specify the index of the com you want to import. The most generic one is the read uncore SQL, similar to the read un CV. So we are going to use a one.

The next thing we need to specify is the uh cury, and the connection is specify cycore 2. And then the next thing I need to do is to specify the connection. So once you run this code, what this will do is to make a connection with your SQL database. It will then uh specif by the cury and it will go ahead and select all the features and import all the features from a table called table name, all the variables, and it will then uh be um stored in the pandas data frame. This is all for this demo, and I will see you in the next one.

In this demo, we are going to continue the process of looking into the data. As in the previous demo, we learn how to load different sorts of data, and in this one we're going to look into it. We're going to learn how to explore the data and how to preprocess it. We are going to discuss the uh inspection of the data, getting information about it, getting to know what the shape of the data is, how to identify missing values, how to drop the missing values, how to fill in missing values, how to get the type of the data you are dealing with, how to access different rows in your data frame by using the infamous iog and lo, and what is the difference between the two.

So from the previous demo, we still have the CSV file, and we saw that uh we got this data structure in the CSV file describing the percentage of the bach degrees uh consisting of woman in the USA. So this how the data look like: we had the year, agriculture, architecture, art and performance, etc., and then uh in total we got 18 columns in this data frame and 42 rows. So the rows are those observations, so you can see in here, and the columns are all the features included in data frame, and we got only one feature describing the year, and the rest of them are all the names of different sorts of Bachelor studies. So as you can see, we have agriculture, architecture, we have social sciences and history, public administration, etc.

The first thing we are going to look into is how to use the Heather functionality in Python in order to get a snapshot of the data. So what we can can do here is to uh write down the name of the data frame, so data.CSV, and what we need to do is to do Dot and then head for the header, and then here uh inside in it uh we can leave it empty. So uh when we do that, what we will get is the following output. So let's actually go ahead and remove that one from the printing temporarily. As you can see, it will print for you the first five rows with all the columns, and if we specific we are looking for each number of rows to be presented as part of the snapshot, then we can specify that as part of the head function. So in here, for instance, we can say show to us the first 100 observations in this data frame, and in that case it will print for you the first 100 observations, but as we have only 42 rows, it means that it would need to print for you all the rows. If we change this to 20, let's say, then in that case we will get the top 20 observations present in the data, and this is how it looks like. All right, so this is about header function, just a good way to uh have a first look at what kind of variables you have, what are the first two columns, what are the last two columns, what is the number of observations you got by looking in here, what is the number of columns you got, and uh also what are the different sorts of variables you got and data types in your data frame. Just by uh visual inspection, you can see that we have, for instance, the year column which consists of the integers; we have the agriculture, architecture, and all these other variables that are of a floating uh number type, which means that we got a number and then we got Dot and then what comes up to the daymr points.

And if this header function will print for you the top X observations, we can do exactly the same only from the bottom observations, and for that we can use a function called SC. So uh we can print, for instance, the last 20 columns by using this dataor CSV.do tail and then within the parentheses the amount of rows that we want to see from the bottom up. So let's go ahead and print that, and as you can see, if the header function will showcase the first 20 rows, the tail function will showcase the last 20 rows. So you can see 22, 23 up to 41. So this is a great way to see uh how the uh the first few rows look like and how the last few rows look like.

The next thing what we can do is to use this info function in order to optain more information about our columns, so they data type specifically. So this is the output of the info function, and this is the number of columns you have. In this case, the year is the First Column, the agriculture is a second column column, and then the social sciences and history is in the last column, and then we have the count of the nonn values. As you can see, all the columns have 42 non-n values, which means that we do not have any missing observations. Then we have the data type corresponding to that specific feature, and we already saw from the snapshot that uh the year was the only variable of integer type, and everything else was floting uh data type, and this is exactly the confirmation for dat observation.

The next thing what we can do is to identify the missing values and drop the missing values. So uh from this we can al already see that we do not have any missing values, but let's actually go ahead and learn how we can do that. So let's say we want to uh drop all the missing observations. For that, what we can do is to do print; is to actually um take the name of the data frame CSV, and then what we can do is to do drop and then Na and then parentheses. So in this way you will be dropping all the Naas, so all the cases where your observation has an NA for that specific column. So as we do not have any missing values in our data frame, this will not do much to our case, but I think it's really important to know how to drop missing values in case you want to quickly remove them from your database. Let's say you do not want to drop your na, so the missing values in the form of Na, but you want to fill them with a certain value. What you can do is to use this uh field na function, and within the parenthesis you just need to specify what is the value that you want to use to fill the nas. So in here you can for instance decide to put n instead of Na, and this will simply go ahead and feel all the values where it's written na, it will replace it with null values.

So let's say you have another issue with your data: so you have rows that are exact copies of each other. One function you can use is what we call drop duplicates. So for that you simply need to take the um data frame's name, and you need to drop and then underscore duplicate, and this will quickly remove all the duplicates from your data frame. So let's actually go ahead and change the CSC file and see whether we can uh nicely remove the duplicates. So let me copy paste this few times, and as you can see it's the uh exact copy of the third row, and now we have it in the fourth and fifth and sixth rows. So let's go ahead and check check whether this dropcore duplicate really removes uh those duplicates. So let's print the actual CSV file before removing the duplicates and after removing the duplicates in here. Let's see. So in here you can see that this is the data frame with in total of 45 rows because we just added three additional rows. So previously we had 42, we added three, so we end up with 45 rows, and the number of cols is the same, and then we apply the drop duplicates function, and as you can see after using that function now we once again end up with the 42 rows, and if we go ahead and look into the specific column we duplicate, you can see that we got only one row corresponding to this here, and this is how we know for sure that dropcore duplicates really works and it will remove the duplicate from your data.

The last thing we will look into in this demo is how to access certain rows in a data frame depending on their index type. So uh sometime and actually most of the time we will get a data frame that has an integer as an index, and that's also what we got in our data frame. As you can see, the index is 0, 1, 2, 3, 4, so it's in the integer format, but there are also occasions when you will get your uh data frame with an index that is of string type. So you will see that instead of the index 0, 1, 2, 3, for instance, you will have ABC or A1, A2, A3 um Etc. So depending on this uh nature of the uh index that you are dealing with, you can then use either the iog or the log functionalities in Python in order to access different rows in the pendous data frame, and this by the way a common question you can expect during your programming related data science interviews: what is the difference between the iog and log and how you can use them.

So let's start with the iog. As our data frame already contains an integer based indices, and let's say we want to access the data in the index uh 10. So uh this is the row that we want to access, therefore what we need to do is to take the name of the data frame, so dataor CSV, and then we need to do Dot and then iope, and then in here we need to specify the row that we want to access, which is a 10, and let's go ahead and print this to see whether we are getting the correct data, and uh let's verify that by looking at the gear. So as you can see, the year is in incremental order um and we do not have duplicates in the year, so uh therefore if we see that the year is equal to 1980, then we have selected the right data and we have accessed the right row. So this is the output, and as you can see the year is equal to 1980, and this is all the uh information that is stored in the 10 row, and in this way you can access any row that you want in your data frame. So it can be for instance the first row or the last row, but it can also be a row somewhere in the middle that you want for for some reason.

And if you want to access a specific column instead of specific row, what you can do is to use again the log function, so data.log, and here instead of providing just one argument, you can then provide two arguments. So we always have the rows in the beginning, and then uh we need to specify the columns, and in case of um the in here uh what we did was to access the specific row, therefore we specified only the X, but if we want to access a specific column, we also want to specify the rows that we want to include, then we need to specify both the index of the rows and the index of the column. So let's say I want to access the column A2. In here I will then specify A2, and as I have specified here a column, this means that I want to take all the rows, so the indices corresponding to all these values. So let's see what the output of this one is, and as you can see here we are getting four, five, and six, so the value corresponding to the index x, y, and z, and this actually the column A2. So in this way you can specify not only the rows that you want to access but also the column. So let's say you only want to access a specific value in that column, let's say you want to access the second row and the second column. In that case, what you need to do, I'm sure you already have guessed that, is to specify the index of the row and the index of the column that you want to access. So the index of the row is y, and then the column that we want to access is A2, so this is the number that I'm chasing. Let's see what this will provide the value, and as you can see it provides five. And let's also look into the case when we are dealing with a string based index. So uh let's create for that a small data frame, and it is the following. Let's look into it, and as you can see this is the uh small data frame that has indices X, Y, and Z, and let's say we want to access the um data that is stored under the index X, so very first R. Well, what we need to do for that is to uh once again take the name of the data frame, and instead of using iog, this time we need to use the log and then the index name, which is similar to what we saw before. So uh the index name is X, that's what we want to access, and let's actually see what is the output of that. Print here we go. So as you can see, the first value is one, the second value is four, and the last value is seven. So by using the log functionality, we can access the uh row in a data frame where the index is of string type type, but if you go ahead and you use the iog in here, you will see that you will get an error, and the reason for that is because iog doesn't allow you to search with uh case when your index is of string type, and for those cases you always need to use the uh log function instead of iog.

Hi there, and welcome back to another demo. We are going to talk about three very important tasks that you can perform as part of your data analysis and data manipulation to kit. So we are going to learn how to do filtering, sorting, and grouping in Python. Data analysis and manipulation involve working with large amounts of data, and you can definitely expect this as part of your data science project, and then what you need to do is to extract meaningful insights. In this context, filtering, grouping, and sorting are really important techniques that allow us to organize, extract, and analyze data efficiently. Python has a very powerful tool Library called pendas that we also S as part of the libraries discussion demo, which can be used in order to perform grouping, filtering, and sorting in a very simple way.

So when it comes to grouping, grouping data helps us to analyze and summarize information and data based on specific criteria. Here we need to have at least one variable to do the grouping based on, but then you can also add extra other variables such that you can aggregate your data not only on one variable but on multiple variable. So let's say you want to uh group your data based on the gender or based on the region, and then you want to perform some uh descriptive statistics calculation. So you can calculate the mean, for instance, the standard deviation, the variance, the median, the mode, the minimum, the maximum, you get the idea. So you can then categorize your data into certain groups, and this way you can group your data and then you can obtain some meaningful information and analyze your data. This is usually a very important part of your data preparation process.

Another thing we are going to talk about is how to filter the data. So whenever you are filtering your data, it helps you to extract subset of data based on specific condition. So let's say you uh know a specific uh year that you are interested in, or you are interested in specific region, you are interested in a specific characteristics, then you can use filtering to filter your data to select only a subset of observations from your data and to perform all the analysis calculation and training of your model based on this specific subset, or it can also be that you want to identify the outlasts in your data or the noise in your data, and you want to identify the 99 percentile or the first percentile. So you want to identify the largest or the smallest observation your data and remove them from your data such that that you won't be dealing with a problem of overfitting as an example.

Another thing that we are going to learn as part of this demo is sorting. So sorting data helps you organize information in specific order. It can be in an ascending order or in a descending order. It can help you to visualize your data; it can help you to identify certain patterns, find extremes or outliers in your data; and sometimes it's also used as part of the time series analysis, or for instance to look into the sales performance, you identify the worst performing uh shops or the best performing shops, and it's essential part of the reranking and recommender systems. So whenever you are dealing with sge recommender systems, anything that relates to the order and importance, then sorting data comes really handy because you want to show the uh best information, the most important information to your customers, and the way that you can do that is by sorting your data. So when it comes to the ascending versus descending, the ascending relates to the case when the smallest values are at the top, and then the values would then increase, and then at the bottom you have the largest observations, and whenever it comes to the descending uh order of the Sorting, then we have the largest values at the top, and then the values would decrease, and then the smallest voles will be at the bottom.

So here I created a very simple data frame using penda library, and here we have four different columns: we have the name, the age, the salary, and the department of an employee, and as you can see we are dealing with eight observations. The name and the department are of string type, and the age and salary are of integer type, and what we want to do here is to sort our data under the data frame name data with respect to salary such that uh in the beginning we have the uh at the top we have uh employees with the smallest salary, and at the bottom we have employees with the largest salary. So what we expect is that uh this person, so this corresponds to saana, saana at the age of 19 from the Department of operations should be at the top because uh this person earns the Leist, and then we have um the uh largest earning, so the highest earning person which is named Bob, and Bob with the age of 20 should be at the very bottom because we want first to sort our data based on salary in an ascending order. So for this what we can use is the default um penders function called sordor values, so data.sortore values. So as you can see, we are already getting a recommendation for this function uh from Python, and uh here you need to specify based on which variable you are sorting, and then you need to specify whether it's an ascending or a descending. So um the reason why we need to specify the variables name based on which we are sorting is because here we could have also sorted the data based on the age, but uh instead what we want to do is to sort the data based on salary such that the highest earners will be at the bottom and the lowest earners will be at the top, therefore we need to specify that buy is equal to salary, and then when it comes to the uh other parameter or the argument in this function

which is sending this is a brilliant type of argument. So um, when ascending is equal to True, which is the default Val value, then uh it means that we are sorting our data data frame based on an ascending order. But if we want our data to be sorted in a descending order, then what we need to do is to change this value. So then the ascending argument should be equal to false. So let's actually go ahead and see that in the implementation.

So here we need to specify by is equal to and then the name of the variable based on which we are sorting, which is the salary. Then the next thing what we need to do is to mention ascending, which is the second argument; it's a brilliant valued argument, so it can only take values true and false, and the default value is true, which means that we are um sorting our data in an ascending order. I could have also skipped this argument; you can see in a bit that we are getting exactly the same uh result when we are mentioning this argument as an equal to true and without mentioning this argument, simply because the default value is equal to true. So let's actually go ahead and remove this for one case only; for the other such you can see that both will result in the same output. Here we go. So let's also add a line in between to make sure that we are getting everything nicely printed. Here we go. So as you can see here, here we are getting as expected Sana at the top and then Bob at the bottom. And in this way uh you can verify that your uh sorting occurred successfully and you have done it in an ascending order because the salary is increasing, and the lowest salary is at top and then the highest salary is at the bottom.

So uh the other thing that you can see here is that we are getting exactly the same result when we are mentioning the ascending equal to true. So this is simply because um there are certain arguments, and the arguments have certain default values in Python, and uh whenever you want to have the uh default value as your arguments value, you can also skip that argument, and you don't need to mention that specifically. Therefore, this results in the same output as this line. But if we do want to change the order, so we want to have another value corresponding to that uh argument which is not a default value, then we need to change that, and we need to specifically mention what is the value that we want. Which means that if we want to order this data, so we want to sort this data based on salary but in descending order, we need to use this argument, and instead of true we need to change this to false, because if the ascending is reported false it means that the descending is reported true. Hope this makes sense. So let's go ahead and print this. Here we go. So as you can see now we have the exact opposite of what we had before. So we have now the highest earner Bob with the age of 20 with a salary of 220k from the tech department at the top, and then the salary would decrease accordingly, and then at the very bottom we have Sana with an age of 19 with a salary of 10K from the operations Department because she's earning the least. So this is about sorting; this is all what you need to know, do keep in mind about this ascending and how you can use the parameters in this function in order to assort your data accordingly.

Nothing thing we are going to learn today is how to group your data. So let's say we want to group our data based on a department. So we want to know per Department what is the number of employees, or it can be that we want to obtain per Department the average salary. Let's go ahead and learn how to do that in Python. So let's say uh we first count the number of people in a department; for that what we need to do is to take the name of the department uh of the data frame which is data and then do and what we need to do here is to use the uh function called groupby, and this a a pendas function that we can use to group our data. And within Groupby we need to specify the variable based on which we are doing the grouping, and as we want to obtain the number of employees per Department, it means we are aggregating the database on department. So here I will mention then the uh Department which is the uh variable we are using for uh grouping our data, and then next thing we need to do is to do dot and then the operation that we are performing; in this case we are counting the number of employees. This means that I can either use the salary um column or I can use the age or the name in order to obtain the number of observations per department. So let's go ahead and do uh count. By the way, we can also uh even skip this part and we can specify that we uh want to just count the number of times the uh Department name appears, because this also will go and calculate the number of employees per department. So let's go ahead and see what I mean here. Then let's also add the closing parenthesis for the print. Here we go. So as you can see we are getting as an index for this uh new data frame the name of the department, so the variable that we use to do the grouping by, and then here we have Healthcare, operations and Tech. What you are getting here is basically the number of times uh each of this department appear here for the variable name, age and salary. So as you can see this is the column name, the first one, and then we have the age and then the salary. If we were to go ahead and actually specify the exact column that we want to do the grouping and then aggregation, so let's say uh the name, then we can expect to do the aggregation only based on the variable name. So as you can see here we are getting this new data frame, and it says that the department Healthcare got only one person working, and then the operations got two, and then Tech got five. If you go ahead and count here you can verify that information actually. So you have here Tech once, two, three, uh four and five, so five times, then the healthcare once, and then operations twice, exactly what we got here. So in this way you can do the uh you can count the number of times uh each of the uh observations appear uh based on the variable that you have chosen, and you can decide either to select the specific column to do the aggregation or you can also leave that part and you just uh implement the function and it will be uh it will do the corresponding operation for all the columns, so for the name, for the age and for the salary.

So let's actually go ahead and calculate the average salary per department. For that what I need to do is to change the name of the variable based on which I wanted to deaggregation, and then instead of using the function count I will use the function mean, and this function so the mean is the same as the average should calculate the mean or the average salary per department. And as you can see here we are getting that the healthcare department has an average salary of 170k, operations has 20K as the average salary, and the tech has the 113k as the average salary. So one way you can verify that this information is really correct is for instance by looking at the healthcare, and this is an easy way to do that because Healthcare got only one observation, and the average of the one observation is equal to that average, which means that if we look at the health care salary, so the only uh person who is from the healthcare department is uh the person with name uh Ellis, and then the corresponding salary is 170k, exactly the same number as we got here. We could also go and uh do the same calculation, only instead of calculating the average we could calculate per Department the uh minimum salary. So let's go ahead and do that. We see we are getting an error because we forgot a parenthesis in here. So now you get not the average but the minimum salary per Department. You could also do the uh maximum per Department; you could also do the um minimum age or the average age by Department to see uh the age groups uh accordingly. So for that what you have to do is to change the salary to a variable called age because we want to do the aggregation based on age, and here we can then change minimum, and in this way we can calculate the um average age per department. So as you can see from the Department of tech the um people are on average 141 years old; from the operations department they are on average 22 years old, and in the healthcare department they are around 65 years old. And this how you can do grouping in Python.

And then the final part in this demo is to look into the filtering, so how we can use the uh filtering in Python in order to uh select data observations based on specific criteria. So uh let's say uh we want to keep only the uh name and the information of people in this data frame that got a salary uh higher than certain threshold. So let's say uh people whose salary is larger than 100K. So for that what we need to do is like always mention the name of the data frame which should be filtered, and then here what we need to do is to add a square parenthesis, and within the square parenthesis we need to specify the constraint. So uh the uh first part, so this part where we are specifying the name of the data frame and then the square parenthesis, this means that look into the data frame and select specific observations, so keep only those observations in this data frame. The second thing you need to mention within the square parenthesis is the actual condition that this observation should satisfy in order to be uh kept and not to be filtered out. For that what we need to First do is to specify the condition and the column based on which the condition uh should be conducted. So data and their Square parenthesis and then the name of the variable based on which we are doing the filtering is the salary, so therefore we are saying look into this specific column, so data salary, and then we are saying that the salary should be larger than 100,000. So in this way what we are telling to python to do is to look into the data Frame data and then look into its salary column, select all the observations for which the salary is larger than 100,000 and then only provide the data corresponding to those observations. So let's go ahead and check it whether this doing everything correctly. Here we go. So as you can see we are removing all the observations from our data frame that do not have a salary higher than 100,000 and only keeping the observations to salary is larger than the threshold. So as you can see we no longer have for instance Anna in our data frame, we no longer have uh the current in our data frame or um the a fifth observation which corresponds to Kevin, and then s is also not included in our data, but only the people whose salary is larger than 110k are included in this data frame.

So let's say we want to add an extra condition, so we not only want to have people whose salary is larger than 100,000 but we also want to filter our people whose salary is too high, for instance who salary is larger than 200k, and in this case what we expect is uh from this data frame Bob should be removed. So as you can see now with this current filtering Bob is still included, but then if we uh also uh remove the people whose salary is larger than 200k then Bob should be removed. So let's go ahead and learn how to do multiple filtering, and here what we will use is the uh and operation because we want the two of the conditions mentioned here to be satisfied. So for that we need to separate the conditions using this parentheses. So then I will add here the second condition which will be based once again uh on the a variable called salary, and then we are saying that the salary should be smaller than 200 100,000. Here we go. So this symbol stands for and, and as you can see now we no longer uh get the information of the Bob because Bob got a salary above 200k. So in this way you can specify not just one but multiple conditions that your observation should satisfy in order to be filtered in this new data frame. And uh the final thing that we will look into is to how filter your data not based on the larger or smaller but based on specific values. So uh let's say you are interested in data of people whose age is equal to uh 65 and uh 20. So in this case uh you cannot uh you can no longer say that the age should be larger than certain value or smaller than certain value, or even if you do that it will be much more complicated then to just to just help to python to select all the observations for which the age is equal to to this specific values. And in those cases uh the uh isin functionality comes very handy. So for that what we need to do is to Simply once again take the name of the data frame, and then here we'll be uh mentioning again the name of the variable that we need to look into which is age, and then here we need to specify that is in, and what this uh function basically does is that it looks into specific values for this age and only keeps the observation that satisfied to uh those values. So uh in here I mentioned that I want information only for people with an age of 65 and 20. For that I need to put uh those two values within an array because they are not just one but two uh values. Here we go. So as you can see now we are getting only the data corresponding to people whose age is equal to 65 or 20. So this is how you can use the isin functionality to filter for specific values in your data, and this becomes even more handy when you are dealing with string type of variables. So you can no longer say that the corresponding uh column the observations value should be larger than or smaller something because you are dealing with shrinks, and in those situations the isin can be really handy. So it's really worth to know how to do the filtering based on specific values. And this is all for this demo where we learned about grouping, filtering and sorting in Python. Stay tuned and I will see you in next demo.

Hi there and welcome to another demo. In this demo we are going to talk about descriptive statistics. We are we are going to learn about calculating the mean, calculating the standard deviation, the variance, the mode, the median, different percentiles, quantiles for arrays, as well as we are going to learn how to get the descriptive statistics table for pandas data frame. So descriptive statistics play very important role in analyzing and summarizing your data. Here are a few reasons why I believe knowing how to calculate descripted statistics is important and also how you can use it. So first of all, descripted statistics help us to summarize our data. So descriptive statistics provides a concise summary of the main features of your data set. It can help you to understand the data by providing measures such as the mean, dispersion, so uh the variance and the shape of the distribution. Summarizing your data will help you to gain more insight about your data in an efficient way, making it much easier to interpret as well as to communicate with other people during the presentations whenever you want to explain something about your data. And also it's the essential part of every data science and data analytics projects. The first thing you need to do is to obtain the descriptive statistics about your data and present it to your stakeholders in order to tell a story about your data. Another thing you can do with descriptive statistics is to perform data exper. So descriptive statistics is a great way, a good starting point for exploring and understanding your data. It provides an overview of a data distribution; it can help you to identify certain patterns; it can help you to identify outliers by looking at the mean as well as the minimum and the maximum of your different variables; it can also help you to identify potential data issues, so the outliers, noising your data as well as missing values; it can also help you to understand the type of variables you are dealing with, because if you get an output from your descriptive statistics you'll get an understanding whether you are dealing with a categorical string variables or numeric variable or a floating data point uh or an integer data point etc. It will help you to further investigate your data and to have a good understanding what should be uh your steps in order to clean your data and to prepare your data in the best possible way for your machine learning model. Another thing you can do by using descriptive statistics is to get an understanding whether the sample that you have sampled from the main population it's a good representation of your population. So as part of the fundamental to statistic section in this course we'll learned about the difference between sample and population and how we use sample and we randomly sample a small part of your entire population in order to make conclusions about your population, but then the criteria is that your sample should be true and an unbiased representation of your population. By using descriptive statistics you can then compare for instance the mean and the variance of your sample to the actual population in order to get an understanding whether you are dealing with a good sample or whether you need to go back and then sample again in order to get a good sample. Another thing you can do is to visualize your data. So by using descripted statistics you can visualize your data in order to represent presented to different stakeholders; you can use histograms, you can use box plots, you can use bar charts or pie charts in order to represent your data in a clear and understanding way.

So without further Ado let's learn how to calculate different statistics for our array. So here I've created an array which consists of the following numbers, as you can see 100, 205, 20, 45, 100 and 46. So first thing we are going to do is to calculate the mean. The mean is the average of a set of numbers, and in this case it will be the mean of all these numbers, and it is calculated by summing up all these different numbers that we have here in the set and dividing it to the number of uh observations we got here. So as you can see the length of this of this array is equal to seven, which means that we need to sum up all these values and divide it to seven, and this will be our mean or the average. We can do that by us using the numpy library, so np.mean, and this the function that we're calculate the average, and then here within the parenthesis we need to specify what is that the variable or the array that we want to calculate the average. Let's calculate and let's see what that value is. So as you can see the mean of this array is equal to 90.43. Thing we are going to learn is how to calculate the median. So sometimes when we are dealing with a sampling distributions, the sampling distribution might be skewed, so uh it can be left skewed or right skewed. For those cases calculating the mean might not be the smartest thing to do, and therefore it would be better to calculate the median because the median is that the better representation of the overall uh data instead of the mean. So uh it's usually handy to calculate both of them, the mean and the median, and whenever the mean is different from the median it means that you are dealing with a skewed distribution. So the median is the middle value in a set of numbers when they are arranged in an ascending or descending order. If there is an even number of values then the median is the average of the two medial elements, and uh the median is also the second quantile from the statistical terminology, so it's the 50th percentile. Let's go ahead and learn how to calculate the median for this array. So median_ is equal to and np.dot, as you can see we are already getting the recommendation of the function called median, so very straightforward, and let's go ahead and print this median of array here we

Go. So, as we can see, the median of this area is equal to 20. One thing that we can also see is that the mean is very close to the median, indicating that we most likely are not dealing with a skewed distribution.

Next thing we are going to learn is how to calculate the mode. The mode is a value or the values in the set of provided numbers that occurs the most. So, in our case, in our data array, we can see that there is a single value that appears the most, which is two times, and that's the value 20, and this is the mode. So, in this, to calculate the mode of our array, we are going to use a different library. So, we are not going to use NumPy, and instead, we are going to use SciPy. The reason for this is because NumPy does not contain this corresponding functionality to obtain the mode. And the reason for that is is because mode is not a popular measure of central tendency. Usually, we only calculate the mean, the median, or the mean of the data, and we are good to go. But then, if you want to calculate the mode, it is still useful to know how to use this corresponding library to do that.

So, let's go ahead and for that import from SciPy the library called stats, and then from here we will use the stats.mode function to calculate the mode of our array. So, as you can see, the output is slightly different, but the idea is the same. So, as you can see, the mode of this array is this value, so 20, and it appears two times. So, we are getting two outputs: we are getting the actual value, so the value that appears the most, and also how many times it appears, so count, and it's equal to two because we got two of those 20s in our data.

The next thing we are going to learn is how to calculate the variance and the standard deviation. So, variance measures the spread or the dispersion of the data. It quantifies how far your numbers are from the mean. A higher variance indicates a greater variability in your data, and a lower variance indicates a smaller variability in your data. Standard deviation is highly related to the variance; the two are basically explaining the same thing, only standard deviation is the square root of the variance, and it is at the level of your numbers, which makes it more prepared whenever it comes to interpreting the results. It's the square root of the variance, as I mentioned, and it provides a standardized way to explain and understand the average distance between each data point and the mean. Therefore, it's almost always preferred to use standard deviation when you are explaining your data and how much variability is there in your data.

So, let's go ahead and use the NumPy library once again in order to calculate the variance and the standard deviation of this data. So, variance_ is equal to np.var, and then data, and the standard deviation, sd_, is equal to np.std data. Let's go ahead and print it. The variance, here we go. As you can see here, we are getting the variance of our data and the standard deviation of our array. All right, so this is about calculating various statistics given the array.

Another thing that would be worth knowing is how to get descriptive statistics whenever you are dealing with a data frame. So, let's go ahead and bring our data that we used previously. So, you might recall that we saw previously this data set describing the percentage of women that completed certain Bachelor studies. Let's also import the pandas data frame as we need that to load this data frame as well as to compute the descriptive statistics. So, this is how the data looks like, and most of the time whenever we are dealing with a pandas data frame, we want to get the nice descriptive statistics table that will describe our data. And we can simply do that by using this nice functionality in pandas. So, we need to specify the name of the data frame and then this .describe(), and this will go ahead and print for us the descriptive statistics of this data frame. Let's go ahead and print that. Here we go. So, this is how the descriptive statistics table looks like when it comes to the pandas data print. So, we are getting the count, so the number of observations that we got per variable. Now, we are getting the mean, which we just saw for an array. Now, we are getting the mean per column in our data frame. Then we have the std, which stands for the standard deviation. Then we got the min, which is the minimum value per column in our data frame. Then we have the 25th percentile, which is the lowest 25th percentile in your column, which is basically the first quartile from the statistical point of view. And then we got the 50th percentile, which is the median, what we also just saw when we were calculating the median of an array. The 50th percentile is also the second quartile from statistical terminology. Then we got the 75th percentile, which is the third quartile, and then we got the max, which is the maximum corresponding to that column. So, as you can see, this is a great way to summarize your data. So, you can look, for instance, the year, and you can see that the minimum of the year is 1970, the maximum is 2011, which means that you can say that you have data spanning from 1970 till 2011. Then you have the mean; in case of year, it's not really meaningful, but when you look at other columns, for instance, when we look at the architecture, which describes the percentage of women that completed the study of architecture across different years, you can see that across the years, so spanning from 1970 till 2011, there were on average 34% women who completed this study. So, in this way, you can then tell a story about your data, and you can also identify some problems in your data.

So, this is all for this demo where we learned how to calculate different statistics for an array as well as how to get the descriptive statistics table whenever we are dealing with a pandas data frame. So, this is all for this demo, and I will see you in the next one. If you're looking for machine learning, deep learning, data science, or AI resources, then check out the free resources section in Launchpad or our YouTube channel where you can find more content and you can dive into machine learning and AI.

Hi there, and welcome to another demo. In this demo, we are going to learn how we can combine, so we can merge different tables that we have in our database. Because most often we get our data not in one file but in multiple files, and sometimes we need to do some preprocessing, some filtering, and then at the end we need to join multiple tables together in order to end up with a single table such that we can use that in our analysis, in our data visualization, and in our machine learning training process. And we will train our model only on a single data set. For that, you need to know what are all the different joins out there, what are the possible combinations, what are the possibilities, and how you can do each of them in Python. So, in here, in this picture, you can see the most popular joins out there, and we have here a left join, we have an inner join, we have the right join, we have a left anti join, and a right anti join. So, let's, in here, we will be looking at two tables. We have a table X, which will contain certain features and then certain observations, and then we will have a table Y, which will contain a different table with different observations and different features. And our goal is to merge the two tables in different ways, and the way we can do that it really depends on what kind of join we want to do. So, let's go each of these joins one by one. We will go into the definition, and here we will use the idea, and we will assume that there is one key, so there is a key identifier present both in table X and table Y that we can use in order to find out whether a certain observation exists in X and in Y or not. Because whenever we are trying to merge two tables, we need at least one key identifier to use that to do the merge on that. Let's say we have a table containing the sales of a shop, and then we have a table containing the customers of the shop. At least we need to know the identifier of a shop in order to say that this customer in this shop has both this item and then this shop had the corresponding sales. In this way, the shop identifier will be the key identifier to be used in order to merge this sales data with the customer data based on shop. Using this idea, we will then look into these different joins.

First, we will look into the inner join. And by definition, an inner join returns only the matching rows. So, what you see here, the intersection between two tables based on the common column, which will be the key identifier. So, if we have certain observations that are in table X but then they are not in table Y, which means that we are talking about all these observations not highlighted, and if we have certain observations that are in table Y but not in table X, so those are all this on this part of the Y that is not highlighted, then all these observations will not be included in the final join table, and we will only end up with observations that are in this intersection, so they appear both in X and in Y. So, in terms of the example that I just mentioned, it means that we will be only keeping the data for the shops for which we have both the sales information and the customer purchase information. The result will include only the rows where the key values will be present in both table X and in table Y.

Let's now look into the left join. So, in here you can see the left join, and the definition of the left join is that a left join returns all the rows from the left table and the matching rows from the right table, in this case table Y, based on the common column, so the key identifier. So, if we have certain observations that we do not have information about in the table X, so those are all the observations in here, so in table Y that we do not have in table X, then those observations will not be included in the final output, and instead all the observations in X, independent whether they are in table Y or not, they will be included. So, basically, we are selecting all this part of the two join points. So, this will be our output. So, the result will include all rows from the left table, so table X, and the matching rows from the right tables, so table Y. And if there is no match, it includes no values for the columns of the right table. So, you will see some null values appearing in your end result because there will be cases for which you can see that observation contains information in here from table X but not from table Y.

Then another interesting join to look into is the right join. Right join is basically the exact opposite of the left join. So, right join returns all the rows from the right table, so table Y, and the matching rows from the left table, so table X, based on the common column. So, those are all the matching rows, and those are all the rows that are only in table Y, and this will be the output of our table. And then we have the left anti join and right anti join. Left anti join is very, it's kind of close to the left join, but it is basically the derivation from the left join and inner join. So, unlike in the left join where we were including both the observations that were matching and observations that were only present in X, in case of left anti join we are only including, so the output will be only the observations which are not matching and they are only in X, so this highlighted part. By definition, left anti join returns all the rows from the left table, so table X, that do not have a match in the right table based on a common column. So, it will be only this part. And in case of right anti join, it is the exact opposite of the left anti join. So, by definition, a right anti join returns all the rows from the right table, so table Y, that do not have a match in the left table X based on a common column. Let's say we are looking at this example of a shop for which we have the sales information and for which we have the customer purchase information. If we only want to have the customer information for the shops that do not appear in the sales data, so the sales data is the table X, right, and the customer purchase data per shop is the table Y, and we want to only have information for the shops and their customer purchase information for which we do not have the corresponding sales data, so in those cases you can then use the right anti join. It might not be reasonable for this specific example, but sometimes whenever we are looking at samples in our database, sometimes using the left anti join and right anti join might be handy. Therefore, it's worth to know how to do that in Python.

So, here we have two different small data frames that I created in Python, data1 and data2, and in the first data frame we have in total of seven observations, so you can see a, b, c, d, e, f, and in the second one we have c, d, e, f, g, and the corresponding indices are from 8 to 13, whereas for the data1 the indices are from 1 to 7, so that we do not have any intersecting indices because we are going to merge this data and we want to avoid cases when we have a different value corresponding to the same index. So, without further ado, let's first learn how to do an inner join between these two data frames. But before that, let's actually look into them. So, let's print the data1 and the data2, and as you can see here, we are getting our two data frames. So, you can see that this is the data1 and this is the data2. So, what we are going to do is to learn how to do an inner join, so how to do a merge where our way of joining is the inner join. So, for that, let's first call the corresponding data frame that we want to create, so merge_inner_join, and this will be equal, and here we are going to use the pandas merge function, so merge, and then whenever we are using this, we first need to specify the data frames that we want to merge. So, in our case, it's data1 and then data2, and then what we need to do is to use the argument on to specify the key, so the identifier that we are going to use in order to merge the two data frames. In this case, it is the key because it does make sense to join the data frames on a variable which we do have something in common in the other table. And in this case, as you can see, we have certain rows, so for instance the letter c, letter d, letter e, and f appear in both data1 and data2, and as we have only two variables, only one of them makes sense to be used as an identifier. So, in here we will use the key as our variable based on which we are going to do the join, and then we need to specify the exact way we want to do our join. So, the inner join has the corresponding parameter of inner. Whenever you are writing inner as a value for an argument how, then Python understands that you want to do an inner join between data frame one and data frame two. So, let's go ahead and actually print this, so merge_inner_join, and before printing the output, let's actually understand what is the expected output. So, there are common keys that appear both in data1 and data2, and those are the rows corresponding to the key c, key d, and then e and then f and the g. So, we expect that those keys that appear both in data1 and data2 and their correspondent values will then be in the inner join. So, let's go ahead and print it. Here we go. So, as you can see, as expected, we are getting the value one and value two corresponding to the key c, d, f, g in this output because those are the ones that appear both in table data1 and in table data2.

Now, the thing that we are going to learn is how to do a left join. So, as we learned, left join will provide all the values that appear in the first table, in the left table, as well as all the matching values that appear both in the left table and in the right table. So, what we need to do for that is basically to have the same, so left join, let's rename it to avoid confusion, and then here the only thing that we are going to do is to change this to left. Let's go ahead and print this, and this is our left join. So, as you can see, we are getting all the values, so all the keys that we got in data1, and we are also getting all the keys that appear in data2. So, as you can see in data2 we got the key c, d, e, f, g. So, those values which also appear in here, you can see that the value two corresponding to 8, 9, 10, 11, and 12 also appear in here, but then the other values for which we do not have the matching observations in the table data1, they do not appear in here. So, here you can see that we have NaNs corresponding to the key a and the b because those keys do not appear in the second table in the left table. So, this is the output of the left join. As you can see, the variables that only appear in the right table, they end up getting some NaN values, but all the fields and observations that were included in the first table, so data1, they do appear in full in here.

Now, let's go ahead and do our right join. So, as we just learned, right join is basically the opposite of the left join, which means that we can expect to get all the rows from the data2, so the right table, but then some of the rows that were in data1 but they were not in the data2, they will not appear. So, we will get only the matching ones and the ones that are present in the data2. So, let's also change the parameter in the argument how by right, and we should see some NaN values in the value1 because value1 is a variable appearing in data1. So, as expected, here we are getting all the matching keys that are both present in the table data1 and data2, as well as some NaN values corresponding to the value1 because this row does not appear in the data1 but it is present in data2, and then all the rows that are present in data2 are in here.

Finally, let's go and learn how to do left anti join and the right anti join. I will leave it to you, but the idea should be the same as it's in its concept, they are very similar. So, in case of left anti join, we need to go the extra mile and do some extra steps in order to get our left anti join. So, the first thing we are going to do is to do a left join, and then from the left join we want to remove the intersection part. So, if you bring back the diagram that we just saw, you might recall that in case of left anti join we are doing something very similar as the left join, only instead of also choosing the matching observations that appear...

Both in the left and right tables, in case of a left anti-join, we are only selecting observations that appear in the left table but not the matching observation. So we are removing the intersection part. Therefore, what we are first going to do is to do the uh left join. So uh let's call it merge and then uh left on T, it's equal to and then pd.merge, so the same function. And here we are going to specify the uh left table and then the right table. And then once again the uh on argument here, we are specifying the variable based on which we are doing the join and then how we are going to do the join, which is equal to left, as we want to do a left join. So first we want to do the um the left join and uh we also this time want to uh save the indicators uh that uh come as a result from the joins. So um by default the uh indicator is actually set to false, and we are going to change that and we are going to set it to true. And what this indicator does is that it shows whether the observation belongs to uh the left table only or it belongs uh to the matching part. So um it shows whether the observation was in the table data one only or uh whether the observation belongs both, move to the data one and data two. So it is in the intersection part. And then using this indicator and using this classification that will come from this indicator, we can then identify all the observations that were part of the intersection and we can remove them, and we will end up with all the observations that belonged only to the left table, so the data one. Exactly what is the point behind a left anti-join?

Let's go ahead and print the output of this table to uh show you what this uh indicator does. So as you can see, now we are getting pair observations beside of the left join of the two tables; we are also getting this underscore merge column which says whether the observation belongs only to the left part or it belongs to both parts. So it's in the matching, in the intersection zone. So as you can see, we got two observations: so observation with the key A and the B that belongs only to the table data one, and those are the observations that we want to keep, and we want to remove all these observations from the intersection zone, so from the uh keys C, D, E, F, and G, all right.

So next thing we are going to do is to define our left anti-join data. So let's do merge and then underscore left on the join. Sure, let's make this to left because it makes much more sense. And then we will use this data frame, and here we are going to apply the filtering that we learned previously. So we are going to say, look into the variable underscore merge that just came from the indicator and uh look into all the cases where this variable is equal to left_only, and this will then keep only the observations that appear in data one. Let's go ahead and print this, and you will see that we will end up only keeping the observations for which the uh for which the underscore merge will be equal to left_only. Here we go, and this is our left anti-join. Of course, we don't want to keep this underscore merge anymore because we have already used it and uh there is no purpose of keeping it. So what I will do is I will drop it. So let's do it actually in a new line. I will put the name of the data frame, I will do dot and then drop, and then here I will do underscore merge because this is the variable that I want to drop, and I need to specify the axis. So as it's a column, I want to remove a column; I need to specify that the axis should be equal to one. In Python, axis equal to zero means rows, and axis equal to one it means column. Underscore merge is a column; I'm mentioning that axis is equal to one. And let's actually go ahead and overwrite this data frame as I want to keep just one copy of the data frame, and this is the output. Here we go. So this is our left anti-join. As you can see, it is a bit more complicated than the left join or the right join, but I think it's worth to know how to do it because sometimes it can be very useful to implement this in practice. So I will leave the right anti-join to you, and this actually completes our demo for today, where we learn how to do left join, inner join, right join, and also left anti-join. This is all, and I will see you in the next demo.

Hi there, and welcome back to another demo. In this demo, we are going to learn how to perform data visualization with Matplotlib in Python. Data visualization is a very important technique for gaining insights from your data and to effectively communicate your findings to your audience, whether it's presenting to your stakeholders or whether it's putting in your case study or in your paper. It's really important to know how to plot those visuals by using Matplotlib because it's a simple way to go from your data analysis to your data visualization, which sometimes we call exploratory data analysis or EDA. And EDA sometimes can be the uh essential part of the case study to showcase your data, to find some correlations. It can be also a stepping stone towards the next step in your case study, whether it's causal analysis or modeling. It can help you to identify features that explain your dependent variable; it can help you to identify unimportant features, or it can help you to identify noise in your data. So therefore, it's really important for you to know how to make those visuals.

The first type of visualization we are going to learn is the line plots. Line plots are a great way to visualize trends or patterns in the data. They are a great way to visualize time series. So whatever you are dealing with a graph where the x-axis is in the form of time, and then the y-axis are the values that evolve over time, this can be for instance stock prices or the stock returns or the ROA of a company; you get the idea. So for that, what I have here is a set of X values and Y values that I created in the form of an array, and what we're going to do now is to plot this arrays. So therefore, I've imported here the Matplotlib pyplot Library. So pyplot is a directory in the Matplotlib library, and um as a uh way uh of shortening the name of this library is calling it PLT. So this is similar to the idea of using uh PD for the pandas and NP for the NumPy. Let's go ahead and use the library. So it is PLT.dot and then we have a plot, so plot, and then here we need to specify first the X values and then we need to specify the Y values, so the X_values and then Y_values. So the idea is that for each specific X we need to have the corresponding Y's. If your X array is different from the Y array, which means that for certain Xs you don't have the corresponding Y values or the other way around, so for certain Ys you don't have the X values, then you will get an error. So those two arrays should be the same, and for each X you need to have the corresponding Y values. Let's go ahead and run this, and you will quickly see that you're not getting any output, and the reason for that is because in Python whenever you are using the pyplot Library you need to uh specify a PLT.show such that uh your uh plot will actually be visualized. Here we go. So this is the plot that we are getting. As you can see, those are the X values starting from uh one and then ending with 10, and then the Y values starting with one and then ending with 20. So uh as you can see this plot is very basic; we don't have any extra information explaining it, but we want our plots to be self-explanatory. We don't want to add too much information, and we want our audience to look into the graph and the visuals and to understand what it's about. For that, you need to uh make use of extra functionalities in Matplotlib to add more information to your visuals. For instance, it would have been really handy to know what this x-axis represents or what this y-axis represents or to have a title on the top of the visual saying what this graph is about. All those can be done by using this uh PLT, and in here we can say PLT.dot and then xlabel, and in this way we can add a text to our x-axis saying for instance what is the variable that the x-axis represents, so x-axis placeholder. And then the same we can do with the y-axis; here we need to change y, here we need to change y. So let's say you're visualizing the time series of stock prices, and your x-axis represents the time and then the y-axis represent the stock prices. Well, in this case you can then uh put in x-axis placeholder that that um it is the date, and then the y-axis placeholder can be the uh stock price and then the name of the stock that you are looking into. And then finally we can also add a title, plt.title, and here then you can put the um uh title placeholder. For now I will put it as um text, but what you can do here is you can replace it with the uh title, let's say the stock prices of stock X from a time period X to Y. So in this way you can then add more information to your graph. Let's see how this looks like in the actual visualization. Here we go. So here you have the title, here you have the text under the x-axis, and here you have the text uh beside of the y-axis. So uh another thing you can do is to uh work with your plot and to uh make it uh nicer, and you can do that by for instance changing the way the line is represented. So you can go from a line to dots or dashes. It can be that you want to change the color of your plot; this one is really popular and I think it's really uh worth to know. So for instance, let's say your presentation is in the green uh color, so it's in the green palette, and you want your uh visual to match the color of your palette. What you can do for that is to use this argument color, and here you can for instance specify that it should be of green color. Here we go. So as you can see, this graph then changes the color; the plot is in the green, and this looks uh much more appealing compared to what we had before.

All right, so this is about line plots. The next thing we are going to learn is how to plot scatter plots. Scatter plots can be really useful when we are trying to visualize a relationship between two variables. So let's say we have um uh two features in our data, and we want to understand whether there is a certain relationship between the two, whether there is a correlation, because we want to know whether um we have a strong, perfect uh correlation between the two, which is uh something that we need to check as part of the linear regression model. We don't want to have two features being um multicollinear and perfectly multicollinear, and um we also want to check sometimes the relationship between the uh feature and between the dependent variable because we want them to be highly correlated. In all those cases we can use then the scatter plots as a way to identify this correlation and to see whether there is a pattern or there is no pattern, and a scatter plot uh can be uh in those exact cases super handy. So let let's say we have exactly the same data, and instead of the line plot we want to have the scatter plot. So for that what we need to do is to do PLT.dot and then scatter, so instead of plot we are using the function scatter, and then once again we are specifying the X values and the Y values. So let me go ahead and repeat the rest. Here we go. So as you can see, now we are getting the uh scatter plot, and uh this is basically uh the same plot that we saw before, only instead of lines now we are getting dots. So this is a scatter plot coming from our first case study where we were looking into the uh what factors make a playlist successful, and this is a vivid example of a scatter plot that helps us to understand whether there is a relationship between the number of albums in the playlist versus the average weekly active usage, and also you can see that there is a positive relationship, so in here. So in this way, using a scatter plot you can identify whether there is a relationship between a pair of variables, whether it's your two independent variables, whether it's your one independent variable and your dependent variable.

The next type of visualization we are going to learn is the bar chart, so how to plot bar charts in Python. And bar charts can be really useful for comparing different categorical uh values. So if you're dealing with categorical data and you have a certain variable that has categories and you have the corresponding values, then you can visualize it nicely by using the bar charts. And uh for this I have created here the sample data, very basic one, where we have the categories in the terms of the names of animals, so we have cat, we have dog, we have horse, and we have mouse, and then uh we have here categorical uh values. So uh this represents for instance the uh weight of an animal, so this is the weight of a cat, weight of a dog, weight of a horse, and then weight of a mouse, and we want to visualize this. Let's go ahead and use once again the library and the function PLT, so PLT and then bar, which stands for bar chart, and here we need to first specify the categories and then we need to specify the corresponding values, so cat and then values, which stand for categorical values, and once again we are need to add the label, we need to add the uh title, and we need to add PLT.show. So here we can replace this for instance by um animals, and then here we can add for instance a uh weight of an animal, and then as a title here we can say um weight of animal, and finally we want to do plt.show in order to show it. And I also would like to add a color to this visualization, so let's say I want the color to be forest green. By the way, if you're wondering what are all the possible um colors that can be used for this argument color, then uh what you can simply do is to use the ChatGPT and um try to search for uh different colors available in Matplotlib and Matplotlib, and you will get the name of all the possible colors that you can use, and then you can use a nice color palette that matches your presentation or your case study. Let's go ahead and run this. So as you can see, we are getting this uh bar chart, this nice visualization, and we can see that the cat has the uh corresponding weight, and then we see the dog and the horse and the mouse. We can see for instance that the mouse has the smallest weight and the horse has the largest weight, and in this way you can nicely visualize categorical data.

Let's now learn how to plot histograms. So histograms are useful for visualizing the distribution of numerical data. It can be that you want to visualize your population distribution or you want to visualize your sample data, and you want to compare for instance your sampling distribution to the population distribution to know how your sample is representative of your population, whether it's an unbiased and a true representation of the population, which means that your sampling distribution should be close to your population distribution. So for all those kind of tests you can definitely use the histograms. And by the way, this is a very uh common question as during data science interviews when you are asked to uh randomly sample from a normal distribution or from a uniform distribution and to plot this distribution in Python using histograms. Well, that's exactly what we are going to learn today. So you might recall from a demo where we learned how to randomly generate data and to um create a simulated version of data that we use this NumPy library to randomly sample from a normal distribution, and here we are sampling from a standard normal distribution with a mean of zero and standard deviation of one, and here we are sampling 100 observations. So uh now we are going to plot this distribution by using histograms, and here once again I'm using PLT.dot and then I'm specifying hist, and then here I need to specify the values that I want to plot, and then once again I need to specify the X label. So as you can see here we are getting our sample data, and here we have the frequency, so how often we are getting that corresponding value, and we can see that the uh distribution of the sample is symmetric around zero as expected because a normal distribution is symmetric and it's well-shaped and it's always uh symmetric around its mean, and we sample data from the standard normal distribution with a mean of zero, and as you can see the standard deviation, so how spread out the observations are from the mean, are uh very close to one. So um one thing that you will also notice is that as we increase this amount, so we go um we increase the sample size, so we make it for instance 2000, then this distribution should look more and more, this histogram should look more and more like the actual normal distribution. Here we go. Let's actually go ahead and do something a bit more advanced to show you how this looks like when comparing the histogram with a plot, and what I want to do here is to visualize the uh sampling distribution, so when uh we are generating our own sample randomly sampled from a normal distribution and we are comparing it to the actual population distribution by using this norm function that comes from the library called SciPy, and this one should be so a population distribution, and what we had before should be the sampling distribution. Let's go ahead and plot this. Here we go. So here in this plot, as you can see, now we are no longer uh plotting the frequencies, but we are plotting the actual probability corresponding to the sampling distribution, and we are also in the same graph visualizing the population distribution, so the actual normal distribution with the same parameters. And the reason why I wanted uh to show this because uh in this way we can compare your uh population distribution to the sampling distribution, and one thing that you will see as you change the uh number of observations that you sample is that the higher the number of observations, so the sample size, more will this histogram, so the um green bars, will look like to the actual population distribution, and this is also the entire idea behind what we call the Central Limit Theorem. So if you're wondering what this normal distribution is, what the sampling distribution, population distribution, what the Central Limit Theorem is, then head towards the fundamentals statistics section of this course to learn everything about this uh topic. So uh one thing that you will notice uh in this specific graph is that we saw uh that there were also legends added to the visualization. So uh those are really helpful when you are plotting not just one but two sets of datasets in the same plot, and you want to explain what is the difference between them. In our case we had this uh sampling distribution and we had a population distribution, and we had to specify that the uh bars correspond to the sampling distribution and the plot, so the line corresponds to the uh population distribution. Well, for that we use what we call legend, and as you can see the way that I'm doing it is by using plt.legend, and what I'm doing here is simply specifying the X values using the NumPy range uh function, which arranges values uh between the minimum and the maximum and then the corresponding incremental value, and then uh we have here the X values which uses the norm function coming from the SciPy to um to uh generate the corresponding probability distribution uh values, so the probabilities, and then here I'm specifying the...

Counts number of beings that I want to be visualized and also what should be ignored. So uh, here I'm using the histogram and then I'm specifying the first, the histogram that I want to plot, which is similar to what we had before. And uh, the only difference is that I'm specifying that we are dealing with the density, which means that it's going to plot the probabilities instead of frequencies. Then I'm specifying the color and then I'm specifying the label. So the way that the legend works is that I need to specify per plot what is the name of the plot by using this label argument. And then once you add all these labels and then you add here pl.Legend, it will then pair a plot, so pair type of visualization; it will then in the right corner or somewhere in the left corner, it will then specify the name of the plot.

If you want to see similar visualizations, how you can for instance randomly draw observations from a distribution—from binomial distribution, exponential distribution, geometric distribution, normal distribution, poison distribution, or student T distribution, or uniform distribution—and how you can visualize them using the histograms, then head towards this GitHub repository that I will post as part of the resources where you can see all those visualizations and their corresponding Python codes. And this is as part of this mathematics, statistics, or data science GitHub repository that I created. And this actually concludes this demo where we'll learn how to create different visualizations in Python using MatPlotLib. We learned how to do line plots, how to do Scatter Plots, how to do bar charts and histograms, and also how to combine, for instance, the plots and the histograms in a single visualization.

This video was sponsored by Lunarch. At Lunarch, we are all about making you ready for your dream job in Tech, making data science and AI accessible to everyone with data science, artificial intelligence, or engineering at Lunar Tech Academy. We have courses and boot camps to help you become a job-ready professional. We are here to help also businesses and schools and universities with top-notch training modernization with data science and AI corporate training, including the latest topics like generative AI. With Lunar Tech, learning is easy, fun, and super practical. We care about providing an end-to-end learning experience that is both practical and grounded in fundamental knowledge. Our community is all about supporting each other, making sure you get where you want to go. Ready to start your Tech Journey? Lunar Tech is where you begin. For students, for aspiring data science and AI professionals, visit Lunarch Academy section to explore our courses and boot camps and just in general our programs. Businesses in need for employee training, upscaling, or data science and AI solutions should head to the technology section on the Lunarch page. Enterprises looking for corporate training, curriculum modernization, and customized AI tools to enhance education, please visit the Lunarch Enterprises section at Lunarch.com for a free consultation and customized estimate. Join Lunarch and start building your future one data point at a time.

AB testing is an important topic for data scientists to know because it's a powerful method for evaluating changes or improvements to products or services. It allows us to make data-driven decisions by comparing the performance of two different versions of a product or service, usually referred to as treatment or control. For example, A/B testing allows data scientists to measure the effectiveness of changes to a product or a service, which is important as it enables data scientists to make data-driven decisions rather than relying on intuition or assumptions. Secondly, A/B testing helps data scientists to identify the most effective changes to your product or a service, which is really important because it allows us to optimize the performance of a product or service, which can then lead to increased customer satisfaction and sales. A/B testing helps us also to validate certain hypotheses about what changes will improve a product or service. This is important because it helps us to build a deeper understanding of the customers and the factors that influence customer behavior. Finally, A/B testing is a common practice in many industries such as e-commerce, digital marketing, website optimization, and many others. So data scientists who have knowledge and experience in A/B testing will be more valuable to these companies, no matter in which industry you want to enter as a data scientist and what kind of job you will be interviewed for. And even if you believe more technical data science is your cup of tea, be prepared to know at least a high-level understanding and the details behind this method. It will definitely help you to know about this topic when you are speaking with product owners, stakeholders, product scientists, and other people involved in the business.

Let's briefly discuss a perfect audience for this section of the course and prerequisites. There are no prerequisites for this section in terms of A/B testing concepts that you should know already, but knowing the basics of statistics, which you can find in the fundamentals of statistics section, is highly recommended. This section will be great if you have no prior A/B testing knowledge and you want to identify and learn the essential A/B testing concepts from scratch. So this will help you to prepare for your job interviews. It will also be a good refresher for anyone who does have A/B testing knowledge but who wants to refresh their memory or wants to fill in the gaps in their knowledge. In this lecture, we will start off the topic about A/B testing, where we will formally define what A/B testing is and we will look at a high-level overview of the A/B testing process step by step. By definition, A/B testing, or split testing, is originated from the statistical randomized control trials and is one of the most popular ways for businesses to test new UX features, new versions of a product or an algorithm to decide whether your business should launch that new UX feature or should productionize that new recommender system, create that new product, that new button, or that new algorithm. The idea behind A/B testing is that you should show the variated or the new version of the product to a sample of customers, often referred to as the experimental group, and the existing version of the product to another sample of customers, referred to as the control group. Then the difference in the product performance in the experimental versus control group is tracked to identify the effect of these new versions of the product on the performance of the product. So the goal is then to track the metric during the test period and find out whether there is a difference in the performance of the product and what type of difference it is. The motivation behind this test is to test new product variants that will improve the performance of the existing product and will make this product more successful and optimal, showing a positive treatment effect. What makes this testing great is that businesses are getting direct feedback from their actual users by presenting them the existing versus the variated product version, and in this way they can quickly test new ideas. In case an A/B test shows that the variated version is not effective, at least businesses can learn from this and can decide whether they need to improve it or need to look for other ideas.

Let us go through the steps included in the A/B testing process, which will give you a higher-level overview into the process. The first step in conducting A/B testing is stating the hypothesis of the A/B test. This is the process that includes coming up with business and statistical hypotheses that you would like to test with this test, including how you measured the success, which we will call the primary metric. The next step in A/B testing is to perform what we call power analysis and design the entire test, which includes making assumptions about the most important parameters of the test and calculate the minimum sample size required to claim statistical significance. The third step in A/B testing is to run the actual A/B test, which in a practical sense for the data scientist means making sure that the test runs smoothly and correctly; collaborate with engineers and product managers to ensure that all the requirements are satisfied. This also includes collecting the data of control and experimental groups, which will be used in the next step. The next step in A/B testing is choosing the right statistical test—whether it is a z-test, t-test, Chi-Square test, etc.—to test the hypothesis from step one by using the data collected from the previous step and to determine whether there is a statistically significant difference between the control versus experimental group. The fifth and final step in A/B testing is continuing to analyze the results and find out whether, besides statistical significance, there is also practical significance. In this step, we use the second step's power analysis, so the assumptions that we made about model parameters and the size, and the fourth step's results to determine whether there is practical significance besides statistical significance. This summarizes the A/B testing process at a higher level. In the next couple of lectures, we'll go through the steps one at a time, so buckle up and let's learn about A/B testing.

In this lecture, lecture number two, we will discuss the first step in the A/B testing process. So let's bring our diagram back. As you can recall from the previous lecture when we were discussing the entire process of A/B testing at a high level, we saw that the first step in conducting A/B testing is stating the hypothesis of the A/B test. This process includes coming up with a business and statistical hypothesis that you would like to test with this test, including how you measured the success test, which we call a primary metric. So what is the metric that we can use to say that the product that we are testing performs well? First, we need to state the business hypothesis for our A/B test from a business perspective. So formally, a business hypothesis describes what the two products are that are being compared and what is the desired impact or the difference for the business. So how to fix a potential issue in the product where a solution of these two problems will influence what we call a key performance indicator, or the KPI, of interest. A business hypothesis is usually set as a result of brainstorming and collaboration of relevant people on the product team and data science team. The idea behind this hypothesis is to decide how to fix a potential issue in the product where a solution of these problems will improve the target KPI. One example of a business hypothesis is that changing the color of the "Learn More" button, for instance, to green will increase the engagement of the webpage.

Next, we need to select what we call the primary metric for our A/B testing. There should be only one primary metric in your A/B test. Choosing this metric is one of the most important parts of an A/B test since this metric will be used to measure the performance of the product or feature for the experimental and control groups and then will be used to identify whether there is a difference, or what we call a statistically significant difference, between these two groups. By definition, a primary metric is a way to measure the performance of the product being tested in the A/B test for the experimental and control groups. It will be used to identify whether there is a statistically significant difference between these two groups. The choice of the success metric depends on the underlying hypothesis that is being tested with this A/B test. This is, if not the most, one of the most important parts of the A/B test because it determines how the test will be designed and also how well the proposed ideas perform. Choosing poor metrics might disqualify a large amount of work or might result in wrong conclusions. For instance, revenue is not always the end goal; therefore, in A/B testing, we need to tie up the primary metric to the direct and the higher-level goals of the product. The expectation is that if the product makes more money, then this suggests the content is great, but in achieving that goal, instead of improving the overall content of the material and writing, one can just optimize the conversion funnel.

One way to test the accuracy of the metric you have chosen as your primary metric for your A/B test could be to go back to the exact problem you want to solve. You can ask yourself the following question, what I tend to call the metric validity question: So if the chosen metric were to increase significantly while everything else stays constant, would we achieve our goal and would we address our business problem? Is it higher revenue? Is it higher customer engagement? Or is it high views that we are chasing in the business? So the choice of the metric will then answer this question. Though you need to have a single primary metric for your A/B test, you still need to keep an eye on the remaining metrics to make sure that all the metrics are showing a change and not only the target one. Having multiple metrics in your A/B test will lead to false positives since you will identify many significant differences while there is no effect, which is something you want to avoid. So it's always a good idea to pick just a single primary metric but to keep an eye and monitor all the remaining metrics. So if the answer to your metric validity question is higher revenue, which means that you are saying that the higher revenue is what you are chasing and better performance means higher revenue for your product, then you can use as your primary metric what we call a conversion rate. A conversion rate is a metric that is used to measure the effectiveness of a website, a product, or a marketing campaign. It is typically used to determine the percentage of visitors or customers who take a desired action, such as making a purchase, filling out a form, or signing up for a service. The formula for conversion rate is: Conversion rate = (Number of conversions / Number of total visitors) * 100%. For example, if a website has 1,000 visitors and 50 of them make a purchase, the conversion rate would be equal to 50 / 1,000 * 100%, which gives us 5%. This means that our conversion rate in this case is equal to 5%. A conversion rate is an important metric because it allows us and businesses to measure the effectiveness of their website, a product, or a marketing campaign. It can help businesses to identify areas for improvement, such as increasing the number of conversions or improving the user experience. A conversion rate can be used for different purposes. For example, if a company wants to measure the effectiveness of an online store, the conversion rate would be the percentage of visitors who make a purchase. And on the other hand, if a company wants to measure the effectiveness of a landing page, the conversion rate would be the percentage of visitors who fill out the form or sign up for a service.

So if the answer to the metric validity question is higher engagement, then you can use the click-through rate, or CTR, as your primary metric. This is, by the way, a common metric used in A/B testing whenever we are dealing with an e-commerce product, search engine, recommender system. Click-through rate, or CTR, is a metric that measures the effectiveness of a digital marketing campaign or the user engagement or some feature on your webpage or your website, and it's typically used to determine the percentage of users who click on a specific link or button or call to action (CTA) out of the total number of users who view it. The formula for the click-through rate can be represented as follows: CTR = (Number of clicks / Number of impressions) * 100%. Not to be confused with click-through probability, because there is a difference between the click-through rate and click-through probability. For example, if an online advertisement receives 1,000 impressions, which means that we are showing it to the customers 1,000 times, and there were 25 clicks, which means 25 out of all these impressions resulted in clicks, this means that the click-through rate for this specific example would be equal to 25 / 1,000 * 100%, which gives us 2.5%. This means that for this particular example, our click-through rate is equal to 2.5%. A click-through rate is an important metric because it allows businesses to measure the effectiveness of their digital marketing campaigns and the user engagement with their website or webpages. A high click-through rate indicates that a campaign or the webpage or feature is relevant and appealing to the target audience because they are clicking on it, while a low click-through rate indicates that the campaign or the webpage needs improvement. A click-through rate can be used to measure the performance of different digital marketing channels, such as paid search, display advertising, email marketing, and social media. It can also be used to measure the performance of different ad formats, such as text advertisements, banner advertisements, video advertisements, etc.

Next, and the final task in this first step in the process of A/B testing, is to state the statistical hypothesis based on the business hypothesis and the chosen primary metric. Next, and in the final task in this first step of the A/B testing process, we need to state the statistical hypothesis based on the business hypothesis we stated and the chosen primary metric. In the section on fundamentals of statistics of this course, in lecture number seven, we went into details about statistical hypothesis testing, including what a null hypothesis is and what an alternative hypothesis is, so do have a look to get all the insight about this topic. A/B testing should always be based on a hypothesis that needs to be tested. This hypothesis is usually set as a result of brainstorming and collaboration of relevant people on the product team and data science team. The idea behind this hypothesis is to decide how to fix a potential issue in a product where a solution of these problems will influence the key performance indicators, or the KPIs, of interest. It's also highly important to make prioritization out of a range of product problems and ideas to test while you want to prove that fixing this problem would result in the biggest impact for the product. We can put the hypothesis that is subject to rejection, so that we want to reject in the ideal world, under the null hypothesis, what we define by H0. Well, we can put the hypothesis subject to acceptance, or the desired hypothesis that we would like to have as a result of A/B testing, under the alternative hypothesis, defined by H1. For example, if the KPI of the product is to increase the customer engagement by changing the color of the "Read More" button from blue to green, then under the null hypothesis we can state that the click-through rate of the "Learn More" button with blue color is equal to the click-through rate of the green button. Under the alternative, we can then state that the click-through rate of the "Learn More" button with green color is larger than the click-through rate of the blue button. So I really want to reject this null hypothesis, and we want to accept the alternative hypothesis, which will mean that we can improve the click-through rate, so the engagement of our product, by simply changing the color of the button from blue to green. Once we have set up the business hypothesis, selected the primary metrics, and stated the statistical hypothesis, we are ready to proceed to the next stage in the A/B testing process.

In this lecture, we will discuss the next, second step in the A/B testing process, which is designing the A/B test, including the power analysis and calculating the minimum sample sizes for the control and experimental groups. Stay tuned as this is a very important part of the A/B testing process, commonly appearing during data science interviews. Some argue that A/B testing is an art, and others say that it's a business-adjusted common statistical test, but the borderline is that to properly design this experiment you need to be disciplined and intentional while keeping in mind that it's not really about testing but it's about learning. Following other steps you need to take to have a solid design for your A/B test. So let's bring the diagram back. So in this step, we need to perform the power analysis for our A/B test and calculate the minimum sample size in order to design our A/B test. A/B test design includes three steps: The first step is power analysis, which includes making assumptions about model parameters, including the power of the test, the significance level, etc. The second step is to use these parameters from power analysis to calculate the minimum sample size for the control and experimental groups. And then the final, third step is to decide on the test duration depending on several factors. So let's discuss each of these topics one by one. Power analysis for A/B testing includes these three specific steps: The first one is determining the power of the test. This is our first parameter. The power of the statistical test is the probability of correctly rejecting the null hypothesis. Power is the probability of making a correct decision, so to reject the null hypothesis when the null hypothesis is false. If you're wondering what is the power of the test, what are these different concepts that we just talked about, what is this null hypothesis, and what does it mean to reject the null hypothesis, then head towards the fundamental statistics section of this course, as we discuss this topic in detail as part of that section. The power is often defined by 1 - beta, which is equal to the probability of not…

Making a type two error, where type two error is the probability of not rejecting the null hypothesis while the null is actually false, is common practice. It's common practice to pick 80% as the power of the AB test, which means that we allow 20% of type two error. This means that we are fine with not detecting—so failing to reject—the null hypothesis 20% of the time. Which means that we are fine with not detecting a true treatment effect while there is an effect, which means that we are failing to reject the null hypothesis. However, the choice of the value of this parameter depends on the nature of the test and the business constraints.

Secondly, we need to determine a significance level for our AB test. The significance level, which is also the probability of type one error, is the likelihood of rejecting the null hypothesis—hence detecting a treatment effect—while the null is actually true and there is no statistically significant impact. This value, often defined by a Greek letter Alpha, is the probability of making a false discovery, often referred to as a false positive rate. Generally, we use the significance level of 5%, which indicates that we have a 5% risk of concluding that there exists a statistically significant difference between the experimental and control variant performances when there is no actual difference. So we are fine by having five out of 100 cases detecting a treatment effect while there is no effect. It also means that you have a significant result—a difference between the control and the experimental groups—within 95% confidence. Like in the case of the power of the test, the choice of the alpha is dependent on the nature of the test and the business constraints that you have. For instance, if running this AB test is related to high engineering cost, then the business might decide to pick a higher alpha such that it would be easier to detect a treatment effect. On the other hand, if the implementation cost of the proposed version in production are high, you can then pick a lower significance level since this proposed feature should really have a big impact to justify the high implementation cost; so it should be harder to reject the null hypothesis.

Finally, as the last step of power analysis, we need to determine a minimum detectable effect for the test. The last parameter as part of the power analysis we need to make assumptions about is what we call minimum detectable effect, or Delta. From the business point of view, what is the substantive—to the statistical significance—that the business wants to see as a minimum impact of the new version to find this variant investment worthy? The answer to this question is: what is the amount of change we aim to observe in a new version’s metric compared to the existing one to make recommendations to the business that this feature should be launched in production—that it’s investment worthy? An estimate of this parameter is what is known as a minimum detectable effect, often defined by a Greek letter Delta, which is also related to the practical significance of the test. So this MD, or the minimum detectable effect, is a proxy that relates to the smallest effect that would matter in practice for the business, and it’s usually set by stakeholders as this parameter is highly dependent on the business; there is no common level of it. Instead, this minimum detectable effect is basically the translation from statistical significance to practical significance, and here we want to see and we want to answer the question: what is this percentage increase in the performance of the product that we want to experiment with that will tell the business that this is good enough to invest in this new feature or in this new product? And this can be, for instance, 1% for one product; it can be 5% for another one, and it really depends on the business and what is the underlying KPI.

A popular reference to the parameters involved in the power analysis for AB testing is like this: so one minus beta for the power of the test, Alpha for the significance level, Delta for the minimum detectable effect. To make sure that our results are repeatable, robust, and can be generalized to the entire population, we need to avoid p-hacking to ensure real statistical significance and to avoid biased results. So we want to make sure that we collect enough amount of observations and we run the test for a minimum predetermined amount of time. Therefore, before running the test, we need to determine the sample size of the control and experimental groups, as well as—later on in this lecture—we will see also how long we need to run the test. So this is another important part of AB testing which needs to be done using the defined power of the test (which was the 1 minus beta), the significance level, and a minimum detectable effect—so all the parameters that we decided upon when conducting the power analysis. Calculation of the sample size depends on the underlying primary metric as well that you have chosen for tracking the progress of the control and experimental versions of the product. So we need to distinguish here two cases.

When discussing the primary metric, we saw that there are different ways that we can measure the performance of different types of products. If we are interested in engagement, then we are looking at a metric such as click-through rate, which is in the form of averages. So case one will be where the primary metric of AB testing is in the form of a binary variable; it can be, for instance, conversion or no conversion, click or no click. And in case two, where the primary metric of the test is in the form of proportions or averages, which means mean order amount or mean click-through rate. For today, we will be covering only one of these cases, but you can find more details on the second case in my blog, which I have posted also as part of the resources section. This blog post contains all the details that you need to know about AB testing, including the statistical test and their corresponding hypothesis, the descriptions of different primary metrics that go beyond what we have covered as part of this section, as well as many more details that you need to know about A/B testing. So let’s look at case two, where the primary metric of the test is in the form of proportions or averages.

So let’s say we want to test whether the average click-through rate of control is equal to the average click-through rate of the experimental group. Under H2 we have that the m control is equal to m experimental, and under H1 we have that the m control is not equal to m experimental. So here the mu control and mu experimental are simply the average of the primary metric for the control group and for the experimental group, respectively. So this is the formal hypothesis we want to test with our AB test, and we can assume that this m control is, for instance, the click-through rate of the control group, and the mu experimental is the click-through rate of the experimental group. So this is the formal statistical hypothesis we want to test with our AB test. If you haven’t done so, I would highly suggest you to head towards the fundamental statistic section of this course, where in lecture number seven and eight of the statistical part of this course I go in detail about statistical hypothesis testing, the means, averages, significance level, etc. This also holds for the theorem that the sample size calculation is based upon, called the Central Limit Theorem. So check out the last lecture about inferential statistics where I covered the central limit theorem, which we will also use in this section. And finally, also check lecture number five in that section where we cover the normal distribution—another thing that we will use as part of this section.

The central limit theorem states that given a sufficiently large sample size from an arbitrary distribution, the sample mean will be approximately normally distributed, regardless of the shape of the original population distribution. This means that the distribution of the sample means will be approximately normal if we take a large enough sample, even if the distribution of the original sample is not normal. So when we are dealing with a primary performance tracking metric that is in the form of an average, such as this one that we are covering today, which is a click-through rate, we intend to compare the means of the control and experimental groups. Then we can use the central limit theorem as stated that the mean sampling distribution of both control and experimental groups follow a normal distribution. Consequently, the sampling distribution of the difference of the means of these two groups also will be normally distributed. So this can be expressed like this, where we see that the mean of the control group and mean of the experimental group follow a normal distribution with mean mu control and mu experimental, respectively, and then with the variance of sigma control squared and sigma experimental squared, respectively. Though the derivation of this proof is out of the scope of this course, we can state that the difference between the means of the two groups—so x bar control minus x bar experimental—also follows a normal distribution with a mean mu control minus mu experimental and with a variance of sigma control squared divided by n control plus sigma experimental squared divided by n experimental—the sample size of the experimental group and the sample size of the control group.

Hence, the sample size needed to compare the means of the two normally distributed samples using a two-sided test, which pre-specifies significance level alpha, power level, and minimum detectable effect, can be calculated as follows. So here you can see the mathematical representation of the minimum sample size: so the n, which stands for the minimum sample size, is equal to—and in the denominator we have sigma control squared plus sigma experimental squared, multiplied by z 1-alpha/2 squared plus z 1-beta/2 squared, divided by delta squared. And here the alpha and the beta and the delta, we have made assumptions about as part of the power analysis, and the sigma control squared and sigma experimental squared are the estimates of the variance that we can come up with using the so-called A/A testing. I would say you do not necessarily need to know this derivation as there are many online calculators that will ask you for the alpha, the beta, and the delta values, as well as the sample estimates for the sigma squared control and experimental, and then these calculators will automatically calculate the minimum sample size for you. If you’re wondering what this A/A testing is and how we can come up with the sigma control squared and sigma experimental squared, as well as all the other values, then make sure to check out the blog that I posted before and that I mentioned before, as I explain in detail all these values, as well as check out the resource section where I’ve included many resources regarding this. But for now, just keep in mind that the z 1-alpha/2 and z 1-beta are just two constants and come from the normal distributed and standard normal distributed tables. I would say you do not necessarily need to know this derivation as there are many online calculators that will ask you for these alpha, beta, and delta values, as well as the sample estimates for the sigma squared control and sigma experimental control, and then will calculate automatically the sample size for you for the control and experimental group.

Effectively, one example of such a calculator is this AB test online calculator, but if you Google it, you will find many others that will ask you for the minimum detectable effect, for the statistical significance or the statistical power, and then it will automatically calculate for you the minimum sample size that you should have in order to have a statistical significance and in order to have a valid AB test. One thing to keep in mind is that you will notice that the statistical significance level is set to 95% in here, which is not what we have seen when we were discussing the alpha significance level. So sometimes these online calculators will confuse or will interchangeably use the significance level versus the confidence level, which are the opposite. The significance level is usually at the level of 5% or 1%; the confidence level is around 95%, which is basically 100% minus the alpha. Therefore, whenever you see this 95%, know that this means that your alpha should be 5%. So it’s really important to understand how to use this calculator, not to end up with the wrong minimum sample size, conduct an entire AB test, and then at the end realize that you have used the wrong significance level.

The final step is to calculate the test duration. This question needs to be answered before you run your experiment and not during the experiment. Sometimes people stop the test when they detect statistical significance, which is what we call p-hacking, and that’s absolutely not what you want to do. To determine the baseline duration time, a common approach is to use this formula: as you can see, duration is equal to n divided by the number of visitors per day, where n is your minimum sample size that we just calculated in the previous step, and the number of visitors per day is the average number of visitors that you expect to see as part of your experiment. For instance, if this formula results in 14 days or 14, this suggests that running the test for two weeks is a good idea. However, it’s highly important to take many business-specific aspects into account when choosing the time to run the test and for how long you need to run it, and simply using this formula is not enough. For example, if you want to run an experiment at the end of the month of December, with Christmas breaks, when higher than expected or lower than expected number of people are usually checking your webpage, then this external and uncertain event has an impact on the page usage. For some businesses, this means—for example, if you want to run an experiment at the end of the month of December with Christmas breaks, when higher than expected or in some cases lower than expected number of people are usually checking your webpage—so depending on the nature of your business or the product, then this external and uncertain event can have an impact on the page usage. For some businesses, this means that for some businesses a high increase in page usage can be the result, and for some a huge decrease in usability. In this case, running an AB test without taking into account this external factor would result in inaccurate results since the activity period would not be a true representation of common page usage, and we no longer have this randomness which is a crucial part of AB testing.

Beside this, when selecting a specific test duration, there are a few other things to be aware of. Firstly, too small a test duration might result in what we call novelty effects: users tend to react quickly and positively to all types of changes, independent of their nature. So it’s referred to as a novelty effect, and it varies over time and is considered illusionary. So it would be wrong to ascribe this effect to the experimental version itself and to expect that it will continue to persist after the novelty effect wears off. Hence, when picking a test duration, we need to make sure that we do not run the test for too short an amount of time period, otherwise we can have a novelty effect. Novelty effect can be a major threat to the external validity of an A/B test, so it’s important to avoid it as much as possible. Secondly, if the test duration is too large, then we can have what we call maturation effects. When planning an AB test, it’s usually useful to consider a longer test duration for allowing users to get used to a new feature or product. In this way, one will be able to observe the real treatment effect by giving more time to returning users to cool down from an initial positive reaction or a spike of interest due to a change that was introduced as part of a treatment. This should help to avoid the novelty effect and has better predictive value for the test outcome. However, the longer the test period, the larger is the likelihood of external effects impacting the reaction of the users and possibly contaminating the test results.

If you like this content, make sure to check all the other videos available on this channel, and don’t forget to subscribe, like, and comment to help the algorithm to make this content more accessible to everyone across the world. And if you want to get free resources, make sure to check the free resources section at LunarTech. And if you want to become a job-ready data scientist and you are looking for this accessible boot camp that will help you to become a job-ready data scientist, consider enrolling to the data science boot camp—the ultimate data science boot camp at LunarTech. You will learn all the theory, the fundamentals to become a job-ready data scientist; you will also implement the learned theory into real-world multiple data science projects. Beside this, after learning the theory and practicing it with real-world case studies, you will also prepare for your data science interviews. And if you want to stay up to date with the recent developments in tech—what are the headlines that you have missed in the last week, what are the open positions currently in the market across the globe, and what are the tech startups that are making waves in the tech world—short to subscribe to the Data Science and AI newsletter from LunarTech. This is what we call maturation effect, and therefore running the AB test for too short an amount of time or too long an amount of time is not recommended. As it’s a very involved topic, we can talk for hours about this part of the AB test, and also a topic that is asked a lot during data science and product scientist interviews. Therefore, I highly suggest you to check out this book about AB testing, which is a hands-on tutorial about everything you need to know about AB testing, as well as check out the interview preparation guide in this section that contains 30 most popular AB testing-related questions you can expect during your data science interviews.

Looking to elevate your data science or data analytics portfolio? Then you are in the right place. With this AB testing end-to-end case study, you can showcase your AB testing and coding skills in one place. I’m Dev Vasan, a data scientist and AI professional, and I’m the co-founder of LunarTech, where we are making data science and AI accessible to everyone—individuals, businesses, and institutions. In this case study, we are going to complete an end-to-end case study with AB testing, where we are going to test in a data-driven way whether it’s worth it to change one of our features in our UX design in the LunarTech landing page. This is a real-life data science case study that you can conduct and you can put it on your resume in order to showcase your experience in data-driven decision making, where you will showcase your statistical skills, experimentation skills with AB testing, and your coding skills in Python using libraries such as scikit-learn, but also pandas, NumPy, matplotlib, and Seaborn. We are going to start with the business objective of this case study, then we are going to translate the business objective into a data science problem, then we are going to start with the actual coding. We are going to load libraries; we are going to look into the data, visualize the data, the click data; we are going to look into the motivation behind choosing that specific primary metric, which is the click-through rate; then we are going to talk about the statistical hypothesis for our AB testing. I will also teach you step by step all the calculations, starting from the calculation of the pooled estimate from the click-through rate and then a computation of the pooled variance, the standard error, but also the motivation behind choosing the tests for this test that I will be using, such as the two-sample Z test, and then how you can calculate the test statistics, how you can calculate the p-value of the test statistics, and then use that with the statistical significance to test the statistical significance of your AB test. After this, we will also then compute the confidence interval, comment on the generalizability of the AB test, and then at the end we will also test for the practical significance of the AB test. Then we will conclude and we will wrap up and we will make a decision based on our data-driven approach using the AB test to check whether it’s worth it to change a feature in our UX design in the LunarTech landing page. So without further ado, let’s get started. So let’s now start our case study. In here I have, in the left-hand side, this version of our landing page, so which is our control version, so to say, the existing version, where you can see that here we have “Start Free Trial,” and here we got our button “Secure Free Trial.” In the right-hand side, we got this new experimental version that we would like to have, which is the “Android Now” button. As we saw in the introduction, what we are trying to understand is whether our customers click more on the new version—the experimental version—versus the existing version—the control version. So…

As of the day of, uh, loading this and, uh, conducting this case study, our landing page, uh, has a secure free trial. But what we wanted to test with our data is whether the, uh, "Enroll Now" button is more engaging, such that we can go from the secure free trial version to the "Enroll Now" version.

And, uh, here, um, for this specific case, and not only but also in general, as we know from A/B testing, is that whenever we got an existing algorithm or existing feature, existing button, then we are referring to this group that we will, um, where we will expose this existing version of the product; we are referring to this as a control group. So all the users to whom we will show the existing version of our landing page, we will refer to them as the, uh, control group participants. And then we have, in the right-hand side, our experimental version and our experimental users. So the users, our existing customers, that are selected to be taken part, um, in our experimental group and our experiment, they will be then, uh, exposed to this new version of our landing page, which contains this "Enroll Now" button.

So our end goal, in terms of the business, as we saw in the introduction, is to understand whether we should release the new button, which will end up being higher in engaging, which means that we will have higher CTR, or higher, uh, more, uh, clicks that will come from our user side, which, uh, automatically means better business because we want to have highly engaging users. If they are clicking on this button, it means that it interests them more compared to the control version. And, uh, if something on our landing page, in this case our call to action, is more interesting and highly engaging, it means that we are doing something right, and our users might, uh, either make use of our free products or, uh, purchase our products or, um, just stay engaged with us, to keep R Tech in mind. And whenever there is someone who, uh, is interested in data science or AI, um, solutions or products, then they can at least refer their friends, if they are just clicking to understand and to learn more about our products; that's also a possibility.

So from a business perspective, we therefore are using here as our primary metric, uh, our click-through rate, the CTR of this specific button, which in our control version is the "Secure Free Trial," and in our experimental version is the "Enroll Now." And what we want to understand is that whether this new button will end up having higher CTR or not, because higher CTR, from the technical perspective, will translate to higher engagement from the business perspective. So here we are making this translation from business versus technical.

Um, when it comes to A/B testing, we can have different sorts of primary metrics. We can have a click rate as a primary metric; we can have a conversion rate as a primary metric, or any other primary metric. What we want to have as our metric that will work as the single measure that will, will compare our control and experimental group to understand which version performs better is, first, to understand what this definition of "better" is, and how that translates back to the business. Because if the engagement is what we are referring to as "Better Business" for some reason, and I will explain to you in a bit why we think the engagement in this case is what we, what matters for us at L Tech, then it means that click-through rate can be used as a primary metric. This is just a universal metric that has been used across, um, different web applications, search engines, recommender systems, and many other digital products to understand whether the engagement of that specific algorithm, feature, web design—whether that is better or not. And in this case, in this specific case study, we are also going to use the CTR because we are interested in the engagement.

So at Larner Tech, we really care about the engagement, um, with our users, and we want our users to make use of our products, but, uh, ultimately to engage with us. Because if they engage with us, it means that our products are being seen, our, uh, landing page is being visited, and the user is actually interested to click on that button, and then the action point, and then to start either a free trial or to enroll to see what is going on, because all these are signs of interest coming from the user side. And in the control version, uh, our click-to-action is to secure a free trial, which directly, uh, lends the user to our free trial to our ultimate data science boot camp. But given that we are expanding, which means that we are now offering more courses, we are offering free products, and also we have, uh, enterprise clients, uh, we have businesses as clients who want data science and AI solutions and who want corporate training, therefore we want to go from this niche, uh, version of a landing page—so "Secure Free Trial"—to "Enroll Now," because we already have a lot of engagement in terms of the free trial; we want to make it more general. So that's the business perspective. And on the other hand, we also want to change, besides of changing this, um, main, um, call to action, we want to make it generalized, and at the same time we want to see whether this generalized version will end up leading us, um, a higher engagement, not only in terms of the other product but also for the free trial itself, because we always are looking for educating people and providing this free trial such that they can make use of our flagship product, which is the Ultimate Data Science Bootcamp.

So now when we understand why we care about the engagement here at Larner Tech, and we understand why we want to check whether this new button in our UX design will end up increasing the engagement or not, we can now make this translation back to the data science terms. Because we know now, from the business perspective, all we care is to understand whether this experimental version of the product is performing better or not. But then this means that we need to conduct an A/B test, and we need to understand whether the ideas that we got and the speculation that the "Enroll Now," more general button as a call to action, will be better than the "Secure Free Trial" version—whether this is actually true or not from the, uh, customer perspective. Because if we want to call ourselves a data-driven company, we cannot just base our conclusions and our decisions for our products, or for, in just general for our product roadmap, based on intuition or logic; we want this to be data-driven, which means that the customers are at the first place; we are customer-driven, and our customers need to tell us whether the new, um, button is better or not. And here we have conducted an, conducted an A/B test, and, um, here I won't be using the real data; I will be using the, uh, proxy data or simulated data that I, uh, generated myself, and, um, this one contains the similar structure and this, uh, the same, um, idea of the data that we got when we were conducting our A/B test and collecting this data.

And what is our business hypothesis? In our business hypothesis, we can say that we have at least a 10% increase in our click-through rate—so 10% higher engagement—when we have our "Enroll Now" versus the "Secure Free Trial" version of the product. So this is our business hypothesis, which means that our "Enroll Now" CTR, so click-through rate of the "Enroll Now" button, will result in at least 10% higher CTR than the "Secure Free Trial." So there exists, uh, 10%, at least 10% difference in terms of the engagement when we compare this new version of the product versus the old version of this new, uh, button. And when we translate this back to statistical hypothesis, we can say that under the null hypothesis, we are saying that there is no statistically significant difference between the, um, control p and then p experimental, which means the, um, um probability, click-through probability, click rate for control group versus experimental group. So under H0, the null hypothesis, we are stating what we ideally want to reject; we are saying there is no difference between the experimental and control group CTR. And under the alternative hypothesis, so the H1, we are saying, "No," uh, we do have a difference, which means that the, uh, control group's CTR is different from the experimental group CTR. And one key part here is to mention that they are not just different, but they are statistically significant, completely different.

So, uh, when it comes to starting the case study, first things first is to load the libraries. In this case study, we are going to use NumPy; we are going to use Pandas, as usual, for any sort of data analytics, data science, um, case studies; you always need those two. Usually Pandas will be needed for our data wrangling, to load the data, process the data, visualize it; NumPy will be used to, uh, work with different arrays and parts of the data. Then we are going to use a `scipy.stats` model, and from that we will import the `norm` function. Later on, um, we will see that we are using this in order to visualize this, um, uh, rejection region that we get from for our test, to understand whether we need to reject our null hypothesis or not. Then in this case study, we also want to visualize our results and visualize our data, for which we are going to need our visualization libraries from Python, which are Matplotlib and Seaborn.

Let's look into our data. So what we have in our data: we have four different columns, and of course this is filtered data that contains the information that we need, but in general you can have a larger database, you can have more sorts of, um, um, matrices, and, uh, different other matrices, but for conducting your A/B test, the pure A/B test, you actually need only the following information. So you need your user ID to understand, uh, what are the users you are dealing with; so it's user one, user two, user 10; it can be that you have other ways of referring to your users, and, uh, those can be, for instance, these long strings that we use to refer to our users. But given that our case is a simple one, our case study, we have just a user ID, and this user ID is just integers that go from one until, uh, until the end of our, uh, data. And here we got in total 20,000 users; therefore, this number, user ID, goes to, um, 20,000. And those 20,000, um, are all part of the user group, which means that they are all users, and they contain both the experimental and control users. Then we have our, uh, `click` variable, and this `click` variable, it's a binary variable, which can be, uh, either one or zero, where one refers that the user has clicked on the button, and zero means the user didn't click on the button. This is our primary metric for our A/B test. Then we have the group reference, which is this, um, string variable, and this string variable helps us to understand whether the user comes from the experimental group or from the control group. So this can, can contain only two different values, two strings, and it is "X," referring to the experimental, and "control," referring to the, uh, control group. If you can see here, we got just the three letters "X," referring to the experimental group, and then if we go in here, because we have first the experimental and then the control ones, you can see that here we got the, uh, control group. Then we have also some timestamp, which is, uh, not something relevant, so we'll be skipping that for now.

Now, um, given that this, uh, data that we have here, it's not the actual data, our data, but it's a synthetic one, but similar in terms of its structure, in terms of the, uh, nature of variables, and you can implement exactly the same steps when you have your data and you are getting it from your A/B test, and then you are conducting your A/B test, uh, case study. So in here, what we are going to make use of the most is our `click` variable and the `group` variable, because we want to find out per group what are the users that have clicked on the, uh, button, and to be more specific, we are looking for these averages. So we are not so much interested that that specific user from that specific group has clicked on the product or not; that's something that we can explore later. But for now, we are interested in the more high level: so what is this, uh, percentage? What is the click probability or click rate per group? And here we got groups of experimental and control, as it should be in any source of A/B test. So once we have conducted our A/B test, then I will also provide you more insights on what you can do with your data, especially with this user ID, to learn more about, uh, the idea behind these different decisions, or whether your A/B test is different per group. But the idea is that this A/B test that we are conducting, by following all the steps and by ensuring that the, uh, pitfalls are avoided, that we are making a decision that, um, represents the entire population. So we are using a sample that is large enough for us to make a decision for our product and for our business that will be generalized and will be a representation and representative when we apply this decision on our population. So let me close this part because we no longer need this, and let's go ahead and load this data.

So here I'm using the Pandas library and the common, uh, abbreviation of `pd`, and I'm saying `pd.read_csv`, and then I'm here referring to the name of the data that contains my click data. And here you can see that the data, that data is here: `ab_test_clicks_data.csv`. And I will be providing you this data because you won't have this in your own Google Colab; you will have the link to this Google Colab, and I'll provide you also the data such that you can put that data, you can download it first from my source, and then load it in here by using this specific button in here. And by doing that, you can then go to that specific folder where you downloaded the data, and then you will have also this, uh, corresponding CSV file in your folders. So once you have that, then you will, uh, smoothly run this code. And, uh, here I'm loading that data and putting under the name of `df_uncore_ab_test`; basically, the data frame containing my A/B test click data. What I want to do is to showcase you how the data looks like. So here you will see the header, given that here I haven't provided any argument; it just looks at the top five elements, so the top five rows. And here I got only the first five users from the experimental group; I see that some of them have clicked, some of them didn't click, and the corresponding user ID and the timestamp, uh, that they, um, done the click action.

Then, um, when we look at the `.describe()` function, you can see here that this gives us more general idea, uh, of, uh, what the data contains; it's not so much what the top five rows just look like, which is great in terms of to understand what kind of data you are dealing with, with what kind of variable you have. Now you can see more the, uh, total, uh, picture, so high-level picture, what kind of, um, data, what amount of data you got, so the descriptive statistics. So here we can see that in total we got 20,000 of users included in this data—so 20,000 observations, 20K rows—and then we have the mean for the user ID; of course, it, it's not relevant; the mean is 10,000, and, um, this is an interesting number. So we see that the average click, when we look at both user and control, the experimental and control groups, it is 40%, so 0.40, 52, so 40.52%; however, this is not what we are too much interested in; this is not to be confused with the click-through rate per group. What we are interested in is the click rate or the mean click-through, um, when it comes to the experimental group and the control group. So then we have our standard deviation; we see a high standard deviation, which is understandable, given that we have this, uh, large variation in our data; we got a control group and experimental group, and this variation shows that we have a huge difference in these different values, uh, when it comes to the click event. And then we have the mean and the maximum, which doesn't give us too much information because the click event, so the `click` variable, is a binary variable; it contains the zeros and ones; so naturally the minimum will be the value zero, because the click can take value zero and one, and the largest one is of course one, which means the maximum would be one. And then for the rest, the 25%, so the first quantile, the second quantile, the 50%, which is the median, or the third quantile, the 75th percentile, is not that much relevant. So when it comes to the descriptive statistics for this kind of data, especially if it's filtered, it's not super relevant. But if you would have a larger data, more matrices beside of `click`, which is your primary metric, but you also have measured some other metrics, which is recommendable, then you would see more, um, values which would be interesting to look at; so not only to look at the click rate but also to look at, for instance, the mean or maybe the median of conversion rate, or the, uh, mean, uh, amount of time, the average amount of time the user has spent on your landing page, or how much time did that user end up spending before making that decision of a click; those can be all very interesting metrics to look into from the product, uh, data science perspective, to understand the decision process and the channel and the funnel of these clicks. But for now, for our case study, what we are purely interested in is our primary metric, which is the click event.

So what we can also see in here is that we got, um, uh, in our group, um, when it comes to the control group, we got, uh, 989 users out of all, uh, control users that end up clicking, versus the experimental group, where we have 6,116 users who did click. So do not confuse this with the total amount of users per group; this amount is the, um, grouping of the, uh, data, so using the `.groupby()` and then `.sum()`. So we are grouping the data per group, and we want to see per group what is the sum of this variable, sum of the clicks. And given that the `click` is a binary variable, we know from basics of Python that we are basically accounting the number of click events, because if you got a binary variable containing zeros and ones, if you do the sum of the clicks, adding the zeros doesn't have any impact, which means that, um, you end up just summing up all the ones to each other, and then you end up getting the number or the total amount of, uh, cases when this `click` variable is equal to one. So in this case, when there is a click event. Therefore, we can see that per experimental group, um, we, we got 6,116, uh, users out of all the experimental users that end up clicking, and then out of control group, this amount is much lower, so we end up having, uh, only 989 users clicking.

Let's now go ahead and visualize this data. I want to showcase in a bar chart, using this `clicks`, what is the total number of clicks. So I want to show the distribution of the clicks when it comes to, um, the, uh, click event per group, and here I want to, uh, see next to each other the experimental group and control group. And as you can see here, here we are getting our bar charts, and the yellow corresponds to the "no," which means that there was no click, versus the, uh, black corresponds to the "yes," which means there was a click. So whenever you see this amount, it means that that amount, uh, corresponds to no click, no engagement from the user side, and this is per group. So this is what we are referring to as a click distribution in our, uh, data, in our experimental, uh, and control groups. And the way that I generated this bar chart is by first creating this, um, uh, list that will contain the colors that I want to assign to each of my groups, and I'm saying zero corresponds to the yellow and one corresponds to black, which means that if my variable contains an amount of zero, in this case my `click` is equal to zero, it means that I don't have a click, so it's a "no," and this I want to visualize by yellow; otherwise, I have a black.

Which means that um, the um, the one corresponds to the case when we have um, click, and in this case we will get a black, as you can see here. The uh, yes, which means a click is um, visualized by this black color.

And then what I'm doing is that I'm initializing this uh figure size by saying that I I want to have a figure size of 10 and 6. You you can also skip it, but I I think it's always great to put the size of a figure to ensure that you are getting the size like you want it to be. Such lat on you can also download or take a screenshot.

Then we have this uh, here I'm using, as you can see, a combination of the Matplotlib.pyplot Library as well as the uh, Cabo, because Cabo has much nicer colors. And here I'm saying uh, we are going to uh, make use of the countplot to um, create um, count plot because we are going to count, and we are going to showcase the counts per group. Uh, what is the number or the count of the clicks versus no clicks for a group called experimental, and what is the number of um, or the percentage of clicks versus no clicks when it comes to the group control?

And then here I'm specifying that the Hue should be on the click, which means that we are looking at the click variable, and we are going to use the data dfab_test, which means that we are going to look in this data from here. We are going to select this specific variable called click, and we are going to use this in order to group our data based on this group. So you can see that we are doing the grouping on the variable called group, so the argument is called x, x is equal to group. We're grouping our dfab_be has the on this group, and we are going to do the count in our count plot based on this variable click.

Basically what I'm saying here is that go and group our data dfap_test based on Group, which means that we will group based on experimental versus control. And then I'm saying go and count the click events count pair group, so pair experimental, pair control group. What is the number of times when we have a no, so we have a zero, and what is the number of times when we have a yes, or we have a one as a value for click variable?

And then as a palette I'm using my custom palette that I just created, which should be in the form of a list, as you can see in here. If I would have here also my third group or fourth group, then I of course need to extend this color palette because I need to have the same amount of colors as the number of groups paite tget variable. In this case, the Click has only two possible values, 0 and 1, which means that I'm only only specifying the two colors in my list.

So then we have the title of our plot, always nice to add, by, and then we have our labels, which means that I want to emphasize uh, as my X label. So here I want to have my group. You can see here is my group because I will either have group experiment or control; that's my variable on my X axis, and on my y axis of course I have the the count. So I'm counting the number of times I got uh, the uh, no click versus click event.

So here note that the um, y axis is in terms of discount. So here you can see it's uh 8,000, here saou 7,000 or 6,000, 5,000, which means that we are talking about the numbers and the counts rather than percentages, and this is important because um, another thing that I'm also doing is that I'm going the extra mile and I'm also adding beside of this counts on the top of each bar. I I want to visualize and clarify what are the corresponding percentages. It's always great to enhance your data visualization with some percentages. Percentages is easier for the uh person who follows your presentation to understand. For instance, if you got an experimental group and the the users is here 6,000 and um 4,000, they they might not quickly understand that you got for instance in total 10,000 of users, and then 6,000 has then uh clicked and then 4,000 didn't click.

So um, then the idea is that by adding this percentages we can then see that 61.2% has clicked in this experimental group and 38.8% has not clicked. Of course this a simulated data; I specifically pick the extreme in such way that we can clearly see this difference in the clickr rates, but um, in the reality you can have a clickthrough rate of 10% up to 14%, which is usually a good number. If you have a click through rate of 40% is great, but it's really depend on underlying user base, what kind of product you got, how large is your user base, because if you have very large user base then 10% can be a good clickr rate versus if you have a very small user base maybe uh 61% is considered uh good or average. So uh, in here we have just a simulated data of course, and I have added these percentages uh by using the following code. So I won't go too much into detailing here um uh feel free to check and see uh, and if something doesn't make sense go back to our python for data science course that contains lot of information on the basics in Python, but but here just quickly what I'm doing is that I am uh calculating the percentages and I'm annotating the bars. So I want to know what are these percentages, which means that per group I want to take the total amount of clicks. I want to understand what is number of Click event when the click variable is equal to one so and what are the number of cases when there was no click from the user side, which is what are the number of cases when the click variable is equal to zero, and then I'm counting those amounts and then using the total amount to calculate the percentage.

For instance, in this specific case I'm filtering the data for experimental group, I'm looking at the total number of users for this group, which is 10K, and then I'm counting the number of times when out of this 10,000 users the amount of users that end up clicking on that button, which is the click is equal to one case, and then I'm taking that number dividing it to the total number of users for this experimental group, multiplying by 100 in order to get that in percentages, and this is the calculation that you can see here. One thing that is important here is that here I'm using this um uh percentage um, so for the current bar I'm saying U as a way to identify whether we are dealing with experimental or control group is by getting by looking into this uh p, and uh this p in here is the basically the patches. So in this case I'm basically saying if I'm dealing with the experimental group then go ahead and calculate what is this uh total amount of observations and then take what is the uh number of clicks and then divide the two numbers uh C multiply this with 100, and this will then give as the percentage. And then I'm doing this for each of those groups, so I'm doing it for this group, I'm doing for this group and for this one and for this one. So I got two groups, but then within each group I got clicks and no clicks, and I'm calculating this four different percentages, and then I'm adding these percentages on the top of those bars. So I not only want to have numbers represented in my visualizations, but I also want to add this corresponding percentages at the top just for visualization purposes. I wanted to put this out there because this can help your uh data visualization toolkit, and it also will um make your audience from your presentations be more thankful to you when you are telling the story of your data.

So uh, this is about the data that we have. We see that uh 38.8% of our experimental group users have not clicked on the button versus the 61.2% have clicked on the button based on the simulated data. And then uh, in the control group we have a quite the opposite situation; we got the majority of the users, 80.1%, not clicking on the button versus the remaining 19.9% % have actually clicked on that button. So we got a huge difference, a dissonance when it comes to the experimental group and control group. This kind of gives us an indication, hey something is going on here; we kind of uh have already um higher level intuition what the remaining analis will look like um, which is that there most likely will be a difference in their CTRs when it comes to the uh the um uh control versus experimental group and the corresponding buttons, but uh hey let's continue. That's the entire goal behind A/B testing is to ensure that our intuition, our conclusions are all based on the data rather than on our intuition.

So what are the parameters that I'm using here for conducting our A/B test? When I was designing this A/B test uh, the first step was to of course do all these different translations that we learn as part of our A/B test course um, conducting it properly, which means coming up with this three different parameters when doing our power analysis, and usually this should be done when you are collaborating also with your colleagues and uh with your product managers or your product people, domain experts, because they have um a lot of information on what it means to have um threshold that you need to pass in order to say that for instance this new version of your feature is different and is uh considerably uh different from the existing one. And here um, in order to for us to understand this uh and make these conclusions we need to come up with the three different parameters that can help us to properly conduct an A/B test as we learned when we were looking into designing a proper A/B test.

So first we we have our significance level, the significance level or the alpha, the Greek letter that we are using to refer to the significance level, which is also the probability of the type one error, and that amount we have chosen following the industry standard, which is 5%, given that we didn't have any uh previous information or specific reason to choose a different significance level, so lower or higher, we decided to go with the industry standard, which is the 5%. This means that we want to have um, we want to compare our P value of our uh statistical test to this 5%, and then say whether we have a statistically significant difference between the control and experimental group based on this 5% significance level. And let's refresh our memory on this Alpha. This Alpha uh or significance level is also the probability of type one error, so this is the amount of error that we are comfortable making when we um reject the null hypothesis, well the null hypothesis is actually uh true, which means that we are detecting a difference between the experimental and control version while there is no difference, and we are making that mistake. And here we are saying that we are fine and we are comfortable with making this mistake at a maximum of 5%, but higher than that it's not allowed; we are not comfortable making uh error um higher than 5%.

Then the next variable uh, in this case the b or beta, the probability of type two error, which is the opposite of the type 1 error, which is a false negative rate or the amount of time the um um proportion of time when we end up failing to reject the null hypothesis while null hypothesis is false and it should have been rejected. Then the 1 minus beta is actually power of the test, so what is the amount of we are correctly rejecting our null hypothesis and correctly stating that there is indeed a statistically significant difference between our experimental group and our control group. So we have chosen for this the uh industry standard as well, which is the 80%, but given that for your results analysis in this case for conducting this case study that part of the power analysis is not relevant; we use that when calculating our minimum sample size, but we don't need that when conducting our results analysis. Therefore, I'm not initializing that as part of this code. So here I'm only providing to my program the values for my significance level, which is 0.05 or this is the same as 5%, and then the Delta, which is the third parameter, and this Delta is our minimum detectable effect.

So this a Greek letter Delta, which is the minimum detectable effect, helps us to understand whether beside of having this statistically significant difference, whether this difference is large enough for us to say that we are comfortable making that business decision to launch this new button. So it can be that when we are conducting an A/B test we are finding out that the experimental group has indeed higher engagement than the uh control group, and we are uh getting a small P or at least smaller than the alpha, and we are seeing that P is more than the alpha level, which means that we can reject the null hypothesis and we can say that the uh CTR or the clickr rate of the experimental group is statistically significantly different from the control group at 5% significance level. But we know from the theory of ab test that only that is not enough; only statistical significance is not enough for the business to make that important decision to launch an algorithm or to launch a feature; in this case to change our landing page, the button from the start free trial to the enroll now. Which means that we want to have enough users and we want to have enough difference, large difference in our clickr rat or enough users saying that we are more happy with this uh new version of the landing page for us to go and change our feature. And what is the definition of enough? What is the difference in the click through rate that we need to detect after we have detected the statistical significance in order for us to say that we also have a practical significance? So practically we are also comfortable making that business decision and then launching this new feature and changing our landing page button. And that is exactly what we have under our Delta, this minimum detectable effect. In this case we have chosen for Delta of 10%, so you can see here 0.5, this is 10%, this means that our Delta or the MDE, the Min detectable effect, is 10%. This means that we are saying not only we should have a statistically significant difference between the experimental group and control group, but also we need to have this difference to be at least 10%, which means that we need to have detected that the experimental version of the landing page results in at least 10% higher click through rate compared to the control version for us to go ahead and to launch this new version and deploy this new uh UX uh feature.

So this is really important because many people go and check for statistical significance, so they do their Alpha and then check uh whether the P values for the alpha and then say hey we have a statistically significant difference and then they are done with that, but that that's not correct. After you have conducted your uh statistical significant analysis and you have detected that your uh experimental version has a statistically significant different um CTR than the control version at your Alpha significance level, the next thing you need to do is to ensure that you also have a practical significance beside of the statistical significance, and this practical significance you can detect and you can check when you use your MDE or your Delta and you compare it to your confidence interval that you have calculated, something that we have also learned as part of the theory of conducting a proper A/B test. But once we come to that point, so after we check for our statistical significance, I will also explain how exactly uh we will need to do this check, and at the same time we will also be refreshing our theory on the Practical significance.

So let's now go ahead and calculate the total number of clicks per group by summing up these clicks, and I also want to calculate and group by this amounts just to showcase how you can do that on your own. So here what I'm doing is that I'm taking my A/B test data, I'm grouping by by group. Group is the uh variable that contains the reference when we are dealing with experimental group or control group, and as you know from our uh python series and demos, python for data science course, that uh whenever we want to group that data, a pandas data frame first we need to say pandas data frame name.groupby within parenthesis the variable that we are using to do the grouping, which is in this case group, and then within square braces I want to emphasize and put the name of a variable that I want to um apply operations on. So I want to group my data on the group variable and I want to count the number of times I have a click in my control group and in my experimental group. This will be my x_control and x_experimental variables, so x_control will then compute contain information about number of Clicks in my control group, and then x_experimental will contain the number of Clicks in my experimental group. And given that um I want to refer to the name of that uh Group after I did my group grouping, so I am getting this kind of this shape of data frame, of course I then need to uh use my .to_dict function in order to properly call that amount, so to understand what is this amount corresponding to this index and what is this amount corresponding to this index, and given that my index is in strings I'm then using here my .to_dict function, something that we also learned as part of our python for data science course. So here is basically the printing, just writing nicely what are the results, which means that we are counting that the let me count again that the uh number of uh clicks for my control group is 1,989. So you can see that it is want to double check and see what we got, yes, so we got the same number, so we are dealing with the same data set just to make sure. And here the number of clicks for experimental group is equal to 6K and 116, so 6,116 clicks.

So then we are calculating the uh pulled estimates for the clicks per group. Let me quickly fix typo. So calculating the uh pulled estimate for the clicks per group, which means the P estimate for the experimental group and for the uh control group. So let me quickly add here how I can calculate the uh total cases when we got uh experimental group users, so what is the number of users in the experimental group and what is the number of users in the uh control group. So here what I want to do is that I want to say that the group, the df_test group should be equal to experimental, and this of course should be my filter, and I want to count this, and let me quickly copy this. I saw that it's already under the control, so here I'm changing to the control, and this will need to give me the number of users in each of these groups too, so number of users in control and number of clicks per group. There we go. So now when we have done this what we are ready to do is to go ahead and calculate the P estimate for clicks per group, which means pair control group and pair experimental group. For that what we need to do is to take the number of clicks of the control group divide to the number of all users for control group, as you can see in here x_control / to n_control, and we are referring to this variable as p_control_hat because we know that the estimate of this click probability um is always with a hat; it's just the way that we reference it in um statistics and in A/B testing. So this is the estimate, something that we are estimating, therefore we are saying hat, and then we have the same for experimental group, which means that the estimate of the experimental group uh click probability is equal to x_x and then divided to n_x. Then um, in order to calculate the uh pulled estimate or uh pulled click probability, which means the value that will describe of the uh control group and experimental group, we need to follow this formula, which means that we are taking the x_control, we are adding to that x_control the x_experimental; this is our nominator of our uh value, and then we are dividing this to

The uh, sum of the sizes of each of those groups, which is n control and N experimental, so this is the common formula of the pulled estimate. Uh, when it comes to this type of experimentation, when you are dealing with um, primary uh, metric that is in the form of zeros and ones. And if you want to refresh your memory on this type of formulas, then make sure sure to also check our AB testing course, because in there we go in detail in this uh, specific lesson of the uh, AB test result results analysis. We are looking into this uh, all these formulas on how we can calculate the pulled estimate of this uh, click probability. So click probability, but then we are calling it P, click probability T. And then what we got is this volum, so that amount is then 0 and 40. This number should look familiar because this is then the mean that we saw when we were looking at the um, uh, descriptive statistics table. If you can recall this table, let me see this number. So now basically we are then calculating this manually because we need a variable that will hold this uh, volume. So it is simply summing up all the clicks for control group and experimental group to get the total number of clicks, and we are dividing it to the total number of users, so n Control Plus n experiment.

So now when we have this, we are ready to also calculate what we are referring as a pulled variance, also something that we have learned as part of the theory for AB testing. So the pulled variance is equal to the pulled estimate of the clicks, so P ped has something that we just calculated multiplied by one minus P head. So the uh, click event, the estimate of the click probability multiplied by the estimate of no click, and we know already this idea of berol distribution that the variable that uh, describes this process of clicks and no clicks follows kind of this idea of bero distribution when we have a click and no click. So we have probability of click and then we have probability of no click, which is the one minus that click probability. So that's the idea or the part of the formula that we are following as kind of an intuition, and then this multiplied by 1 / to n control+ 1 / to n experimental. So here I'm purely following the formula for the pulled variance. If you want more details and explanations and sure to check the cores responding Theory lecture, because we are going into details of each of those formulas and understanding why we calculate this um, P variance and P estimates uh, in this specific way and using these specific formulas. So here by just follow following the uh, formula I'm getting that the uh, pull uh, variance is this amount. So this is in nutshell how I calculated my uh, pulled click probability and a pulled variance of that click event, and we are going to need that in the next very important step, which is calculating the standard error and calculating the test statistics. Because in this case what we are doing is that we are dealing with a case when the primary metric is in the form of zeros and one, so we let's Now quickly talk about the uh, choice of a statistical test be uh, before conducting the actual calculation of standard eror and the test statistics.

So here I went for the two samples at test, and let me explain you why and what is the motivation. Because as we learned as part of the theory um, whenever we have a primary metric that is in the form of an averages like we have now because we are using the P control head and P experimental head head, so we have a primary metric that is the uh, click true rate which is the average clicks per group. So we have calculated the average click per experimental group and per control group, then the primary metric the form of it already dictates given that it's in averages that we need to look at uh, either parametric test corresponding to this averages or non-parametric test corresponding to the um, averages. In this case I went for the parametric case because uh, it has better properties if I have this information about the distribution of my data, and why do I have this information and then this also dictates the uh, choice of my um, statistical test. Well I have a size of my sample which is over 100 and actually over 30, that's the threshold that we tend to use in statistics and in a testing in order to say whether we have a large size or large data or not. If our sample is not large so it contains less than 30 users per group which happens as well, then we say that we need to go for um, statistical test uh, that will be specific for this kind of cases because we can no longer make use of the uh, statistical theorems like the central limit theorem which helps us to um, uh, to take the uh, to the inference, so to make use of the inferential statistics and make conclus I regarding the distribution of our population just having the sample. And what do I mean by that? So if my sample is larger than 30 like in this specific case I got 10,000 users per group, so it is definitely larger than 30 uh, users, then in that case I can say that by making use of the central limit serem I can say that my sampling distribution is normally distributed, and this is simply making use of the central limit theorem, something that we have also learned when we were looking into this concept of inferential statistics as part of the fundamental statistics course uh, course um, in lunar Tech. So this is a powerful theorem that we use in AB testing in order to make our life easier because when we have a sample that is larger than 30 for each of these groups, then we can say that even if we don't know the actual distribution or the name of the distribution that our uh, sample follows when it comes to the click um, event, so the random variable that describes this number of clicks or the average click through rate, what is that um, distribution exactly, but given that we have that this size is large enough it's large than 30 users we can say that by making use of the central limit theorem we can say that the uh, the uh, sample distribution follows a normal distribution if given that the sample size is large enough, and this helps us to say that well in that case it doesn't matter whether we make use of the two sample Z test or two sample T Test, we can make use of either of these test in order to conduct our analysis, and we had this specific template to make this Choice easier uh, in our AB test course at the loer tech where we were making all this decisions and saying if the SLE size is this we need to do this if the SLE size is this we need to do this, and in this specific case following that exact structured and organized approach I ended up seeing that my sample size is large so it's larger than 30 so I can then make use of the central limit theorem. I then know what is the random uh, what my random variable describing this click through rate um, follows the kind of distribution in this case a normal distribution, and then this means that whether I use a t test or Z test doesn't really matter I'm going to end up with the same conclusions therefore I will just go with a two s set test simply because um, it is just easier for me to do. For example you can also go with the two sample T Test, and you can even change this case study and tweak it and then make it your and put it on your resume in that way by making it more unique and that will be totally fine because you will see that you are going to end up with exactly the same conclusions as we do in this specific case study because if you have a large enough sample it won't matter whether you have a two sample Z test as your parametric test or the two sample T Test and um, if you want to know why why this matters and all the different details statistical insights make sure to check the actual uh, course is dedicated to AB testing because there will we cover this all and you will then become a master in the field of AB testing.

Now we know this uh, decisions and the motivation behind choosing the uh, two samples that test, let's now go ahead and do the actual calculations. So here we have a standard error which we calculate by taking the pulled variance and taking the square root of it, and this is again using the idea of this formulas that we learned as part of the ab test. So we are using this P variance taking the square root of this which gives us the standard error, and the standard error as you can see in here is then equal to 0.0069 29499 this amount. Then we calculate our test statistic for our two sample at test. So the test statistic is equal to P control head minus P experimental heads divided to standard error. So here uh, you can now see the motivation behind not only Computing the P pulled head but really also the p uh, control head and P experimental head, and then I take the P control head and subtract the P experimental head and I divide it to the standard error to compute my test statistics. Once I did this as you can see this is this amount, so test statistics for our two sample that test is this amount minus 5956, rounded it. Then um, we can also compute the critical value of our Z test which is uh, by using this Norm function that we uh, loaded in here from the C high and this will help us to understand what is this value from our normal distribution table, the standard normal distribution table uh, where by making use of this table we identify what is this critical value that we need to have to uh, create our rejection regions and to say whether we can uh, reject our n hypothesis or not. So to conduct our test we need to have a critical volum for uh, to which we will compare our test statistics, and this critical value will be based simply on the standard normal distribution. So this is this norm.ppf and then uh, probability um, uh, function basically uh, the the probability function that comes from the normal distribution standard normal distribution, and as you can see this corresponds specifically to this percent Point function which is the inverse of the cumulative distribution function. So this based on the alpha / 2, so 1 minus Alpha / 2 is the argument that we need to put for our percent Point uh, probability function and why divided to two because we have a two sample test so because we have a two-sided two sample test sorry. So if you want to understand this difference between uh, two sample um, test two sided Test please check out the uh, fundamentals to statistics course at ler Tech because we cover this uh, Topic in detail and it's a very involved topic it contains many complexities U from statistical point of view. So I won't be spending in this case T too much time on that. Here I'm assuming that you know this formula already, but if you don't and if you quickly need to do your case study NAB testing feel free just just to copy this line which basically is a value that we need based on the corresponding chosen statistical significance level that we need to compute to compare our test statistics. So our test statistics is this value and the value that we need to compare it to is the Z critical volum.

So so we can see that this critical value is then equal to 1.96. This is actually a very common value that we know even without looking at a standard normal table. When you make use of this test enough often then you know that the uh, critical value corresponding to two-sided test when it comes to normal table is equal to uh, 1.96. This is just a value that we know and in here by even with without calculating the next step which is a P value we can even say already what is the decision we need to make in terms of statistical significance because we know that one way we can test our hypothesis statistical hypothesis is by Computing the test statistics and checking with the test statistics the absolute value of it is larger than the critical value and we see that the test statistics is equal to minus 5956, the absolute value of that is 59.5 6 and that value is much larger than our critical value which is equal to 1.96. This already gives us an idea that we can reject our null hypothesis at 5% statistical significance level, but I want you H go on to the next step actually because that's um more structured more organized way to doing and conducting experimentations as in the industry we tend to make use of the P values instead of making use of this econometrical approach and statistical approach of um, testing the statistical test. So once we have calculated our test statistics the next thing we need to do is to calculate our P value and then use that P value compare to the significance level Alpha and then make a decision whether we need to reject our n hypothesis and say that we have a statistical significance or we cannot reject our n hypothesis and and then we need to say that we don't have a statistical significance so we don't have enough evidence to reject anal hypothesis. So the idea here is that we need to make use of our uh, normal function and specifically the norm.SF so making use of exactly the same Library the norm from CI dos and then this time we're using the survival function which is the one minus the cumulative distribution function of normal distribution. This comes again from statistics and then using the absolute value of our test statistics multiplying it by two given that we have a two-sided test I'm calculating my P value. This is simply by making use of the same formula that we saw when we were uh, studying the ab test from a technical point of view because we learned that the P value is then the probability that Z will be smaller than equal the minus test statistics or that the test statistic is smaller than equal to Z. So uh, we basically want to calculate what is this probability the P value which is equal to the probability that our test statistics will be smaller than the critical value or our negative of the test statistics will be larger than equal of the critical value and we want to know this probability because what this probability represents is that what is the chance that we will get a large test statistics well this is due to a random chance and not because we have a uh, actual statistical difference between the clickr rate of the experimental group versus control group. So this is the idea behind P value. So what is this chance that we are uh, mistaking this random mistake this random observation that we got a large test statistic and saying that there is a statistical significant well there is no such thing and we are purely getting this large test statistics um, because of the random chance. If the probability of getting a large test statistics by random chance is small so if this P value is small then we can say that we have a statistical significance that's the idea behind it and this P value when we calculate uh, we are storing it in this variable called pcore value and then the next thing what I'm doing is that I'm writing this function quote is statistically significant which takes argument as P value in Alpha. So I just need the P value that I just calculated for my test Set uh, test uh, statistics and then I want the statistical significance level that I want to use for my test and then this is the value that comes from my power analysis as I mentioned before that's the 5% this P value I'm calculating for my test statistics so in here and then I'm taking the two and I want to compare them so I want to assess whether I have a statistical significance by comparing my P value to my statistical significance level Alpha and what is this comparison well we know from the theory that um, if we have a low P value and specifically in the P that we are getting the P value is small than equal the 5% or 0.05 which is the significance level then this indic Ates that we have a strong statistical uh, evidence that uh, the N hypothesis is false and we need to reject it so we have a strong evidence against the null hypothesis and otherwise if the P value is larger than 0.05 so it's larger than 5% that we have chosen as the maximum threshold of that mistake so the significance level is uh, uh, no longer the largest element but the P vales larger than your significance level then this indicates that you don't have enough evidence against the null hypothesis so your evidence is weak this means that you fail to reject the N hypothesis. So this is what I'm doing in here with this code so I'm saying print the P value first and we are rounding it up with this round function I'm rounding it to the three decimal and then I want to check and determine whether I have a statistically significant or not and the way that I'm doing that is I'm saying if my P value is more than my alpha or actually lets at smaller than equal than Alpha then we can print that there is a statistical significance which indicates that the observed differences between the experimental and control groups are un unlikely to occur due to random chance which means that this is not random chance and uh we have a strong evidence that there is a statistical significance and this suggests that this new feature that we got this new version of our lending page with this um, uh, call to action um ASD and Dr Now is better and result in higher statistically significantly higher click through rate than the existing version of the control uh, group so there is a real effect then otherwise if this is not the case which means that my P value is larger than my Alpha then I'm saying print that there is no seral significance and that the observed difference that we see in the clickr rate is not because uh, of the real difference in the performance but TR truly this is just a random chance.

So here we can see that once we run our we call the function in here which is simply the function name and the argument so P value and alpha alpha comes from the initialized value that we had from our power nails so from here we initialize this value 0.05 and then here we got the P value that we just calculated then what we are getting in here is that our P value is actually so small that it's um rounded to the zero so what this means is that that there is evidence that suggests that at 5% statistical level significance level that the uh, click through rate of the experimental group is different from the clickr rate of the control group. Note that I'm not saying higher or lower because our stal test was two-sided so under n hypothesis we had that the uh, P control so in here as you can see our P control was equal to P experimental and under the alternative we had that the P control is not equal top experimental. This means that we um, have now rejected the null hypothesis we have found evidence that suggests that the null hypothesis can be rejected since our P value is zero and it's smaller than the statistical significance level 5%, and this means that we can reject the H and we can say that uh, there is enough evidence to say that P control is not equal to P experiment and given that that we saw from the uh, visualizations from our calculations that the um, clickr rate for our experimental group is much higher then the click rate of the uh, control group we can also say that we have found evidence that at 5% significance level we have found out that there is a statistically significant difference between the experimental and control groups clickr rate and that the experimental groups clickr rate is actually higher so statistically significantly higher than the control versions click to rate. So this is really important because this suggest that this difference in their click to rate is not due to random chance alone but truly that there is evidence statistical evidence that can support this hypothesis that there is a true difference between the performance of the experimental version of the product so in this case in our case the landing page that has enroll Now button versus the control version of the product which had the uh, uh, start free trial version of the landing page the existing version. So beside of calculating this P value it's always a great practice to also visualize your results and this is great for your audience who are technically sound and who know uh

These different concepts, and you want to visualize the results that you got. Not only by showing some number that is the P-value and say, "Hey, I have a statistical significance," but you also want to showcase the actual picture of what you got. What is your test statistic? What is the significance level that you use to kind of tell a story around your numbers? And that's the art behind data science, I would say. So let's go ahead and do some art.

So what I'm doing here is that I am making use of my standard normal distribution, or the Gaussian distribution—the way that we are referring to the standard normal distribution in statistics. I'm saying that my mean, or the μ, is equal to zero; my σ is equal to one, which is my standard deviation. And I'm saying that my—I want to now plot my standard normal distribution by getting my X values, which are the number of X elements that I want to have on my x-axis, and then taking the PDF, or the probability distribution function, for the normal distribution by using the SciPy library. I'm then providing my X values for which I want to get my corresponding values of Y. So basically, here are all the values between, let's say, minus something—minus three—and then so between minus three and three, and I want to find all the Y's corresponding to this, which basically plots the probability distribution function of the Gaussian distribution or the standard normal distribution.

And then I want to add to this graph also the corresponding rejection region, and as you can see, it is here. So then what I'm adding here, by using this part of the plot, is that I want to fill in the rejection regions. So I'm saying, for all the values in this figure, whenever the value is lower than that threshold—in this case, the threshold is Z critical 1.96—so whenever my threshold is smaller than minus this 1.96 and larger than this 1.96, then we are in the rejection region. We are saying then, if my test statistic is falling in the rejection region—in this case, you can see that we are in the far left—so the test statistic is minus 5.944, and it's much lower than this threshold, as you can see in here. This is this left blue line in here; then, in this case, it falls in this rejection region. So actually, this entire thing is the rejection region; it starts from here and it goes all the way to here. Anything, anything in this region means that we need to—we have a test statistic fully in the rejection region, which means that we can reject the null hypothesis. If we were to get a test statistic that is very large and very positive, it means we would be in this part of the figure, and again in the rejection region. Anything above this line is then going under this category of rejection region, and also anything in here. So for anything in here, we are in the rejection region. Being in the rejection region means that we can reject the null hypothesis, and we can say that we have a statistically significant result.

So now when we have our statistical significance, it's always a great idea to go on to the next step, and it's actually mandatory to do this because not only is statistical significance important, but also the practical significance, as I mentioned in the beginning of this case study. So for that, what we are going to do is first we are going to calculate the confidence interval of the test, and this confidence interval will help us to, first of all, make comments regarding the quality of our test and its generalizability at our entire population and the accuracy of our results. And then we will use this confidence interval to make comments and to test for the practical significance in our A/B test. So let's go ahead and calculate the confidence interval.

So as we learned as part of our lectures, the confidence interval can be calculated by first taking the P experimental hat and P control hat and the standard error and the Z critical. So here we need the two different estimates of the experimental group's click-through rate and the control group's click-through rate. We also need the standard error of our two-sample Z-test, as well as the critical value. And then we need to first calculate the lower bound of our confidence interval, and then we need to calculate the upper bound of our confidence interval. And in this case, given that the statistical significance level we are using is alpha, the Z critical is based on that; therefore, we are also saying that we are calculating the 95% confidence interval. So in here, the way we will calculate the lower bound is by taking the P experimental hat, subtracting from that the P control hat, and then once we have done that, we then subtract from that the standard error multiplied by Z critical value, and we are just rounding this up to the three decimals behind the zero. Then we are doing the same thing, only with a plus sign in here, for the upper bound calculation of the confidence interval. So this is just pure following the formula of the confidence interval that I will show you here. And let's go ahead and print this value, which is this interval.

So what we are seeing here is that we have a confidence interval that is from 0.399—so 0.4 to 0.43—quite a narrow confidence interval, I would say, which is actually a good sign because this confidence interval that provides this range of values within which the true difference between this control and experimental groups' proportions, or the click-through rate, is likely to lie within a certain level of confidence—in this case, 95% confidence. This is very narrow, and if it's a narrow confidence interval, it means that the accuracy of our results is higher, and it means that the results we are getting based on our smaller sample will most likely generalize well when we apply these changes and deploy these changes and we put this new product in front of the entire population of users. Because now we are doing all this experiment for a small group, for the sample, and this confidence interval that is narrow—it's not wide, it's narrow—it means that the results that we are getting are accurate, more or less accurate, and this means that we—the results that we are getting based on the sample are most likely a true representation of the entire population that we got. This is the idea behind the width of the confidence interval: the narrower it is, the higher is the quality of your results, which means that the more generalizable are your results.

So let's now go on to the final stage of our case study, which is to test the practical significance of our results. So now when we know that the statistical significance is there—the experimental version of our feature is statistically significantly different from the control version in terms of the click-through rate—and we have seen that the confidence interval is narrow, which means that our results are accurate, quite with quite high accuracy, then we can now comment on the practical significance of our results. This means we want to see whether the significant difference that we obtained—whether this difference is actually large enough from the business perspective to say that it's worth to put our engineering resources and our money and our product into—to put through this change and to say that it's worth from the business perspective to change this button and to put this into production and in front of our users. And of course, here we are not only talking about the engineering resources that it will take from us to change this and the deployment and the monitoring, but also in terms of the quality of the product we are providing to our users because whenever we are making a change to our product, it is a risk because we are changing what our user is used to see, and this can always be scary when it comes to the business because we don't want to make our customers scared. Therefore, we need to also check for this practical significance.

So for that, what I'm doing is that I'm creating this Python function that will take two arguments—so two values—that is the minimum detectable effect and then the 95% confidence interval that I just calculated. Those will be the two arguments for my function, and I'm calling this function `is_practically_significant`. And this function will go and check whether the practical significance is there or not, and it will then return true or false, and then it will also print whether we have a practical significance or not. And we learned from the theory, and we know from this A/B testing concept, that whenever the MD, or the delta that we got—the minimum detectable effect—is larger than the lower bound of our confidence interval, it means that the lowest possible value that we can get based on the results that we obtain in our sample—that that amount is smaller than the minimum detectable effect that we assumed before even conducting our A/B test. This suggests that we have a practical significance, and the difference—the minimum difference that we will obtain—is large enough for us to have motivation to make this change in our product. For that, what I'm doing is that first I'm taking my 95% confidence interval and I'm taking the first element because we know that a confidence interval is actually a range, so two of two numbers—the lower bound and upper bound. I need the lower bound because all I care for this practical significance is to compare the lower bound of the 95% confidence interval to this minimum detectable effect, which is my delta. So therefore, I'm taking this lower bound of the confidence interval, putting that into a variable, and then I'm using this variable—this lower_bound_CI—and I'm comparing this to my delta. I'm saying, if my lower bound of the confidence interval—actually, I'm noticing that here I got a mistake; it should be the other way around. We need to say that if our delta is larger than or equal to the lower bound of the confidence interval, which is the same as if our lower bound of the confidence interval is smaller than or equal to our delta. So if—however, we can also write this the other way around. So if our delta is larger than or equal to our lower_bound_CI, then we can say that we have a practical significance. So the MDA of—in this case—so I want to use my initial delta; therefore, I won't be initializing this. So you might recall here a delta of 10%; I want to still make use of that delta, so therefore I will just go ahead and then in here what I want to do is to call this function by using that specific delta. So I want to have 10% as my MD, and whenever this delta will be larger than the lower bound of my confidence interval that I just obtained, I will then say that we have a practical significance. And with an MDA of 10%, the difference between the control and experimental group is also practically significant. So you can see that the lower bound is 0.04 something that we obtain here, and that amount is then compared to this delta, and here you can see that we have concluded that we also have a practical significance.

Amazing! We have come to the end of this case study. In this involved case study, we have conducted an entire A/B test results analysis. So this case study—it has—and to end, going from the point of loading the data and then understanding this business concept or business objective of the A/B test, where we were testing whether the "Enroll Now" button—which is the new version, the experimental version—should replace the existing button, which is the "Secure Free Trial," and based on this case study, what we found out is that we have a statistical significance at a 5% significance level, suggesting that we can reject the null hypothesis, and we can say that indeed there exists a statistically significant difference between the click-through rate in the experimental group versus the control group, and specifically that the "Enroll Now" experimental button results in a statistically significantly higher click-through rate than the "Secure Free Trial" button. And beside this, we also checked the accuracy of our results by looking at a confidence interval, and we saw that the confidence interval was quite narrow, suggesting that the results we obtained were quite accurate, and this means that the results that we got for the sample will generalize to our population of users. And finally, we have also checked the practical significance of our results by using the 95% confidence interval and comparing the lower bound of that interval with our minimum detectable effect, delta, and we saw that we will have at least a 10% significant difference between the control group's CTR and the experimental group's CTR, and the experimental group's CTR will be at least 10% higher than the control group's, and this suggests that from the business perspective we also have motivation. Beside this statistical significance, we also have practical significance, suggesting that we also have enough motivation and reason from the business perspective to put this new button into production, and we can conclude that based on this data-driven approach and conducting an A/B test, we can see a clear motivation of deploying this new button, "Enroll Now," and replace the existing one, "Secure Free Trial" version, and we will then expect to see more users clicking on this and engaging with our product.

And for now, this will be all for this case study. If you want to learn more about A/B testing, make sure to check our A/B testing course, as well as the Ultimate Data Science Bootcamp. Don't forget to try our free trial this time using our "Enroll Now" button. And if you want to see more case studies like this, make sure to check our other case studies. We have many case studies also included as part of our Ultimate Data Science Bootcamp, where we go in detail of these different steps and we conduct different sorts of case studies to put our data science theory into practice, including from the field of NLP, machine learning, recommended systems, advanced analytics, and also A/B testing, and soon also from AI. So for now, thank you for staying with me and conducting this case study. Happy learning! This video was sponsored by Lunarch. At Lunarch, we are all about making you ready for your dream job in tech, making data science and AI accessible to everyone. With data science, artificial intelligence, or engineering at Lunar Tech Academy, we have courses and boot camps to help you become a job-ready professional. We are here to help also businesses and schools and universities with top-tier training modernization with data science and AI corporate training, including the latest topics like generative AI. With Lunar Tech, learning is easy, fun, and super practical. We care about providing an end-to-end learning experience that is both practical and grounded in fundamental knowledge. Our community is all about supporting each other, making sure you get where you want to go. Ready to start your tech journey? Lunner Tech is where you begin. For students or aspiring data science and AI professionals, visit the Lun Tech Academy section to explore our courses and boot camps, and just in general our programs. Businesses in need for employee training, upscaling, or data science and solutions should head to the technology section on the Lunarch page. Enterprises looking for corporate training, curriculum modernization, and customized AI tools to enhance education, please visit the Lunarch Enterprises section at Lunarch.com for a free consultation and customized estimate. Join Lunarch and start building your future one data point at a time.