FREE Data Scientist Course | Data Science Course With Projects | Data Science Course | Simplilearn

Welcome to this free data scientist course designed especially for beginners and aspiring data professionals.

So in today's fast-evolving digital world, data science has become one of the most in-demand and transformative fields. This session will guide you through the core foundations and tools essentials to becoming a data scientist.

So we will start with an introduction to data science and the role of a data scientist. Then we will move into hands-on tools like R and Python, two of the most widely used programming languages in data analysis. We will also get a solid grounding in SQL and statistics, crucial for data manipulation and insights. Next, we will explore some data visualization using Tableau, helping you turn complex data into compelling stories. For those just starting out, our data analytics for beginner section offers a friendly entry point. Finally, you will discover real-world data science projects that bring everything together through practical applications.

So now let's move on to the introduction to data science part. So data science helps businesses understand and utilize data effectively. It uses math, stat, AI, computer engineering techniques to analyze large data sets. By doing this, data scientists can discover insights about past events, their causes, future trends, and how to apply information.

So data science is a multi-disciplinary field that uses scientific methods, algorithms, processes, and systems to extract knowledge and insight from structured and unstructured data. It combines computer science, statistics, domain knowledge, and data visualization expertise to solve complex problems and make data-driven decisions and predict future outcomes.

So now let's move forward and see what does a data scientist do. A data scientist extracts, analyzes, and interprets data to uncover valuable insights and inform strategic decisions. They collect and clean data from various sources and analyze expiry data to identify patterns.

Now let's move forward and see roles and responsibilities of a data scientist. The first one is data collection and cleaning. Data scientists are responsible for acquiring, collecting, and cleaning data sets. This involves sourcing data from various databases, APIs, or other sources and ensuring its quality by handling missing values, outliers, and inconsistencies. The second one is EDA, exploratory data analysis. Data scientists perform EDAs to gain a deep understanding of the data. They use statistical and visualization techniques to identify patterns, correlations, and outliers which inform subsequent analysis. Third one is model development. Developing predictive models is a core responsibility. Data scientists build machine learning models to solve specific business or research problems, selecting appropriate algorithms and fine-tuning them for optimal performance. Then comes data visualization. Data scientists create informative visualizations and reports to communicate findings and insights to non-technical stakeholders. Effective visualization is crucial for making data-driven decisions. Then comes feature engineering. Feature engineering involves selecting, creating, or transforming relevant variables to improve model accuracy. Data scientists are skilled in identifying the most informative features for a given problem.

So after understanding roles and responsibilities of a data scientist, let's see some skills required to become a data scientist. So here are six essential skills one should possess. The first one is programming. Proficiency in programming languages like Python or R is fundamental for a data scientist. They use these languages to manipulate data, build machine learning models, and develop data analysis scripts. The second one is statistics and mathematics. A strong foundation in statistics and mathematics is essential for understanding the underlying principles of data analysis, hypothesis testing, and machine learning algorithms. The third one is machine learning. Data scientists should have expertise in machine learning techniques including supervised and unsupervised learning, deep learning, and reinforcement learning. They need to choose, train, and evaluate models effectively. Then comes data manipulation. Skills in data preprocessing, cleaning, and transformation are crucial. Data scientists work with messy and complex data sets. So the ability to prepare data for analysis is a key skill. Then comes data visualization. So proficient data visualization skills help data scientists communicate their findings effectively to both technical and non-technical stakeholders. Tools like Matplotlib, Seaborn, or Tableau are commonly used.

So at last, let's see the roadmap to become a data scientist. So becoming a data scientist is a rewarding journey that requires dedication and a structured approach. So here is a comprehensive roadmap to help you become a proficient data scientist. First one is prerequisite educational foundation. Obtain a bachelor's degree in relevant fields such as computer science, mathematics, statistics, engineering, or a related discipline. Consider pursuing a master's or PhD for more advanced roles. Second comes programming skills. Start with Python, a widely used language in data science. Learn the basics of programming, data structures, and algorithms. Step two, fundamentals. First one comes mathematics and statistics. So master the key mathematical concepts including linear algebra, calculus, and probability. Understanding statistical methods and hypothesis testing. Then comes data manipulation. Learn data manipulation libraries like pandas for Python. Practice cleaning, reshaping, and exploring data sets. The third step is data visualization. Explore data visualization libraries like Matplotlib and Seaborn. Create meaningful visualizations to communicate insights effectively. Then the fourth step is machine learning fundamentals. Study the fundamentals of machine learning including supervised, unsupervised, and reinforcement learning. Learn about model evaluation and validation. The next comes machine learning libraries. Familiarize yourself with machine learning libraries such as scikit-learn and TensorFlow. Implement various algorithms and models. Step five comes advanced topics such as deep learning. Dive into deep learning using frameworks like TensorFlow or PyTorch. Understand neural networks, convolutional networks, and recurrent networks. Then in this comes natural language processing, NLP, and computer vision. Explore NLP techniques for text analysis and sentiment analysis. Learn computer vision for image recognition and object detection. Then step six, real-world applications or experiences. Kaggle and personal projects. Participate in Kaggle competitions to apply your skills. Work on personal data science projects to build a portfolio. Then comes internship or entry-level positions. Gain practical experience by securing an internship or entry-level position in data-related roles. Step seven is advanced learning specializations. Consider specialization in areas like data engineering, data analysis, or machine learning engineering based on your interest. Then step eight, soft skills. In this comes communication skills. Improve your ability to communicate data-driven insights to non-technical stakeholders. Step nine, continuous learning. Stay updated. Keep up with industry trends, new tools, and techniques through blogs, online courses, and conferences. Then comes professional networking. Attend data science meetups, conferences, and join online communities to connect with peers and mentors. Then the last step, then job search and career development. In this job search, apply for data science roles that match your skills and interests. Tailor your resume and cover letter to highlight your relevant experience. Then comes interview preparation. Prepare for data science interviews by practicing coding challenges, data analysis exercises, and behavioral questions. At last comes career growth. Continuously seek opportunities for career advancement and skill development. Remember that becoming a data scientist is a journey that requires patience and persistence. Adapt your roadmap based on your individual goals and pace of learning and never stop exploring new challenges and opportunities in this dynamic field.

In the next module, we will learn data science with R. What's in it for you? We're going to go through an introduction to R, why R, CRAN, comprehensive R archive network, and cover installing R. Then we'll get into simple linear regression using R, line of best fit using error summation, correlation analysis in R, and then we'll get into classification using R, use case, predict the class of a flower.

Let's start with an introduction to R. First, it's an open source. R is completely free and open source with active community members. Extensible. It offers various statistical and graphical techniques. Compatible. R is compatible across all platforms, Linux, Windows, and Mac. Library. R has an extensive library of packages for machine learning. Easy integration. It can be easily integrated with popular softwares like Tableau, SQL Server, etc. This only touches a little bit on these different aspects of R. The fact that it's free, it's very extensive. It probably has one of the most extensive sets of data analysis packages currently out. It has a compatibility that is continually growing. So, it's integrated with everything from cluster computing to Python integration which is now coming out. And its extensive libraries allow you to import the different libraries to use for whatever your needs are, makes it a very diverse and easy-to-use coding source for analyzing data.

R is more than just a programming language. As I just touched upon, it has a worldwide repository system, comprehensive R archive network, and it can be accessed at https://cran.r-project.org. It provides up-to-date versions of code and documentation for R. CRAN hosts around 10,000 packages of R. That is a huge repository focused on just data analytics.

Let's install R to get started. You can easily download the executable file for R and install it from the CRAN website. If you go to under the downloads, in this case, since I'm on a Windows machine, we'll walk through it from the R 3.5 version for Windows. So, here we are at https://cran.r-project.org. And you can see down here we have the different options for the downloads. If we go under Windows, which is I'm on a Windows machine. If you're on a Mac machine, you can do the Mac or Linux. Uh there's install R for the first time. Just click on that and open and run it. The installation is pretty simple. Follow the default options to finish the installation. After the installation is complete, you'll see the R icon on your desktop. Alternatively, there are mirror sites. And so we can go into RStudio. Let's just take a quick look at that. If you're on the RStudio website, that's www.rstudio.com. You can go down under products and RStudio download. You'll see a number of options here. The first one is the RStudio desktop open-source license. That's the same thing you just downloaded. So, if you download that, that's what you're getting. There's also an open-source server setup, which is also free. You can download from here. And then RStudio offers a number of paid packages which includes all kinds of different supports for a company and individual level setup. Whether you go through the CRAN site or the RStudio site, the installation is pretty simple. Just a quick note, if you're a Debian distribution, including Ubuntu, you can install R using its regular package management tools. And when you're using a Linux system, that is preferred because then it registers it properly on the setup system.

Let's go ahead and open up our RStudio and take a look. Here we are. And I've opened this up in the RStudio version, which automatically opens up some extra windows, which is nice. Let's go and take a look at that. We have our console on the left. This is your main workspace. So, if I go in here and I do uh click the mouse and I do a 4 + 4, I'll come up and say the answer is eight. And you have some environmental information. And over on the right, you have plots. Usually, when you're working in here, let's do a script. We'll go up here to the plus sign in the upper left-hand corner and just add some script in here. In this case, it showed up on the top, but you can move these windows around. And let's say I do y = 3 + 4, x = y + 2, and then I'm going to do just x. And let's do a um plot. We'll throw a plot in there. c is a notation that these are going to be Cartesian points. So we got 1, 2, 3. That'd be like your x and then y 3, 4, 5 for just a standard call scatter plot. Um you don't have to memorize this. We'll go into some of this later on. And then I can take all of this. If I go under code since I'm working in the code console and I go down to run region and just run all, it takes this code and just runs it through my console. And you can see down here it's executed uh y = 3 + 4, x = x + 2. And if you add 3 + 4 + 2, you get nine. That's what the x does. And then I did a plot. I threw the plot in there where I'm plotting 1 2 3 3 4 5. And if you use just a straight plot, it's a scatter plot. And you can see that this appears on the bottom right where you have your plots coming in. So it it's a very quick way to show data. And that's one of the wonderful things about R is it's very easy and quick to go through um and do different functions on the data and analyze it. Uh so it's a very popular package.

Before you start programming in R, you should install packages and its dependencies. Packages provide pre-assembled collections of functions and objects. Each package is hosted on the CRAN repository. Not all packages are loaded by default, but they can be installed on demand. Remember earlier we were talking about all the different packages available. You don't want to install everything from R. It would just um be a huge waste of space. You want to just install those packages you need. So to install packages in RStudio, you go under tools and install packages. When you click on the install packages, you'll get a dialogue box. You'll see where it has a repository CRAN because there are other repositories and you can even download and install your own packages you can build. And then you'll pick out the packages you want and separate multiple with spaces or commas. So you can install numerous packages at the same time. In this case, we've uh installing the forecast package. And you can see down here I just type in forecast, install all the dependencies, uses other packages to build on it, has those and then just click on install and it's done.

Before we cover the basic linear regression model, let's just take a quick look at some of the different parts of R or parts of uh programming you'll need or scripting. Uh first, you need to know there are various data structures in R. We have vectors, we have matrices, we have arrays, data frames, and lists. Vectors are the most basic data structure. Uh if you remember, vectors are a location and a direction. It's how they're generally defined. Although a vector, when you're talking about scripting, can contain numerous different values. You could have vectors with four, five, six, seven, eight different values in it. For example, a picture on the computer might have the location of the pixel and the number for the red hue and the green hue and the blue hue. So now you have something with five different numbers in it consisting of that vector. Matrices allow you to move stuff around. So you might have a a 2x3 matrix that you switch to a 3x2. We looked at an xy plot earlier. It might be that you have everything uh is you have 10 different numbers and each one has two values xy and you need to switch that matrix. So then you have two arrays of five numbers. Arrays are just that, a collection, and you can have arrays of arrays of arrays. Data frames have labels on them which makes them easier to use. So usually we use a lot of data frames when we're working with data uh because they're just easy. You can have a column and you can have a row. So think of rows and columns when you see the term data frames. And then lists are usually homogeneous groups. Uh so in R you're usually looking at similar data that's connected in the list.

So the first thing we do with before we even importing the data, you should have the data ready. So we need to look at the data and see what's going on. Uh you can import data from various sources including Excel, Minitab, CSV, table, text files. Uh CSV is usually a text file, comma-separated variables, tab-separated variables. There's all kinds of different options here. And importing the table is very simple. We have read.table, file = data.table, header = TRUE. And let's just take a closer look at what we're talking about here. Scratch the closer look. I didn't realize it was just for example. So importing a table file is pretty simple. We read.table, file = data.table. So whatever the name of the file is, comma header = TRUE. So if there's a header to it, in this quick flash, they have name, age, gender. We'll actually do this in R with another data set in just a minute. But you can see it's very easy to read. A lot of people use R even in other programming languages. Uh so they can quickly look at the data before they even start analyzing it. CSV file, same thing. read.csv, file = data.csv, header = TRUE, sep = " ". So in this case, even though CSV stands for comma-separated variable, this one is separated by spaces and they have the header, name, age, gender. Uh so it's the same file saved as a CSV file. And there's also Excel. Um Excel has its own issues as far as making sure you know what the tables are and the headers are. Uh but you can see that each one of these is easy to import. So and just like it's uh easy to import the data, you can also export tables in R. So you can see here uh write.table, myfile.txt, sep = "\t". comma-separated and the \t just means it's tab-separated. So if you're using tabbed files on there, example, Excel, so you can write.xlsx to myfile. In this case, they did a text separation = "\t". So it's a tab-separated for Excel file. CSV, same thing. Very easy to write a CSV file to your computer once you've changed the data or altered it depending on what you're doing with it.

And once we have our data imported and we can save it afterwards and export it. Uh graphing visualization in R is very powerful and it's quick. I love doing this before even exploring the data. Sometimes you just want to graph it to see what it is there so you have an idea of what you're looking for. So graphics in R cover a huge amount of different things. R includes powerful packages of graphics that help in data visualization. These graphics can be viewed on screen, saved in various formats including PDF, PNG, JPEG, WMF, and PS, can be customized according to varied graphic needs. You can copy and paste in Word or PowerPoint files. R supports various types of graphics including bar chart, pie chart, histogram, kernel density plots, line chart, box plot, heat map, word cloud. Uh there's even some more obscure ones, but these are the main ones that most people use. Um I know I use a lot of heat maps. Um but word clouds are a lot of fun if you're doing websites and data and uh word analysis. Then histograms are very popular. All of these are very widely used.

Let's look at the box plots, also known as whisker diagrams. Box plots display the distribution of data based on minimum, first quartile, median, third quartile, and maximum. So, right off the bat, we can use a box plot to explore a data set with very little work. To create a box plot, we simply give a boxplot and the data. Very straightforward. So, we might have passenger numbers in the thousands. I guess this is exploring data dealing with airplanes. And you can see here they just have a simple plot. If you break it down, you have your uh if we go back one, you'll notice we have like a minimum, a maximum, our median, first quartile and third quartile. Let's just break that apart. You can see the line on the bottom is your minimum. Your line on the top is the maximum. You have your median and your first quartile right there. First quartile and third quartile. And the way a lot of times you read this is things that are above the box are outliers and things below the box are outliers. And what's in the middle of the box is usually the data you're looking at.

Linear regression. Now we've had a few tools of what R can do. Let's look at the theory behind linear regression and how that applies to what we're going to use in R. Let's understand more concepts by solving a few algorithms. Firstly, linear regression which can be applied in two ways. One, we can estimate the relationship between two variables. Does body weight influence the blood cholesterol level? So we just want to know if it's even a valid connection or not. Will the size of the house affect the house price? So we can estimate the relationship between two variables also. And then we can predict the value of one variable dependent on the basis of other independent variables. So once we know they're connected, we want to use that to go ahead and predict it. Uh in the cases above, maybe the body weight versus cholesterol. So the more the higher the body weight is, probably the higher cholesterol. So we first we explore is there a connection? And if there is, can we actually apply that to a linear regression model where we can see those how they're connected? Same thing with house pricing. The simplest form of a simple linear regression equation with one dependent and one independent variable is represented by y = m * x + c. y being the dependent variable, x being the independent variable, m being the slope of the line, which you can see over here on our graph, m = (y2 - y1) / (x2 - x1), and c being the intercept or the coefficient of the line. Now, you don't have to memorize all the formulas we're going to go into here because most of R does this for you. But you do need to know what's going

On the back end, so that you can understand if it's working or not working, where the problem lies.

Types of linear regression. There's simple linear regression; it considers one quantitative and independent variable X to predict the other quantitative but dependent variable Y. And that's the formula we just saw: Y = M * X + C. And then we have multiple linear regression; it considers more than one quantitative and independent variable. So, instead of having just X, you have X of one, X of two, X of three to predict the other quantitative but dependent variable Y. So, instead of looking at just body mass, we might also look at height. So you have body mass and height; those would be two independent variables. Um, housing prices might have to do with the distance to a certain area. So location might be one of the variables along with the size of the house.

Let's consider a sample data set with five rows and find out how to draw the regression line. So, in here you can see we have an independent variable and our dependent variable where we've taken some random data. In this case, we have 1, 2, 3, 4, 5 for X and we have 2, 4, 6, 5, 8 for Y. And the first thing we do is we might plot it. If you look at the plot, you can see that it has a nice line through it. Just eyeballing it, you can guess that there's probably a linear regression model that will fit this data. Then we go ahead and calculate the mean. We calculate the mean of X and the mean of Y. And you can see that we plotted that on there—3 and 5 for this particular data set.

And so, drawing the equation of the regression line, we'll go ahead and take a summation of X, a summation of Y, take a summation of X², and a summation of Y², and then X * Y. This is why it's so nice for the computer to do this work for you because, I don't know about you, I hate sitting there with a calculator. And if you have just five variables, that would be very long—not a big deal. But if you're doing hundreds of variables, definitely much easier to let the computer do it all.

So the intuition behind the regression line breaks down to the linear equation, which is represented as y = m * x + c. In this case, m equals this very lengthy formula where you have your summations. So all the values that we've already computed fit right into this equation. Um, and so you have n being the number of variables—that's what the n stands for. Your x times y. We computed the sum of x, sum of y over the number of variables times the sum of x² minus the sum of x squared. And then, of course, we have—you put the numbers in and you get a value of 1.3 and you find out that your c is equal to 1.1. And once we have this formula, we can easily compute the y values: y = m * x + c = 1.3 * 3 + 1.1 = 5. And this would be the example of x = 3. You could also put in x = 6, an unknown value.

Let's find out the predicted values of y for corresponding values of x using the linear equation where m = 1.3 and c = 1.1. So here's our regression line. And then you can see if we put in all the numbers 1, 2, 3, 4, 5, we don't get the exact prediction because it's a line through the data. You'll see that it's ever so slightly off: 2.4, 3.7, 5, 6.3, 7.6. But it is a good estimate of what the data is going to generate.

Let's find out the predicted values of y for corresponding values of x using a linear equation where m = 1.3 and c = 1.1. The best fit line should have the least sum of squares of these errors, also known as square, the predicted y-value, residuals of errors, the actual y-values. So one of the things we're looking at when we put this information together is we want these distances to be minimized. So it has the smallest amount of error possible.

Let's find out the predicted values of y for corresponding values of x using the linear equation where m = 1.3 and c = 1.1. And here you can see we've done the same kind of chart. We have our y predicted, and then we have the actual y minus the y predicted because we are looking for the error squared or the e squared values. The sum of the squared errors for this regression line is 3.9. We check this error for each line and conclude the best fit line having the least e values—minimizing the distance. There are lots of ways to minimize the distance between the line and the data points, like sum of squared errors, sum of absolute errors, root mean square error, etc. And no matter how you compute the errors, you can see here this is what it's basically going through. It's taking the line and slowly adjusts it until it has the minimal error available, the minimal distance between the line and the different data points.

Use case. We'll start with linear regression, and then we'll jump in and also do a decision tree so you can see how they are the same and how they differ in the way they function and what they're used for. And while we're going through the use case, you can start linking the theory behind it connected to the actual use. And as you connect those dots, you can see that by making changes in the model based on the theory, you can also fine-tune it. We'll talk about that briefly as we go through this.

For this analysis, we'll use the default cars data set to find the correlation between variables. Cars is a standard built-in data set that makes it convenient to show linear regression in a simple and easy-to-understand fashion. And we'll start with head cars and string cars. Now, I've gone ahead and removed the script on here to make this easier to see. And we don't have to do any kind of importing of data because it's already in there. I'm just going to type in head and cars and hit enter. And what you're going to notice is it's going to display the first six rows: 1, 2, 3, 4, 5, 6. And you'll see speed and distance. Now, this is a little bit like a spreadsheet. So, if you're using an Excel spreadsheet, it should look the same. And this is a standard data frame kind of setup. Um, in this case, whenever you say head, it usually lists the first number of rows. Depending on your package in R, it starts with one and then it does the first six rows. String, when we do a string of cars, it's going to break it down and show us that it's a data frame. It has 50 objects—that's what the 50 OBS of two variables. And then we have speed. The first variable is going to be a number—is what NUM stands for: 4, 4, 7, 7. It just starts listing the data there. And distance. Now, I'm assuming the speed is the speed of the engine. Otherwise, if it's the speed of the car, then the distance should always be the same. And that would just be kind of silly. Uh, but this is the speed of the engine. And based on the speed of the engine, can we correlate that to the distance?

Let us visualize the data using a scatter plot to understand the relationship between predictor and response. And I want you to notice that in this slide, a line has been drawn through the data. Now, when we first start with data, you don't have that line. So, as we plot it, I want you to kind of visualize that line there and just kind of say, hey, does this stuff kind of scatter around a line or not? Uh, before we actually create the model and we simply do plot, and I'm going to do a shorthand here, cars, and I hit enter. And it comes over here to the right and plots our data for us. Now, because there's only two variables, I don't have to do anything special. I'm going to expand this over environment so we can see it. And you can see you can just visualize a line right through the data, and it's all clumped along that line, which makes it really nice for doing a linear regression model. Now I did something a little tricky here because I'm very lazy. I typed in plot cars. A lot of times you don't want to do that because you don't want to plot all the data. We could do plot in this case cars dollar sign speed, the first variable, cars dollar sign distance. When I do that, I get the same plot. All I'm doing is telling it to use just these two columns of data. And although the first one is quick and easy if you only have two columns of data, the second one is what you really want to do when you're plotting data like this. You want to control which columns you're using.

And then we want to do a correlation. Correlation analysis studies the strength of the relationship between two continuous variables. It involves computing the correlation coefficient between the two variables. If one variable consistently increases with increasing value of the other, then they have a strong positive correlation—a value close to one. And again, I can just type in correlation cars, the short form, and we can see we have a nice correlation going on there between the two. What we really want to do though is we want to do—we want to correlate cars, and then we're going to do the speed, cars, and the distance. And so when you look at this, I actually hit the enter key in the middle of that, which is fine. You can, in R, do it line by line. So if you're trying to do a bunch of different columns, you might want to do that to make it easier to read. And you can see here I get the 0.806894. The closer this is to one, the more these correlate. So it's saying there's a pretty good correlation, positive correlation between the two variables. And if you look at it, when I did just cars, which did the correlation over all the variables, you'll see that speed correlates with speed 100% because they're the same. That's just the nature of that. But when you look at speed versus distance, we get the 0.806849. And if you do distance to distance, you get also one since they're identical variables.

So now it's time to build our linear regression model on the entire data set to build the coefficients. Let's just take a look and see what that looks like. Back in our console, I'm going to type in linear mod. And in R, we want to do the arrow—kind of like an arrow and a line—or in this case, the less than minus sign. That's the same as assigning whatever we're going to put after it to this variable. So we're creating the linear variable. lm stands for linear model, and we're going to look at speed and distance. So we put the little squiggly bracket between them. This lets R know we're going to deal with these two columns, data equals cars. And when I hit enter on here, we've generated our linear mod. Now we want to go ahead and summarize it. And we simply do a summary and then brackets linear mod. And it generates all kinds of information on our model. So we can explore just how well this model is fitting the data we have right now. Now if you've done linear regression in other packages and scripts, you're going to see that this is so easy to explore data in R. So even somebody who's working in, say, Python or Spark, Hadoop, coming back to the R console to do some basic exploration of the data is very beneficial. Uh, and if you use just R, R goes into all kinds of different packages, and you can even do cluster computations through H2O. There's a lot of cool things. This is what makes R so wonderful is how easily we can summarize something like a linear model.

And to explore this data, let's go back to the slide, and we'll see here where it says the residuals. The first one, q stands for first quadrant median, third quadrant max coefficients. And we first want to do is look at the estimate standard deviation on here. And you'll see the intercept is at -17.5791 and speed 3.9324. So we can compute the distance equals to the intercept plus the beta times the speed, which just means that we can assign a distance equal to -7.579 plus the 3.932 * the speed. The value of p should be less than 0.05 for the model to be statistically significant. And this is one of the things I love about R. You can see right down here, we punched in the numbers, and it even tells you the significant codes. So, if you're trying to guess or remember, it tells you right away, hey, this 1.49 to the minus 12th is way below 0.001 on there. And even puts three stars next to it. So you can see that that is a very high correlation. And you can see the 0.0123 gets one star because it is greater than the 0.05 or it's less than the 0.05. So this shows that there's a very high correlation—statistically is very significant.

So now we're going to go ahead and create a training and a test data set. And the first thing we want to do is we want to set the seed so the sample can be recreated for future use. Remember, in all these different regression models, they usually have some randomization going on in the back to kind of fit it and guess where it's going to go. And if you want everything to match, you want to start with the same seed in that randomizer. And that way it recreates it identically every time you run it on a different computer. And just like working with data in any package, we're going to create indices for the training data and we're going to model the training data, and then we're also going to do the test data and build our model on the training data. Let's walk through that and see what that looks like in R. And I actually mistyped that one. Let's type in set to set our seed for the 100. Let's go training row index, and that's our name we're making up for our variable, and we're going to set that to sample. Now sample is a command in R. So we're going to take a random sample of some of the data, and then we're going to do one colon in rows cars. So this is a number of rows and cars, and we're going to start with one and sample them. Uh, so row row one of the data, we're skipping any of the titles on the top. And then in the sample, it has two variables that go in. The first one is the in rows of cars. And then we want to do 0.8 times in rows of cars. And this basically says that we're going to take 80% of all the rows in the car sample. We'll hit enter to go and run that. And we see here that I had an error. And that is because it's in row—actually put an s on there. That's something I do a lot. We'll just fix that real quick. It brings up a point that if you use your up down arrow, you can quickly paste through the last things you typed in, and then you can reuse them. So there we go. Now I have our training row index, which is basically a list of all the rows we're going to use. And now that we have a training row index, we'll go ahead and create our training data variable. And we'll assign that cars. And we'll put that as a training row index. And this is supposed to be brackets because we're dealing with an array type format. We want to do the training row index, so we just need the index of the row. And as far as the columns, we want all the columns. So here we have our training data. And now we set up a training data; we also need our test data. And with our test data, we'll assign that also to cars. And because we used the training row index and we used the sample to create it, we can simply do a minus sign training index, comma, so it's identical to the training data, but the test data is going to be the ones, the rows that were not included in the random sample of 80%. So it's going to be 20% of the data. And of course, it's very easy to do mistakes on here. So we'll come back into this, bring back up the error, and then we have training a training index, training row index. There we go. Very easy to correct errors in R. So now we have our training data and our test data. So we have 80% in training data and 20% in test data. And let's go ahead and create our model. We'll call it LM mod. And this is identical to what we did earlier. Here's our linear model. We're going to have our two different columns. If you remember above, we had to tell it which columns we wanted. And we're going to do distance and speed. And then the data we're going to set equal to training data. So as opposed to doing all of cars, we want to use just the training data on here. And now that we've created a model off of the training data, we need to go ahead and run our prediction off of our test data. We want to see how the test data, which has nothing to do with the model so far, how well it fits. And we'll go ahead and call it distance predict. Um, there's our assignment. So we're going to assign the value to it. And with R, we type in predict, and then we called our model. We go ahead and pull up our model lm mod. And so we're going to use that model, and then we want to put in the test data. Here's our test data. And so now we're creating the distance predict variable, and we're going to put all the information as far as predicting on our test data. And we'll run that—just hit the enter key. And then if you remember from before, we want to review the model diagnostic measures. And so we're going to go in here and do a summary. And you'll see in here we have a couple different things going on. But let's go ahead and take a summary of this and just walk through what this means. So summary, and then we call it LM mod. And we hit enter. And it pulls it up. And we have a nice summary of our LM mod. And the first thing we're going to note is that the data that's right now in our LM mod that we're summarizing is our training data. And let's go back to the slide and highlight this. So the data equals the training data—that's where the LM mod comes in. Residuals, we have our minimum, our first quarter, median, third quarter, and the max or quadrant, not quarter. You'd think I was doing a business analysis as opposed to end-of-year taxes in first quarter versus quadrant. We look at our coefficient. We can see where our intercept is and our speed is. And we note over here our significance codes. So they all match up. So a simple correlation between the actuals and the predicted values can be used as a form of accuracy measurement. So we take a look at the model we created, and now we need to take our predictions and just see how accurate it is on the data that we didn't use to program the linear model. So let's create a variable called actuals predicts and assign it data frame. This is how we create a data frame is we just assign data.frame, and we'll do a cbind, and in our cbind we're going to take our data and we're going to create a column called actuals. So this is our choice, the name actuals. This is going to equal test data, and we're going to use a dollar sign and distance. And we're running out of little room there, so let me pull this over so we can see it better. There we go, distance. And then we also want to take and compare that to what we predicted for our other data. And we'll call this column predicted. And that's going to equal our distance predict. And remember distance predict is set to the predicted values of our test data. So what this is is we're saying, hey, here's our test data, the actual data and the distance. And then we have what we predicted that distance to be. And we'll go ahead and assign that. And we can do a quick head, actuals predicts. There we go. And remember, head shows us it's a data frame. So it's going to show us the labels and the first set of data. In this case, you'll notice that the rows have a different count; they're not 1, 2, 3, 4, 5, 6. Well, we randomly picked 20% of the data. So this is the first six rows of that random selection, which comes out as 1, 4, 8, 20, 26, 31. And we have the actuals. So,

The actual value is two. And the predicted value on this first one is minus 5. So, they're way off. Uh, 227, 2620, 2637, 54 to 42, 50 to 50. So one of these is actually pretty right on, where a lot of them are really off at the beginning.

Let's just see what that looks like though, cuz just eyeballing the first six rows does not tell us what's really going on. So let's create another variable, correlation accuracy, and we're going to assign this correlation CO. Remember we do a CO, and we have our actual predictions. So, let's just see how our actual values versus the prediction correlate with each other. And we assign that one actuals predicts. There we go. Let it auto finish for me. And we can just go ahead and type in correlation accuracy and see what that looks like. Uh, so we have an actual 1, a predicts 0.8. And you can see how they kind of correlate. We have the 0.82. And if you remember, um, from before, we're looking for um, either 0.00, 1.01, 1.05, .1. That is the p-value, which we're not looking at here. This is not based on the p-value. This is based on we looked at earlier with correlation that the closer to one, the closer the values are. And so 0.82 gets upwards of one. So you see that there's a normal, pretty much a correlation in here. But we want to dig a little deeper because this doesn't really tell us how accurate it is.

For that, we're going to use another tool. For that, uh, we're going to calculate the min max accuracy, and mate, and the min max accuracy equals the mean value of the minimal actuals and predicts over the max value of the actuals and predicts. And then the mean absolute percentage error, or MAPE as it's called, equals the mean of the absolutes predicted minus actuals over the actuals. That's a lot to follow as far as uh, remembering all these different math and theory behind it. Cool thing though is it's all done pretty much for you. So you can compute all the error matrices in one go using the regress eval function in the DMWR package. Since this is the first time I've used this install, I'm going to have to install the DMWR package. So we can actually install it two different ways. If you remember, we talked about installing packages earlier. We can go under tools and install packages. And I certainly could type in DMWR. It even comes up and lets me know that's one of the main packages up there. Well, that's a tool of the R Studio setup. You can also install this using the console. We simply type in install. uh, you can even see here it comes up install packages and then DMWR. And when I hit enter, it should do the same thing as the other one does. It goes through and installs a DMWR package. And it takes just a moment to zip through all the different package setup. And then the format for DMWR is DMWR. We're going to go colon colon. We are specifically looking at the regression. Eval. And then we're going to do actuals_predicts. And we'll go ahead and put in the individual columns on this one. So we have actuals. Love the auto typing. And then again actuals predicts. And this time we want to do it against the predicted. And we'll go ahead and hit enter on here. And you'll see in here it comes up with our MAE, MSE, RMSSE, and the MATE values.

So, we looked at the linear regression model. Let's go ahead and take a look at the decision trees. So, let's go ahead and talk about another algorithm: Decision tree. Decision tree is a tree-shaped algorithm used to determine a course of action. Each branch of the tree represents a possible decision, occurrence, or reaction. It is a tree which helps us by assisting us in decision-making. Let's look at the basic terminologies to understand decision trees. We have a root node. We have a splitting. We have decision node, decision node. And then those split into terminal node, decision node, terminal node, terminal node. And each decision node continues to split until it ends in a terminal node. Note A is the parent node of B and C. Also note we call the terminal nodes leaf nodes. You can also see that we have a branch or a subtree. So when you have a split, everything under that split under one side is called a subtree. You cannot build your decision tree without knowing entropy and information gain.

Entropy. Entropy is the measure of randomness or impurity in the data set. So we have here a bunch of fruit, and you can see we have it looks like apples, oranges, and bananas. Uh, it's very chaotic. So it's a very random data set. It has a very high entropy. And if we take out one group, let's say we take out the apples, it's a little less random. Now we only have bananas and oranges. So it's a less random data set, and we have a lower amount of entropy. Entropy is a measure of randomness or impurity in the data set. So when you have a homogeneous data set, we will have entropy equals zero. And here you can see it's all oranges. An equally divided data set will have an entropy equal to one. So here we have um, half or um, what is it looks like? A couple bananas, a couple oranges, and a couple apples. So everything is equal. So we have an entropy of one on there since there's two of everything.

Information gain. It is a measure of decrease in entropy after the data set is split. Also known as entropy reduction. So we look over here and we have an entropy equals E1. And the information gain as we split the bananas out. Uh, the size becomes smaller, and you'll see that E1 is going to be greater than E2, or we measure E2 of the apples and oranges in this case. If you love your fruit, it's probably getting you hungry right about now. And then the information gain from our level two equals E2 minus E1, where E2 is greater than E3. As we come down here, we'll see that the E3 is now the third level. We split the oranges and apples out. And each time that entropy becomes less and less and less until, in this case, we have an entropy of zero since they're all homogeneous in their order. All oranges, all apples, all bananas.

So let's take a look at the use case decision tree and actually put this into code and see what that looks like as we go from theory to script. And to predict the class of flower based on the petal length and width using R. And you'll see here we have these beautiful irises, probably the most popular data set for beginning data analysis and statistics. Uh, we have the setosa, the versicolor, and the versicolor. Let's install the packages that will help us in the use case. Uh, so because we're doing decision tree, we have our rpart, rpart.plot, and then we have the library rpart and the library rpart.plot. Let's go ahead and take a look at that. And if you remember, we have two ways of installing it. We can do the uh, install packages. We did that earlier. Here we go. Install packages. Using my up arrow to get to it. Or we can go up to tools, install packages, and I can do rpart and rpart.plot, and then go ahead and hit the install. Packages that will be updated are currently loaded. Oh, looks like some of them already loaded. That's fine. We'll just go ahead and update those. And once we have those updated, let's take a look and see what we have going on here. Oh, still loading. Okay, let's go ahead and type in library rpart and library rpart.plot. So, we want to bring it into our console where we're working in. So, up, and now we just download the packages, and now we want those libraries available to what we're doing. So the iris, um, although it's built into the R package download on the basic download, we actually have to install the data or bring the data in. So we're going to do the data iris and then we're going to do the str iris. And let's just flip back on over to our R studio and set up our data iris, and then we're going to do str. Before we do that, let's do head. Head iris. There we go. And you can see the head has sepal length, sepal width, petal length, petal width, species. In this case, the top part's all setosa, and each row gets a number. And then we did the str iris, and that comes up with information. And if we flip back to our slide, you can see here the structure of the database under str is it's a data frame, as 150 objects, five variables each. So each one has five variables. All features are numerical except species, which is categorical, as it is our target variable which we want to predict. So we want to predict whether it's going to be a versicolor, versicolor, setosa. So you can see here we want to predict all the features are numerical except species, which is categorical. It is our target variable which we want to predict. And you remember that's a setosa, the versicolor, the virginica, the three different categories we saw earlier in the beautiful pictures of the flower.

And so we're going to go ahead and use the set.seed to decide the starting point used in the generation of sequence of random numbers. Remember, we set the seed so that if we ever want to reproduce what we're doing, it will reproduce the same thing each time because it's using the same randomizer seed. It might bring slightly different results, but depending on what you need it for. And then we're going to generate random numbers using runif, and the question is why do we want to um, create random numbers on here? That's kind of an unusual thing to do uh, for inrows iris. Let's go back and just take a quick look here. Actually, we want to go back to our R studio. And you're going to notice when I did the head of iris, what did I have? I had setosa, setosa, setosa, setosa, setosa. So, the data is organized by species. And it's already grouped them together in the data set. Well, we want to randomize that so that doesn't affect our output. And then we'll go ahead and take and create our iris_ran where we recreate the data frame based on a random order instead of setosa all being grouped together. Let's go ahead and put that into our R studio and set.seed, and we'll match their number. It's 9850. This way we'll have identical results to what they're doing on the slides. Then we're going to set the variable J equal to runif and base this on nrow of iris. And finally, we're going to create iris_ran. And we're going to assign our variable. We're creating iris_ran. And we'll set that equal to we're going to do that the original database. We're going to set that equal to iris, and we need brackets because we're assigning the variables and we want to change the order. So there's an actual u command order we can do. It's going to be order J. So this means all the rows in the order that J is. So we just randomized all those rows. And then we want to keep all the columns. We'll just put a comma and we'll leave it blank for the columns. So now iris_ran will have some randomized rows coming in. And if we do head, let's just do this real quick. Iris_ran, we can see there's no longer setosa, setosa. We now have uh, virginica, setosa, versicolor, setosa. And you can see the row numbers at the beginning. We have 103, 20, 63, 17. So they're randomized on there pretty well.

Now that we've done a little data manipulation, we can go ahead and build our model using rpart as follows. And one of the things I want you to notice here is we're going to build our model for our rpart on the first 100 because our data equals iris_ran[1:100, ]. There's 150 objects. So we're keeping a third of them off to the side. That way we have our training set and our test set already pre-built because we randomized the rows going in from the beginning. And then once we've done that, we want to go ahead and print our model out and take a look at it. Let's go ahead and put this into our R studio and take a look. And so we have our model. We're going to call it model. That's our variable name that we're giving it. We're going to assign that rpart, and in brackets we'll start with species, and we want to correlate species to all the other columns. So how do we do that? And the shorthand is just to put a period there, and then we're going to put a comma, and we want to set our data equal to iris_ran. That's the data we put in there. And remember I said we're going to do just the first 100 rows. So we can save the last 50 for um, testing. So we can do that simply by doing rows 1 to 100. So in R you start with one for your rows. And because we want all the columns, we just put a comma, and that denotes that we're going to use all the columns in here. And in the um, rpart there's different methodologies um, we're using the method = class. And you realize that we're doing a classification between the three different types of flowers. And so this is what we want to go ahead and set up our model with. And when we run that, we've now created a nice model for us. We can go and just type in model. Hit enter. And you'll see it'll print out a bunch of information about our model. So let us know what's going on in there. And let's just break that down and see what that looks like. So we have our target, our predictors, that's species, and it's going to relate to we used a period to denote all the columns. We're going to take the first 100 rows to train the data. And we have the defining method to be used as classification. That's what the method = class is. And what we want to do next is we're going to do a plot and take a look and see what this actually looks like. Um, a little visual here, which is nice. And we simply do that with rpart for the model. And let's break this down a little bit and just see what that looks like in the R setup. And we do this rpart.plot. A little bit of a mouthful there sometimes. That's part of the rpart package or module. So when you imported this in, it also included its own plotting format. And we're going to send it our variable model. And there are actually a number of types of plots listed for rpart, but we're just going to do type = 4, fallen = t, and extra = 104. Let's go ahead and hit enter on there. And you'll see right over here on the right-hand side, it goes into our plots. Let me just enlarge that. We have our versicolor, our setosa, versicolor, virginica. And let's go ahead and break that down. As you can see down here, it represents the tree as it splits. It looks at it and says, "Hey, if the petal.length is less than 2.6, it's going to be a setosa." Um, so 34% of them fall into that category. And if the petal length is greater than 2.6, then looks at that and says, "Okay, now I have uh, three different variables. We're going to split it again. Petal width is less than 1.7, it's going to be a versicolor, and if it's greater than 1.7, it's a virginica." And so you can see here we it's a very simple tree of forms. Um, it's only got a couple levels. You can see a branch off to the right. Very nice depiction of what's going on as far as the model now that it's created. Uh, but once we create the model, we need to figure out is this model any good? So let me go back over here. Let's go back to our slides.

And the first thing we're going to do is we're going to take our test data. Remember we saved 50 rows for the end to go ahead and run a predict on there. So let's go ahead and run that prediction. So we'll call it model_predict, and we're going to hit a wrong button there, to predict, and we're going to what are we going to predict? We're going to predict our initial model, and we need to send the data. So data = and we don't actually have to put the data part in. We just put in iris_ran, and then we want to do just the last uh, 50 rows. And we denote that by 101:150, and then comma, and we'll leave that part blank since we want all the columns to show up. And then finally, this is a classification. So we need to let it know that type = class. There we go. And let's just see what that looks like. Model_predict. And so this is what the data is put out as is it says uh, whatever this is. 131 is a virginica, number 10 is a setosa, line number 95 is a versicolor, and so on all the way through. What we really want to know is how good our model is. So we're going to go ahead and install a couple more packages. Uh, we're going to install the packages required to use the confusion matrix. And to do that, we're going to install caret and e1071. And then of course we want to set them to our project by using the library command. And so we can simply install.packages. And we want caret. And u we'll go ahead and just do that right there. There's caret installed. And we went ahead did the install for packages e1071. You see it goes through there. It's a really quick install compared to the caret. And then once we've installed the packages, we need to load it into our workspace. So there's our library caret. Oops, I should load it in there. There it goes. And once that's loaded, we'll go ahead and also load the uh, or set the library to the e1071 so that we have both those libraries available to us. And here we go. Library. And then this one is e1071. And now these two libraries are fully available to us. And then we're going to use these tools to create the confusion matrix. And this one, confusionMatrix, this is the iris_ran. And then we'll go ahead and put the brackets here. And we're only going to do 101:150. And then we only want the first five columns, not the last five columns, the first five columns. So 5 in there. And we'll set that to our model_predict. And then when we hit enter, it's going ahead and just print out all this information. So right here, we're looking at our confusion matrix to see how everything kind of balances out. Let's go over that and see a little closer what that actually means. So the first part we see is a reference. And in here we have our prediction because that's what we call the column if you remember correctly. Uh, and we have it for setosa versus virginica, and then we have setosa versus versicolor, virginica going down each side. And the way you read this is if you look at this, we predicted uh, 16 out of 16 setosa, and then uh, versicolor we predicted 13 of the versicolor, and then two of those we predicted as virginica. So we were wrong on two of those. And with the virginica we predicted 17 correct and we had two wrong that were versicolor. So we look at the overall statistics on here and we have an accuracy of 0.92 and a 95% CI. That's pretty good as we're going through here. Those are good deals on that, and there's a lot more information as you go down this. The one thing I want to remind you on this as we're looking at these predictions, and this is also true for the linear regression model we looked at earlier. When we talk about the accuracy, that is the accuracy on this data, and we say that as a programmer to make that a very clear distinction because if you're in data analysis you should cringe if I say ah, this is a 95% accurate model without that put in there because bad data in, bad data out. We won't go into detail right now on that, but that is a very important to note that whenever you quote any of the accuracies on here or you discuss any of these values that come up.

So a quick summary: Introduction to R. We went over the fact that you can import data very easily from a table, an Excel spreadsheet, a CSV file, and you can also export it. We went over the regression line of best fit and how that is those formulas work to slowly adjust that line until it fits with the least amount of error. We discussed correlation analysis in R where we're correlating data between two objects. And then we talked about the decision tree algorithm and model building and validation. Hello and welcome to this session on data science with Python. So what are we going to do in this session?

We will talk about what is data science and some of the basics of Python. As you may be aware, data science uses either Python or R as some of the tools and programming languages. So this session focuses on Python. So we will talk about the basics of Python, why to learn Python, how to install Python, and then we will talk about some of the important libraries which are required for data analysis. Then we will go into a little bit of detail about exploratory data analysis, and we will take an example there of uh loan prediction. We will see a little bit about data wrangling using pandas, which is one of the libraries of Python, and then we will end up with a small predictive model, or logistic regression model, which is a part of machine learning.

And in case you need more details about machine learning in this session, we will probably not go into the details of machine learning. There is a separate session you might want to look at. There is a separate video on machine learning. You might want to take a look at that if you need more details. Here we are just giving a quick overview to see how we can use the scikit-learn library. So we will not go into a lot of details about the basics of machine learning. All right. So with that, let's move forward.

So what is data science? Data science is about finding insights from the data. So if there is a lot of data, if you have sufficient data, how to analyze and find some insights out of it? This is what data science is all about. A couple of examples here: Customer prediction. Now let's say you have a customer base and you want to find out who are most likely to buy your product. So you can use their past behavior. You can probably develop a model and try to predict who are the people, out of the thousand leads or uh uh potential customers, who will actually buy. So there may be some patterns that you can use to predict.

Similarly, service planning. So, for example, you're running a restaurant and you want to know how many people will be coming or how many customers will be visiting your restaurant on a given day. Now, based on your historical data, you can build a model to predict that as well, so that there is no wastage of food and so on and so forth. So these are very quick and easy examples of how data science can be used in business.

Now let's uh talk about Python. For doing data science, we need some kind of a programming language or a tool and so on. So this session will be about Python. There are other tools like, for example, R, and we will probably do a separate video on that, but this session is on Python, and you must have already heard that Python is really becoming very popular. Everybody is talking about Python, not only in data science, but in IoT and AI and many other places. So it's a very popular, it's getting very popular. So if you are not yet familiar with Python, this may be a good time to get started with it.

So why do we want to use Python? So basically, Python is used as a programming language for data science because it has some rich tools from a mathematics and from a statistical perspective; it has some rich tools. So that is one of the reasons why we use Python. And if you see some of the trends, if you're probably tracking some of the trends, you will see that over the last few years, Python has become the programming language of choice, especially for data science. SAS was earlier one of the most popular tools, but now increasingly Python is being used for doing data science and, of course, as well as R. One of the reasons, of course, is that Python and R are open-source compared to SAS, which is a commercial product. So that could definitely be one explanation. But beyond that, I think it is the ease of understanding this language, the ease of using this language, which is also making it very popular, in addition to the availability of fantastic libraries for performing data science.

What are the other factors? There are speed, then there are the availability of a number of packages, and then, of course, the design goal. All right. So what are each of these design goals? Primarily, the syntax rules in Python are relatively intuitive and easy to understand. Thereby, it helps in building applications with concise and readable code base. So with a few lines of code, you can really achieve a lot of stuff. And then there are a lot of packages that are available that have been developed by other people, which can be reused. So we don't have to reinvent the wheel. And last but not least, the speed. So Python is a relatively faster language. Of course, it is not as fast as, let's say, C or C++, but then relatively it is still faster. So these are the three factors which make Python the programming language of choice.

So if you want to get started with Python, the first thing obviously is to install Python. So there is some documentation. There are some steps that you need to follow. So we will try to briefly touch upon that; otherwise, of course, there are a lot of materials available on how to install Python and so on. You can always look around, but this is one of—again, there are different ways in which you can also install Python. So we will use the Anaconda path. There is a packaging tool called Anaconda. So we will use uh that path. You can also directly install Python, but in our session we will use the Anaconda route.

So the first thing you need to do is download Anaconda, and this is the path for that, and uh once you click on this, you will come to a page somewhat like this, and download—you can do the corresponding download based on whether you have a Windows or Ubuntu. There is a also a download possible for, or package available for, Ubuntu if you're doing something on Ubuntu. So based on which operating system—in fact, this page will automatically detect which operating system you are having and it will actually suggest. So, for example, you see here, if you're running Mac OS, then it will automatically detect that you have a Mac, and the corresponding installers will be displayed here. Similarly, if you are on some flavor of Linux like Ubuntu or any other, then you will get the corresponding download links here. And then beyond that, you can also select which version of Python you want to install. Of course, the latest version is in the 3.x uh range. At the time of recording this, 3.6 is uh one of the latest versions, but some of you may want to do or start with the earlier version, which is Python 2.7, 2.x. And you can download that as well. If you don't have anything installed, then my suggestion is to start with Python 3.6. All right.

So once you do that, you will be able to install Python and you will be able to run Jupyter Notebook. Okay. So now that you know how to install Python, and if you have installed Python, let's take a look at what are the various libraries that are available. So Python is a very easy language to learn, and there are some basic stuff that you can do, for example, adding or printing a hello world statement and so on, without importing any specific libraries. But if you want to perform data analysis, you need to include or import some specific libraries. So we are going to talk about those as we move forward.

So pandas, for example, is used for structured data operations. So if you, let's say, are performing something on a CSV file, you import a CSV file, create a data frame, and then you can do a lot of stuffs like uh data munching and data preparation before you do any other stuff like, for example, machine learning or so on. So that's uh pandas. SciPy, as the name suggests, it is kind of—it provides more scientific capabilities like, for example, it has linear algebra, it has Fourier transform, and so on and so forth. Then you have NumPy, which is a very powerful library for performing n-dimensional or creating n-dimensional arrays, and it also has some of the stuff that is there in SciPy, like, for example, linear algebra and Fourier transform and so on and so forth. Then you have Matplotlib, which is primarily for visualization purposes. It has again very powerful features for visualizing your data, for doing the initial what is known as exploratory data analysis, for doing univariate analysis, bivariate analysis. So this is extremely useful for visualizing the data, and then scikit-learn is used for performing all the machine learning activities. If you want to do anything like uh linear regression, classification, or any of this stuff, then the scikit-learn library will be extremely helpful.

In addition to that, there are a few other libraries, for example, NetworkX and iGraph. Then, of course, a very important one is TensorFlow. So if you are interested in doing some deep learning or AI-related stuff, then it would be a good idea to learn about TensorFlow, and TensorFlow is one of the libraries. There is a separate video on TensorFlow. You can look for that, and this is one of the libraries created by Google, an open-source library. So once you are familiar with machine learning, data analysis, machine learning, then that may be the next step to go to deep learning and AI. So that's where TensorFlow will be used. Then you have Beautiful Soup, which is primarily used for web scraping, and then you take that data and then analyze and so on. Then the OS library is a very common library; as the name suggests, it is for operating systems. So if you want to do something on creating directories or folders and things like that, that's when you would use uh OS. All right.

So moving on, let's talk in a little bit more detail about each of these libraries. So SciPy, as the name suggests, is a scientific library, and it very specifically has some special functions for integration and for ordinary differential equations. So as you can see, these are mathematical operations or mathematical functions. So these are readily available in this library, and it has linear algebra modules and it is built on top of NumPy. So we will see what is there in NumPy. So this is a—again, as the name suggests, the Num comes from numbers. So it is a mathematical library, and one of its key features is the availability of an n-dimensional array object. Uh, that is a very powerful object, and we will see how to use this, and uh then, of course, you can create other, let's say, objects and so on, and uh it has tools for integrating with C++ and also for running code, and then it, of course, also has linear algebra and Fourier transformation and so on—all these scientific capabilities. Okay, what else? Pandas is another very powerful library, primarily for data manipulation. So if you're importing any files, you will want to create it like a table. So you will create what is known as data frames. These are very powerful data structures that are used in Python programming. So the pandas library provides uh this capability, and once you import a data, import the data into a data frame, you can pretty much do whatever you're doing like in a regular database. People who are coming from a a database background or SQL background would really like this because it is very—they will feel very much at home because it feels like you're using, you're viewing a table or using a table, and you can do a lot of stuff using the pandas library.

Now there are two important terms or components in pandas: series and the data frame. I was just talking about the data frame. So let's take a look at what are series and uh what is a data frame. So within pandas, we have series and data frame. So series is—primarily, some of you may also be knowing this as, let's say, an array. So it's a one-dimensional structure, data structure if you will. So in some other languages, we may call it as an array, or maybe some others probably an equivalent of a list in R perhaps. I'm not very sure on that aspect, but yes. So this is like a one-dimensional storage of information. So that is what is series. Whereas data frame is like a table. So you have a two-dimensional structure. You have rows and you have columns. And this is very—people, as I said, who are familiar with SQL and databases will be able to relate to this very quickly. So you have like a table, you have rows and columns, and then you can manipulate the data.

So if you want to create a series, this is how you would uh create a code snippet. And as you can see, the programming in Python is very simple. There are no major overheads. You just need to import some libraries which are essential and then start creating objects. So you don't have to do additional declaration of variables and things like that. So that is, I think, one key difference between Python and other programming languages. And what does this series contain? It has to contain these numbers: 6, 3, 4, 6, and x is my object consisting of the series. So if you display—if you just say x, it will display the contents of x, and you will see here that it creates a default index. Then you have data frame. So if you want to create a data frame, as you can see, the series is like a one-dimensional structure. There is just like a row, one row of items. Whereas a data frame looks somewhat like this. It is a two-dimensional structure. So you have columns in one dimension, and then you have rows in the other dimension.

How do you create a data frame? You need to create—you need to rather import pandas, and then you import—in this case, we are basically creating our own data. So that's the reason we are importing NumPy, which is one of the libraries we just referred to a little bit uh before. So we are using one of the functionalities within NumPy to create some random numbers; otherwise, this is not really mandatory. You probably will be importing the data from outside, maybe some CSV file, and import into the data frame. So that's what we're doing here. So in this case, we are creating our own test data; that's the reason we are importing NumPy as np, and then I create a data frame saying pd.DataFrame. So this is the keyword here. Similarly, here in this case, while creating series, we said pd.series, and then you pass the values. Similarly, here you're saying pd.DataFrame. Now, in order to create the data frame, it needs the values in each of these cells. What are the values in in the rows and what are the values in the column. So that in our example, we are providing, using this random number generator. So np.random is like a class or a method that is available in NumPy, and then you are saying, okay, generate some random numbers in the form of a 4x3 matrix or 4x3 data frame. The four here indicates the number of rows, and the three here indicates the number of columns. So these are the columns: 0, 1, 2 are the columns. And these are the rows here: 0, this is one, this is two, this is three. Okay. And once again, it will—when you display df, it will give us a default index. There are ways to omit that, but at this point we will just keep it simple. So it will display the default index and then the actual values in each of these rows and columns. So this is the way you create a data frame.

So now that we have learned some of the basics of pandas, let's take a quick look at how we use this in real life. So let's assume we have a situation where we have some customer data, and we want to kind of predict whether the customer's loan will be approved or not. So we have some historical data about the loans and about the customers, and using that we will try to come up with a way to maybe predict whether a loan will be approved or not. So let's see how we can do that. So this is a part of exploratory analysis. So we will first start with exploratory analysis. We will try to see how the data is looking. So what kind of data? So we will—of course, I'll take you into the Jupyter Notebook and give you a quick live demo. But before that, let's quickly walk through some of the pieces of this program in slides, and then I will take you actually into the actual code and do a demo of that.

So the Python program structure looks somewhat like this. The first step is to import all your required libraries. Now, of course, it is not necessary that you have to import all your libraries right at the top of the code, but it is a good practice. So you—if you know you are going to need a certain set of libraries, it may be a good idea to put—from a readability perspective, it's a good practice to put all the libraries that you're importing at the beginning of your code. However, it is not mandatory. So in the middle of the code somewhere, if you feel that you need a particular library, you can import that library and then start using it in the middle of the code. So that's also perfectly fine. It will not give any errors or anything. However, as I said, it's not such a good practice. So we will import all the required libraries. In this case, we are importing pandas, NumPy, and Matplotlib. And in addition, if we include this piece of code, `%matplotlib inline`, what will happen is all the graphs that we are going to create, the visualizations that we are going to create, will be displayed within the notebook. So if you want to have that kind of a provision, you need to have this line. So it's always a good idea—when you're starting off, I think it's a good idea to just include this line so that your graphs are shown inline. Okay. So these are the four—we will start with these four lines of code.

Then the next step is to import your data. So in our case, there is a training data for loans by the name `loanp_train.csv`, and we are reading this data. So in this case, you see here, unlike the previous example where we created a data frame with some data that we created ourselves, here we are actually creating a data frame using some external data, and it's the method is very, very straightforward. So you use the `read_csv` method, and it is a very intuitive function name, and you say `pd.read_csv` and give the path of the file, CSV file; that's about it, and then that is read into the data frame `df`. This can be any name; we are calling it `df`; you can call it xyz, anything, there a name, just the name of the object. So `head` is one of the methods within the data frame, and it will give us the first five. So this is just to take a quick look. Now you have imported the data. You want to initially have a quick look at how your data is looking. What are the values in some of the columns and so on and so forth, right? So typically you would do a `df.head` to get a sample of, let's say, the first few lines of your data. So that's what has happened here. So it displays the first few lines, and then you can see what are the columns within that and uh what are the values in each of these cells and so on and so forth. You can also—typically you would like to see if there are any null values, or are there any—is the data, for whatever reason, is um invalid or looking dirty for whatever reason, some unnecessary character. So this will give a quick view of that. So in this case, pretty much everything looks okay.

Then the next step is to understand the data a little bit overall for each of the columns. What—what is the information? So the `describe` function will basically give us a summary of the data. What else can we do? Pandas also allows us to visualize the data, and this is more like a part of what we call it as univariate analysis. That means each and every column you can take and do some plots and uh visualization to understand the data in each of the columns. So, for example, here the loan amount column we can take, and then the `hist`—basically, the `hist` method will create a histogram. So you take all the values from one column, which is loan amount, and you create a histogram to see how the data is distributed. Right. So that's what is happening here. And as you can see, there are some extreme values. So this is again to identify—do we have to do some data preparation? Because if the data is in a completely haphazard way, the analysis may be difficult. So we—these are the initial—or exploratory data analysis is primarily done to understand that and see if we need to do some data preparation before we get into the other steps like machine learning and...

Statistical modeling and so on. So, in this case, we will see that, yeah, by plotting this histogram, we see that there are some extreme values. So, there are some values; a lot of it is around the 100 range, but there is also something—one or two observations—in the 700 range. So, it's pretty scattered in that sense, or they're not really scattered—distributedly scattered—but it is randomly scattered. So, the range is really huge. So, what can we do about this? So, there are some steps that we need to do—normalization and so on. So, we'll see that in a bit. So, this is for one of the columns. Let's take another column, which is applicant income. Similar kind of similar situation. You have most of your observations in this range, but there are also some which are far off from where most of the observations are. So, this is also pretty—this also has some extreme values. So, we'll have to see what can be done. Credit history is the binary value. So, some people have a zero value, and some will have a credit history of one. This is just like a flag. So, this basically is telling us how many people have one and how many people have zero. So, looks like a majority of them have a value of one, and a few—about 100 of them—have a value of zero. Okay. What else can we do? So, we now understood a little bit about the data. So, we need to do some data wrangling or data munching and uh see if we can some bring in some kind of normalization of all this data, and uh we will kind of try to understand what is data wrangling and before we actually go into it.

Okay. So, data wrangling is nothing but a process of cleaning the data. If, let's say, there are—there are multiple things that can happen in this particular example; there were no missing values, but typically when you get some data, very often it will so happen that a lot of values are missing—either they are null values or there are a lot of zeros. Now, you cannot use such data as it is to perform some, let's say, predictive analysis or perform some machine learning activities and so on. So, that is one part of it. So, you need to clean the data. The other is unifying the data. Now, if these ranges of this data are very huge—some of them are going from some columns are going from 0 to 100,000, and some columns are just between 10 to 20 and so on—these will affect the accuracy of the analysis. So, we need to do some kind of unifying the data and so on. So, that is what wrangling—data wrangling—is all about. So, before we actually perform any analysis, we need to bring the data—so, do some kind of a shape—so that we can perform additional analysis—actual analysis—on this and get some insights.

Now, how do we deal with missing values? This is a very common issue when we take data or when we get data from the business—when a data scientist gets data from the business. So, we should never assume that all our data will be clean and all the values filled up and so on, because in real life, very often there will be—the data will be dirty. So, data wrangling is the process where you kind of clean up the data. First of all, identify whether the data is dirty and then clean up. So, how do we find some data is missing? So, there are a few ways—you can write a small piece of code which will identify if, for a given column or for a given row, any of the observations are null primarily. So, this line of code, for example, is doing that—it is trying to identify how many null values or missing values are there for each of the columns. So, this is a lambda function, and what we are saying is: find out if a value is null, and then you add all of them—how many observations are there where this particular column is null. So, it does that for all the columns. So, here you will see that, for loan ID, obviously it's an ID, so there are no null values or missing values. Gender has about 13 observations where the values are missing. Similarly, marital status has three, and so on and so forth. So, we'll see here, for example, loan amount has 21 observations where the values are missing; loan amount term has 14 observations, and so on. So, we'll see how to handle this missing values. So, there are multiple ways in which you can handle missing values. If the number of observations are very small compared to the total number of observations, then sometimes one of the easy ways is to completely remove that data. So, or delete that record, exclude that record. So, that is one way of doing it. So, if there are, let's say, a million records and maybe 10 records are having missing values, it may not be worth doing something to fill up those values; it may be better off to get rid of those observations. Right? So, that is—the missing values are proportionately very small. But if there are relatively large numbers of missing values, if you exclude those observations, then your accuracy may not be that very good. So, there—the other way of doing it is—we can take a mean value of, for a particular column, and fill up wherever there are missing values—fill up those observations or cells with the mean value. So, that way, what happens is you don't give some value which is too high or too low, and it somehow fits within the range of the observations that we are seeing. So, this is one technique. Again, there are—it can be case to case, and you may have to take a call based on your specific situation, but these are some of the common methods. If you see in the previous case, loan amount had 21, and now we went ahead and filled all of those with the mean value. So, now there are zero with missing values.

Okay. So, this is one part of a data wrangling activity. What else you can do? You can also check what are the types of the data. So, df.d D types will give us what are the various data types. So, all right, so you can also perform some basic mathematical observations. We have already seen that mean we found out. So, similarly, if you do call the mean method for the data frame object, it will actually perform or display or calculate the mean for pretty much all the numerical columns that are available in this. Right? So, for example, here applicant income, co-applicant income, and all these are numerical values. So, it will display the mean values of all of those. Now, another thing that you can do is you can actually also combine data frames. So, let's say you import data from one CSV file into one data frame and another CSV file into another data frame, and uh then you want to merge these because you want to do an analysis on the entire data. Okay. One example could be that you have data in the form of CSV files—one CSV file for each month of the year: January, February, March; each of these are in a different—so you can import them into, let's say, 12 data frames, and then you can merge them together as a single data frame, and then you perform your analysis on the entire data frame or the entire data for the year. So, that is one example. So, how do we do that? This is how we do. Again, in this case, we are not importing any data; we are just creating some random values using some random values. So, let's assume I have a a data frame which is by the name one, and I assign some random values here which is a 5x4 format. So, there are five rows and four columns. And this is how my data frame one looks. And then I create another data frame which is data frame 2—again, random numbers of the format 5x4. And I have something like this. Now, I want to combine these two. How do I combine these two? I can use the concatenate or concat method, and I can combine these two. So, pd.concat, and it takes the the data frames one and two. If you have more of them, you can provide them, and it will just simply add all of them—merge all of them or concatenate—whatever you call—whichever term you call it—will. So, of course, we have to make sure that the structure remains the same. Like I said, this could be, let's say, sales data coming for 12 different months, but each of the files has the same structure. So, now you can combine all of them, merge all of them using the concat method. If we have, let's say, structure is not identical. Then what will happen? Let's say we have these two data frames. One has a column by the name key and the second column is lal, and a second data frame which has a column by the name key. But the second column by the name RV valval. So, you see here the structure is not identical. So, you can still combine them, but then the way they get combined or merged is somewhat like this. So, it takes the key as a common parameter between them—some common column has to be there; otherwise, this will not work. And then we have to use merge instead of concatenate. And when we do a merge, then we get—the result will be in this format. What it does is it uses the key as a common thread between them, and then it kind of populates the values accordingly. So, if you see here, the first one had fu and bar for key, and then it had l values of 1 and 2, right? So, if we go back, fu and bar had 1 and 2 l values. So, that's what we see here—1 and 2. Whereas in the right data frame, we had fu, bar, and bar as a second time, and then r values are 3, 4, and 5. So, what it has done—for fu, it has put—for the existing—right—for fu is already existing because it has come from left. So, it will just put the value of r val here, which is 3. Similarly, it will put 4 here because for bar, if you go back for bar, it is—the value is 4. And since it has one more value of bar, it will go and add this 5 as well. The only thing here is that this one had, for example, left had only two values and only one value for bar. But since we are appending or merging, and there are two key values with bar, therefore it will kind of repeat the value of lv val here. So, that's what we are seeing in this case, right? So, L value appears twice; the number two appears twice, but that is because R value—there are two of them. Okay.

All right. So, that is how—when you don't have identical structure—that is how you merge. Now, we will talk a little bit about scikit-learn. So, scikit-learn is a library which is used for doing machine learning or for performing machine learning activities. So, if you want to do linear regression, logistic regression, and so on, there are easily usable APIs that you can call, and uh that's um the advantage of scikit-learn, and it provides a bunch of algorithms. So, I think that is the good part about this uh library. So, if you want to use scikit-learn, obviously you need to import these modules, and also there are some submodules you may have to import based on what you're trying to use. Like, for example, if we know—if we want to use logistic regression—again, people who are probably not very familiar with machine learning, there is a separate module for machine learning. You may want to take a look at that. But we will just touch upon the basics here. So, machine learning has some algorithms like linear regression, logistic regression, and uh random forest classification and so on. So, that is what we are talking about here. So, those algorithms are available, and uh when—if you want to use some of them—you need to import them and uh from the scikit-learn library. So, scikit-learn is the top-level library which is basically sklarn, right, and then it has a kind of subparts in it. You need to import those based on what exactly you will be or which algorithm you will be using.

So, let's take an example as we move, and we will see that whenever we perform some machine learning activity—those of you who are familiar with machine learning will already know this—we split our labeled data into two parts—training and test. Now, there are multiple ways of splitting this data. How do we—either some people do it like 50/50; some people do it 80/20, which is training is 80 and tested is 20, and so on. So, it is individual preference; there are no hard and fast rules. By and large, we have seen that the training data set is larger than the test data set. And again, we will probably not go into the details of why do we do this at this point, but that's one of the steps in machine learning. So, scikit-learn offers a readily available method to do this, which is train_test_split. All right. So, in this example, let's say we are taking the values x and y are our values. X is the independent variables, and y is our dependent variable. Okay. And we are using these two. And then I want to split this into train and test data. So, what do we do? We import the train_test_split submodule from within scikit-learn, which is sklarn. Right? So, within that, we import train_test_split, and then you call this train_test_split method or function or whatever you call it and pass the data. So, x is the all the values of the independent variables, and y is our labels. So, you pass X and Y, and then you specify what should be your size of the test data. So, only one you need to specify. So, if you say test_size is 0.25, it is understood that train_size will be 75. So, you're telling what should be the ratio of the split. So, technically, it doesn't—nothing prevents you from giving whatever you like here. So, you can give test as 80 and train as 20. So, whichever way, but then the normal practices—you will have the training data set would be larger than the test data set, and typically it would be 80/20, 75/25, or 65/35, something like that, right? So, that is the second parameter, and this is just to say that you know the data has to be randomly split. So, it shouldn't be like you take the first 75% and put it in training and then the next 25% and put it in test. So, that—so such a thing shouldn't happen. So, we first set the state—random_state—so that the the splitting is done in a very random way. So, they are randomly picked up the data and then put it into training and test, and then this results in these four data frames. So, X_train and X_test and Y_train and Y_test. Okay. So, that is basically the result it will return.

Now that the splitting is done, let's see how to implement or execute logistic regression. So, in logistic regression, what we try to do is try to develop a model which will classify the data. Logistic regression is uh an algorithm for supervised learning for performing classification. So, logistic regression is for classification, and usually it is binary classification. So, binary classification means there are two classes. So, either like a yes/no or, for example, customer will buy or will not buy. So, that is a binary classification. So, that's where we use logistic regression. So, let's take a look at the code—how to implement something like that using scikit-learn. So, the first thing is to import this logistic regression submodule or subclass, whatever you call it, and then create an instance of that. So, our object is classifier. So, we are creating an object by the name—this is a name, by the way; you can give any name. In our case, we are saying classifier. We say classifier = LogisticRegression. So, we are creating an instance of the logistic regression variable or class or whatever. Okay. And you can pass a variable or a parameter, rather, which is the random_state = 0. And uh once you create the object, which in our case is named classifier, you can then train the object by calling the method fit. So, this is important to note. We don't call any—there is no method like train here, but we call what is known as—there is a method called fit. So, you are basically—by calling the fit method—you are training this model, and uh in order to train the model, you need to pass the training data set. So, X_train is your independent variables—the set of independent variables—and Y_train is your dependent variable or the label. So, you pass both of these and call the fit function or fit method, which will actually result in the training of this model—classifier. Now, this is basically showing what are the possible parameters that can be passed or initiated when we are calling the logistic or the instance of logistic regression. So, this is—but you can also look up the help file if you have installed Python. So, some of these are very intuitive, but some you may want to take a look at the details of what exactly they do.

All right. So, moving on, once we train the model by calling fit, then the next step is to test our model. So, this is where we will use the test data. You need to pay attention here. Here I'm calling. So, there are two things. One is—in order to test our data, we have to actually call what is known as the method known as predict. Right? So, here this is where—so, the training is done. Now is the time for inference, isn't it? So, we have the model. Now, we want to check whether our model is working correctly or not. So, what do you do? You have your test data. Remember, we split it—25% of our data was stored here, right? We split it into test and training, so that 25% of the data we pass to and call the method predict, so that the model will now predict the values for y, right? So, that's why here we are calling it as y_predict, and um if we display—here, as I said, this is the logistic regression, which is basically binary classification, so it gives us the results like yes or no in this particular case, and then you can—so this is what the model has predicted or model has classified, now, but we also know—we already have the labels for this. So, we need to compare with the existing labels with the known labels whether this classification is correct or not. So, that is where is the next step, which is basically calculating the accuracy and so on will come into play. Okay. So, in this case, the first thing—the most important thing to note is—we do the prediction using predict, and here we are passing X_test and not train. Right? In this case, we did x and y train. So, again, one more point to be noted here—in case of training, we will pass both the independent variables and also the dependent variables, because the system has to—internally—it has to verify that—is what is the training process—so what it will do—it will take the x values; it will try to come up with the y-value and compare with the actual y-value, right? So, that is what is the training method; so that's why we have to pass both x as well as y, whereas in case of predict, we don't pass both; we only pass because we are pretending as if this is the actual data. So, in actual data, you will not have the labels, isn't it? So, we are just passing the independent variables, and the system will then come up with the y values which we will then—okay, remember, we also know the actual values. So, we will compare this with the actual values, and we will find out whether how accurate the model is. So, how do we do that? We use what is known as a confusion matrix. So, this is also readily available in the Python library. So, we import this confusion_matrix, and some of you who already know machine learning will find this familiar, but those who are new to machine learning—this confusion matrix is nothing but this matrix—this kind of a matrix—which basically tells how many of them are correctly predicted and how many of them are incorrectly predicted. So, the—some of the characteristics—let's quickly spend some time on this confusion matrix itself. This—the total numbers out here—these are just the numbers—these are like number of observations. Then the accuracy is considered to be highest when the the numbers or the sum of the numbers across the diagonals is maximum. Okay. And the numbers outside of the diagonal should be minimum. So, which means that if this model was 100% accurate, then the sum of these two

there would have been only numbers in these two along the diagonal. This would have been zero, and this would have been zero. Okay. So that is like a 100% accurate model, that is very rare, but just that you are aware. So this is just to give an idea. Okay. All right.

So once you have the confusion matrix, you then try to calculate the accuracy, which is in a percentage. So there are two things that we can do from a confusion matrix, or that we can calculate from a confusion matrix. One is the accuracy, and the other is the precision. What is the accuracy? Accuracy is basically a measure of how many of the observations have been correctly predicted. Okay.

So let's say this is a little bit more detailed view of the confusion matrix. It looks very similar, like as we saw in this case, right? So this is a 2x2 matrix. That's what we are seeing here: 18, 27, 21, 0, 3. So 18, 27, 21, 0, 3. Now, but what are these values? That is what is kind of the labels are shown here in this. So there are altogether 150 observations. So as I said, the sum of all these four—right, 18 + 27 + 103 + 2—is equal to 150. That's the first thing we have to observe. The sum of all these values will be equal to the sum of test observations; number of test observations. We have 150 test observations because remember we had about 500. We split that into 25, 75. So that is why we have 150 here and I think 350 in the training data set. Okay. So that we get the numbers correct. So that's the first thing.

Now the next thing is let's take a look at the actual values. This view is the actual view. So there are actually, right, so in the actual data we have labels yes and no. So as per the actual data, there are 45 observations tagged as no, and similarly there are 105 observations that are tagged as yes or labeled as yes. Okay. Now I know for the first time when you're seeing this, it may be a little confusing, but just stay with me. Okay. So this is the actual part of it, and this side tells us the predicted part of it. So our model has predicted, and it has totally predicted 20 of them as no. Right? So that is what this is—totally 20 of them it has predicted as no, and it has predicted 130 of them as yes. Okay. I hope this part is clear.

So before we go into the middle part, let us first understand what exactly are these numbers. So actually tagged as no, there are 45 total; actually tagged as yes, there are 105; and predicted no, there are 20; predicted as yes, there are 130. This is the result from our model; okay, this is the result from our model, and this is the actual value, which we already know because this is our label data. That's the first thing. Now, now let us take a look at each of these individually; okay. Now what are the options we have once again? Okay. So now what is happening here, let us look at these these values. So this 18 says that these are actually tagged as no, and the model has also predicted as no, which means this is what is known as a true positive, right, or true negative, sorry, right? Which means that our model has predicted it correctly. It is negative because it says no. So and it has also predicted no. So it is known as what is known as true negative. Okay.

Now let's come to this side of it. That way we are talking about the diagonal. Remember I said most of the values should be in the diagonal. Okay. So that means these 18 are correctly tagged. They are labeled as no, and our model has predicted as no. So these are correctly tagged, and these are known as true negative. Okay. Similarly, if we come diagonally down, there are 103 observations which are labeled as yes. Actual value is yes, and our model is also predicted as yes. And these are known as true positive values. Positive because of this yes. Okay. Right. So what is important is this is true; this is also true. So we have to make sure that the maximum number of values are in the true section. Okay. True positive and true negative. That's the reason I said the sum along the diagonal should be maximum. Now let's say if our model was 100% accurate. This sum—in this case it is only 103 + 103 + 18, which is 121. But if our model was accurate, the sum of these two would have been 150. That means it's a perfect model. Okay. All right.

Now what else? Since we covered these two, let's also cover these two. So here this says that 27 of them were actually labeled no, but our model has predicted as yes. That means this is wrong. Right? Similarly, these are two of them where the actual value is yes, but our model has predicted as no. That means it's a wrong prediction. So you get the point. So therefore, along the diagonals are the correct values, whereas in other places it is all wrong values or wrong predictions. Okay.

Now how do we calculate accuracy from this information? So the way to calculate accuracy is, so we'd say, okay, there are total observations are 150, and what are the correctly predicted values? These are the correctly predicted values, which is 18 + 103. So this will give us our accuracy. So 103 + 18, which is 121, divided by our total observations, which is 150, is our accuracy, which is 0.8, or we can say it is 80%. Okay.

Now there is another concept called precision. So precision is given by the formula true positives divided by the predicted positives—totally predicted positives. Okay, what do we mean by that? Which are the true positives here? Remember which are the true positives? We just recall; we just talked in the previous slide, which are the true positives you see here. So this 103 are the true positives, which means that the value is positive, actual value is positive, predicted value is also positive. So that's why it's called a true positive. So 103 divided by—so that is our true positive—103 divided by totally predicted as yes. Now what is totally predicted as yes? Remember 130 of them have altogether been predicted as yes. Not that they are correctly predicted; only 103 have been correctly predicted, but 130 of them have been predicted as yes. So precision is basically the ratio of these two—out of the totally predicted, how many of them are actually true, that ratio. So 103 by 130, which is again about 80%, is the precision. That's how you calculate precision. So this is just a simple formula and the term that you need to remember. So accuracy is you need to take total of true positive and true negative divided by the total number of observations. Whereas precision is true positives divided by the totally predicted positives. Okay. So that is our accuracy and precision.

Now what we did—the accuracy calculation was manual, but we can also use some libraries which are already existing and the functions within that library. So scikit-learn provides uh one such uh method. So for example, accuracy_score is one such method. So if you use that and pass your test and predicted values—only the y you need to pass, right, the dependent variable values. So if you pass that, it will calculate it for you. So in this case, again as you can see, it still calculates the same, which is 80%, which we have seen here as well. Okay. So this can be done using the method. Great. So that's pretty much what we have done here.

Before we conclude, let me take you into the code and show you how it actually looks. Okay. So this is our code. Let me run it. Okay. One by one. We have already seen most of the steps in the slides. So I will, but I will run this in the actual Jupyter notebook. Some of you, if you're not yet familiar with Jupyter notebook, again there are other videos we created on how to install Jupyter notebook and how to set up Jupyter notebook and so on. In this tutorial also we—there was one slide on how to install Python and Jupyter notebook. If you have not yet done, please do that so that then you can actually walk through this code while you're watching this. Okay.

So what are we doing here? We are importing the libraries—required libraries. Recall here we have pandas, we have numpy, and for uh visualization we have matplotlib. And this line is basically reading the CSV file. So we have the CSV file locally on our local drive. And this is where I'm checking the data—just so I'm starting with my exploratory analysis—how the data is looking. So it looks good. There are no major missing values or anything like that. So it will display all the columns and uh it will show me the first five rows if when I'm using this head function, and then I want to see a kind of a summary of all the each of the numerical columns. So that's what I'm doing here. So these are the numerical columns, and it gives a summary like how many observations are there, what is the mean, standard deviation, minimum, maximum, and so on and so forth for each of them, and uh then you can do some visualization. So this is the visualization for this. Okay. The next step is to view the data—data visualization—and we will do that using a histogram for a couple of these columns. So in this case I'm taking a look at the loan amount, and if I create the histogram, it displays the data here in the form of a histogram. One thing that we gather from this, as I mentioned in the slides as well, is how the data is kind of scattered. So while most of the values are in this range—0 to 300 range—there are a few extreme values around the 700 range. So that is one information we get from this histogram. Similarly, for the applicant income, if we draw a histogram, something similar, we can see that while most of the values are in this range—0 to 20,000 range—there are a few in the range of 80,000 and probably 65,000 and so on. Okay.

So the next step is to perform data wrangling, where we will check if any data is missing and how to fill those missing values and so on. So in this case we will just check for all the columns how many data or how many entries are there with missing values. So this is the result. So loan ID has all the columns or all the cells filled. Gender has 13 missing values. Marital status has three missing values and so on and so forth. Loan amount has 21. And this is what we are going to show you how to remove these missing values. So when you have missing values, as I mentioned in the during the slides, there are a couple of ways of handling that. One is you can completely remove those, or fill in with some meaningful values. So in this case we will fill the missing values with the mean value of the loan amount. So let's go ahead and do that. And uh now if we check here, now loan amount—number of missing values is zero—because what we did was for all these 21 cells where the values were missing, we filled with the mean value of the loan amount. So now there are no more missing values for loan amount. We can do this for other columns as well, but this was just one example, so we have shown it here. Okay. So we will run this for credit history and loan amount term as well. And uh then if we calculate the mean of pretty much all the numerical columns, that's the method call. So DF.mean will give us the mean of all the numerical values. And another thing that we can do is we—if you want to find out what are the data types of each of these columns. So you can call DF.dtypes and get the data types. Of course, it may not be that very useful; most of the cases this is an object, but for example this one, it shows as int64, and there are float64 and so on and so forth.

Now, in addition to doing the exploratory data analysis, we can do some machine learning activity as well. So in this case we are going to do logistic regression. So this is the example that I have shown you in the slides as well. This is the actual code for that. All right. So the first step here is to import the libraries, and then the next step is to separate the independent variables and the dependent variables. So X is our independent variable, and Y is our dependent variable. So we separate the data into two parts, and this will be our target as well. Right? So that's how we separate it. Now we have to split the data into training and test data sets. As I mentioned in the during the slides, we use the train_test_split method, and when we call this and pass the independent variables and the dependent variables, and we specify the test size to be 0.25, which means the training size will be 75, which is nothing but you split the data into training data set which is 75%, and test data set in which is 25%. Okay. So once you split that, you will have all your independent variables data in X_train—the training data, which is 75% of it. Similarly, independent variables for test will be in X_test, and dependent variables train will be in y_train, and dependent variable test will be y_test. Once we do this, we have to do a small exercise for scaling. Remember we had uh some data which was kind of very scattered; there were some extreme values and so on. So this will take care of that so that the data is normalized so that before we pass to our algorithm, the data is normalized so that the performance will be much better. The next step is to create the instance of logistic regression object. So that's what we are doing here. So classifier is our logistic regression instance, right? classifier = LogisticRegression, we are saying. So one instance of logistic regression is created, and then we call the training method. The name of the method actually is fit, but what it is doing is it is taking the training data. X is the training data or the independent variables, and Y is the dependent variables. So we are taking both of these, and the model gets trained. So the method for calling the training is fit. Okay. So it gives us the output, and then once we are done with the training, we do the testing. And uh once again, just to recall in the slides when I was showing you the slides also, I mentioned we don't pass Y here while we are testing—while for training we do pass Y, but—right, so for fit we are passing X and Y, but for test we are only passing X—something you need to observe because Y will be calculated by the model, and we will then compare that with the known value of Y to measure the accuracy. So that's what we will do here. And the method that is called here is predict. So this will basically create or predict the values of y. Now we have in this case a binary classification. So the outputs are yes or no. Y indicates yes, and n indicates no. So y or n is the output. Now how do we measure the accuracy? As we have seen earlier, I described how confusion matrix works and how we can use confusion matrix for calculating the accuracy. That's what we are seeing here. So this is the confusion matrix, and then you want to do the measure the accuracy; you can directly use this method, and we find that it is 80%. So we—in the slides we have seen when we calculate manually as well—we get an accuracy of 80%. Okay. Let's go back to our slides and uh do a summary.

So what we have done in this session, we talked about what is data science and uh why Python is being used, why it is becoming so popular, how to install Python, and we talked about the various libraries in Python like Pandas, SciPy, NumPy, and so on. And then we took a couple of examples um and wrote the code and demonstrated the code for performing exploratory analysis and um performing data wrangling or data manipulation, and then we at the end we did one example of machine learning using scikit-learn library and uh performed the logistic regression example. In this rapidly evolving technology landscape, where the world is inundated with huge volumes of data daily, the increasing demand for harnessing the true potential of data has become a paramount challenge for businesses across the globe. Data science has been the hot cake in the IT industry for nearly a decade now. And one of the standard skills necessary to be a good data scientist is being able to retrieve and work with data efficiently. And to do that, you need to be well-versed in SQL, the standard language for communicating with database systems. Now SQL may appear simple on the surface, but beneath its elegant simplicity lies a robust language that empowers data scientists to extract invaluable insights from vast data sets. From startups to MNC's, SQL plays a pivotal role in diverse domains—from marketing and finance to healthcare and automation. SQL's widespread industry adoption ensures its relevance and opens up a plethora of career opportunities for professionals skilled in the ever-evolving landscape of data science. SQL remains a foundational tool that enables data scientists to unlock the true potential of data and derive meaningful outcomes, and learning it will add great value to your array of skills.

On that note, hello everyone, welcome to Simply Learn. In today's video, you will gain expertise on why you should be learning SQL for effectively communicating and extracting data from databases, which is a prerequisite for anyone working in the field of data science. So without any further ado, let's get started. Firstly, let us understand what is SQL. Now, SQL is a programming language designed for managing and manipulating relational databases. It's also the standard way to interact with databases, allowing users to retrieve, insert, update, and delete data efficiently. SQL provides a logical and structured approach to organizing and accessing information stored in tables using commands such as insert, update, modify, and delete. It basically enables users to define relationships between tables, create complex queries, and perform various data operations. It is widely used in web development, data analysis, and database management, making it a fundamental tool for handling structured data efficiently. But now comes the main question: Why should you learn SQL? Now SQL plays an important role for managing databases, and one of the main reasons is it is used to access and manipulate data present in the databases using SQL queries. Now some of the main reasons why you should be learning SQL is it retrieves specific data efficiently. Now SQL allows you to retrieve specific data from databases efficiently. With SQL queries, you can extract relevant information from large data sets based on specific criteria, such as filtering by conditions, sorting, and aggregating data. So this capability is crucial for data analysis and decision-making processes. Secondly, powerful tool for data manipulation. Now SQL provides powerful tools for manipulating data. You can perform operations like inserting, updating, deletion, as well as altering table data and its structure. SQL's data manipulation capabilities enable you to transform and modify data according to your needs, ensuring data accuracy and integrity. Now, it is instrumental in data integration. Now, SQL plays an important role in data integration where you need to integrate data from multiple sources. So, by using, let's say, SQL joins, you can combine data from different tables or databases based on common fields. This allows you to create comprehensive data sets for analysis, reporting, and decision-making. And finally, aggregation and summarizing data easily. Now, SQL enables you to aggregate and summarize data easily. With SQL aggregate functions, you can calculate metrics such as sum, average, maximum, minimum, and count with its simple syntax. So, aggregating data allows you to gain insights and understand trends at a higher level, facilitating effective analysis and reporting.

Now that we have understood what is SQL, let us have a quick introduction to what is data science. Now data science is a field that combines statistical analysis, machine learning, and programming to extract insights, make predictions, and solve complex problems using data. It involves collecting, cleaning, and analyzing vast amounts of structured and unstructured data to uncover patterns to derive actionable insights. Now data science has been rapidly growing and has gained importance due to the exponential growth of data and advancement in computing power and technology. It will be integrated into core business processes, enabling real-time decision-making, personalized experiences, and improved operational efficiency, and it is likely to be approached in various sectors—like marketing, finance, supply chain, automation, customer service, transport, and much more. So let us now understand what is the importance of SQL in data science. Now the first reason is efficient data retrieval and manipulation. Now SQL is designed specifically for managing and querying structured data in relational databases. It provides a powerful set of commands and functions that enable data scientists to retrieve, filter, aggregate, and manipulate large volumes of data efficiently. Seamless integration with databases. Now SQL is a

Standard language for interacting with relational databases. Most organizations store the data in databases such as MySQL, PostSQL, Oracle, or SQL Server. So being proficient in SQL allows data scientists to seamlessly connect to these databases, extract data, perform complex queries, and integrate results into the data science workflows. So SQL acts as a bridge between data stored in databases and the analytical tools and algorithms used in data science.

Next, data exploration and analysis. Now SQL provides powerful capabilities for exploring and analyzing data. Data scientists can use SQL queries to investigate relationships between different variables, identify patterns, summarize data, calculate statistical methods, and create derived tables. So with SQL, data scientists can perform any kind of analysis, generate summary statistics, and gain insights into underlying data structures, which is crucial for understanding the data before applying more analytical techniques. And finally, database performance and efficiency. Now, SQL is designed to optimize data retrieval and manipulation operations. Relational databases are built to handle large volumes of structured data efficiently, and SQL queries are optimized for execution. Now, data scientists can leverage the indexing and query optimization capabilities of databases to improve the performance of their data operations. So, these are some of the main reasons why you should be learning SQL in data science.

So let us now discuss some of the SQL skills that you need to master in order to apply them in data science. Now firstly, having a good knowledge on relational database models. Now we know the field of data science focuses on studying and analyzing data; to effectively analyze the data, it is necessary to extract it from the database, and for data scientists having a solid understanding of how RDBMS and SQL works is certainly valuable as it remains a prevalent and important technology. Now, RDMs is basically a type of database management system that organizes and stores data in a structured manner using tables, columns, and rows. So, SQL is a standard language which is used to interact with these RTBMS and perform various operations like querying and much more. So, having a broader knowledge on RTBMS data model is important for a data scientist.

Secondly, having a good knowledge on SQL query commands. Now, SQL commands are basically the backbone of querying the databases, right? So having a knowledge on various query commands such as DDL, which is helpful to define the data structure or schema, or even DML, which allows users to modify the database instantly by inserting, modifying, deleting data, and data control language which gives access to the data stored in the database by granting and revoking permissions, and finally transaction control language which commands deal with the transactions within the database. So having a good understanding of all these SQL commands allows data scientists to effectively retrieve, manipulate, define, and control data in relational database management systems.

Next, having a good knowledge on table constraints. Now there are various table constraints which are imposed on the table. Now constraints are basically a set of rules imposed on the tables of relational databases which helps in maintaining the accuracy, integrity, and reliability of a database. Now we have different types of constraints, for example, not null, foreign key, default, unique, primary key, and check. So these are a few examples of table constraints in SQL which improves query performance by creating an index on a column or set of columns. So understanding and utilizing these constraints appropriately can help maintain data consistency and improve the reliability of database operations in data science tasks.

Moving ahead, you have to have a good knowledge on various SQL key constraints as well. Now we have various SQL keys which is an attribute or set of attributes which help you to identify a unique row in a table. So SQL keys basically establish relationships between multiple tables in a database. So having a good knowledge on various keys like primary key, candidate key, super key, and foreign key is a must for a data scientist.

Moving ahead, knowledge on SQL aggregation and join operations. Now as a data scientist, you have to learn about table relationships and how to use joins to combine data from multiple tables. So first let us understand what is aggregate functions. Now aggregate function in SQL is performed on a calculation on multiple values and returns a single value. So you have to explore various aggregate functions like sum, count, average, max, and minimum and understand how to group the data using a group by clause and apply aggregate functions to calculate summary statistics on group data. Also as discussed, you have to have a good knowledge on join functions where you need to combine data from multiple tables. So understand different types of joins such as inner join, left join, right join, and full join. Now in order to practice writing these queries that table joins based on common columns, you have to have a good knowledge on SQL join functions which is used to join one or more tables.

Moving ahead, having a good knowledge on SQL clauses and functions. Now clauses are inbuilt SQL functions which are used to filter and analyze data quickly which in turn reduces the complexity of a query. So having a good knowledge on various classes in SQL can surely help data scientists in order to minimize their workflow and work on various complex queries. So you have to have a good knowledge on various clauses like where, which is used to filter records in the database. Order by clause which sorts the data either in ascending or descending order. Group by clause which is used to group identical data in order to obtain summarized database results with aggregation methods. Next we have having clause which filters the rows where aggregate function meets the specified conditions that we have given to the query. Next, having a good knowledge on limit clause which is used to limit the number of results from the query. And finally, testing clause which discards the rows with duplicate values and also familiarize yourself with various window functions which allows you to perform calculations. Learn how to use functions like row number rank to perform database operations on order subsets of data. So practice using window functions to solve complex analytical problems as well.

Moving ahead, having a strong knowledge on SQL views and indexing. SQL views and indexing are crucial aspects of data science when it comes to efficient data access and analysis. So what is an SQL view? So SQL view is a virtual table derived from one or more existing tables in a database. So it is created based on a query and behaves like a regular table but its data is not physically stored. Views help data scientists simplify their queries by abstracting the underlying complexity. So instead of writing complex joins and aggregations repeatedly, they can create a view that will encapsulate the necessary logic. This allows for more concise and readable queries, reducing the potential for any errors, and also having a good indexing strategy will help retrieve database data quickly by providing a lookup mechanism and helps a faster data recovery. So when a query includes a condition on an index column, the database engine can use the index to locate the relevant data rows directly instead of scanning the entire table. So which in turn will improve the search efficiency. So this significantly speeds up query execution, especially for large tables.

And finally, subqueries and CTE. Now SQL subquery and common table expressions are powerful features that data scientists can utilize for advanced data analysis and manipulation. Now, SQL query which is also known as a nested query is a query embedded within another query. It allows data scientists to perform complex operations by using the results of one query as an input for another. On the other hand, we have common table expressions which is a temporary named result set that can be referenced with a SQL statement. So, it allows data scientists to create complex queries by breaking them into smaller and more readable parts. So these were some of the main SQL skills that you need to master for a data science role. I hope you've understood them clearly. And with that, we have come to the end of today's session, guys. By mastering all these SQL skills, data scientists can effectively retrieve, manipulate, and analyze data stored in complex relational databases and perform operations, optimize their performance, and integrate SQL with their data science workflows. These skills are essential for data scientists to work with structured data and derive valuable insights on a daily basis.

Let's begin this lesson by defining the term statistics. Statistics is a mathematical science pertaining to the collection, presentation, analysis, and interpretation of data. It's widely used to understand the complex problems of the real world and simplify them to make well-informed decisions. Several statistical principles, functions, and algorithms can be used to analyze primary data, build a statistical model, and predict the outcomes. An analysis of any situation can be done in two ways: statistical analysis or a non-statistical analysis. Statistical analysis is the science of collecting, exploring, and presenting large amounts of data to identify the patterns and trends. Statistical analysis is also called quantitative analysis. Non-statistical analysis provides generic information and includes text, sound, still images, and moving images. Non-statistical analysis is also called qualitative analysis. Although both forms of analysis provide results, statistical analysis gives more insight and a clearer picture, a feature that makes it vital for businesses.

There are two major categories of statistics: descriptive statistics and inferential statistics. Descriptive statistics helps organize data and focuses on the main characteristics of the data. It provides a summary of the data numerically or graphically. Numerical measures such as average, mode, standard deviation or SD, and correlation are used to describe the features of a data set. Suppose you want to study the height of students in a classroom. In the descriptive statistics, you would record the height of every person in the classroom and then find out the maximum height, minimum height, and average height of the population.

Inferential statistics generalizes the larger data set and applies probability theory to draw a conclusion. It allows you to infer population parameters based on the sample statistics and to model relationships within the data. Modeling allows you to develop mathematical equations which describe the interrelationships between two or more variables. Consider the same example of calculating the height of students in the classroom. In inferential statistics, you would categorize height as tall, medium, and small, and then take only a small sample from the population to study the height of students in the classroom.

The field of statistics touches our lives in many ways. From the daily routines in our homes to the business of making the greatest cities run, the effect of statistics is everywhere. There are various statistical terms that one should be aware of while dealing with statistics: population, sample, variable, quantitative variable, qualitative variable, discrete variable, continuous variable. A population is the group from which data is to be collected. A sample is a subset of a population. A variable is a feature that is characteristic of any member of the population differing in quality or quantity from another member. A variable differing in quantity is called a quantitative variable. For example, the weight of a person, the number of people in a car. A variable differing in quality is called a qualitative variable or attribute. For example, color, the degree of damage of a car in an accident. A discrete variable is one which no value can be assumed between the two given values. For example, the number of children in a family. A continuous variable is one in which any value can be assumed between the two given values. For example, the time taken for a 100-meter run.

Typically, there are four types of statistical measures used to describe the data. They are measures of frequency, measures of central tendency, measures of spread, measures of position. Let's learn each in detail. Frequency of the data indicates the number of times a particular data value occurs in the given data set. The measures of frequency are number and percentage. Central tendency indicates whether the data values tend to accumulate in the middle of the distribution or toward the end. The measures of central tendency are mean, median, and mode. Spread describes how similar or varied the set of observed values are for a particular variable. The measures of spread are standard deviation, variance, and quartiles. The measures of spread are also called measures of dispersion. Position identifies the exact location of a particular data value in the given data set. The measures of position are percentiles, quartiles, and standard scores.

Statistical analysis system or SAS provides a list of procedures to perform descriptive statistics. They are as follows: Proc print, proc contents, proc means, proc frequency, proc univariate, proc g-chart, proc box plot, proc g-plot. Proc print: it prints all the variables in a SAS data set. Proc contents: it describes the structure of a data set. Proc means: it provides data summarization tools to compute descriptive statistics for variables across all observations and within the groups of observations. Proc frequency: it produces one-way to n-way frequency and crosstabulation tables. Frequencies can also be an output of a SAS data set. PROC univariate: It goes beyond what proc means does and is useful in conducting some basic statistical analyses and includes high-resolution graphical features. PROC G-chart: The G-chart procedure produces six types of charts: block charts, horizontal vertical bar charts, pie doughnut charts, and star charts. These charts graphically represent the value of a statistic calculated for one or more variables in an input SAS data set. The trend variables can be either numeric or character. Proc box plot: The box plot procedure creates side-by-side box and whisker plots of measurements organized in groups. A box and whisker plot displays the mean, quartiles, and minimum and maximum observations for a group. Procplot: G-plot procedure creates two-dimensional graphs including simple scatter plots, overlay plots in which multiple sets of data points are displayed on one set of axes, plots against the second vertical axis, bubble plots, and logarithmic plots.

In this demo, you'll learn how to use descriptive statistics to analyze the mean from the electronic data set. Let's import the electronic data set into the SAS console. In the left pane, right-click the electronic.xlsx data set and click import data. The code to import the data generates automatically. Copy the code and paste it in the new window. The proc means procedure is used to analyze the mean of the imported data set. The keyword data identifies the input data set. In this demo, the input data set is electronic. The output obtained is shown on the screen. Note that the number of observations, mean, standard deviation, and maximum and minimum values of the electronic data set are obtained. This concludes the demo on how to use descriptive statistics to analyze the mean from the electronic data set.

So far you have learned about descriptive statistics. Let's now learn about inferential statistics. Hypothesis testing is an inferential statistical technique to determine whether there is enough evidence in a data sample to infer that a certain condition holds true for the entire population. To understand the characteristics of the general population, we take a random sample and analyze the properties of the sample. We then test whether or not the identified conclusions correctly represent the population as a whole. The population of hypothesis testing is to choose between two competing hypotheses about the value of a population parameter. For example, one hypothesis might claim that the wages of men and women are equal, while the other might claim that women make more than men. Hypothesis testing is formulated in terms of two hypotheses: null hypothesis, which is referred to as Hnull; alternative hypothesis, which is referred to as H1. The null hypothesis is assumed to be true unless there is strong evidence to the contrary. The alternative hypothesis is assumed to be true when the null hypothesis is proven false. Let's understand the null hypothesis and alternative hypothesis using a general example. Null hypothesis attempts to show that no variation exists between variables, and alternative hypothesis is any hypothesis other than the null. For example, say a pharmaceutical company has introduced a medicine in the market for a particular disease, and people have been using it for a considerable period of time and it's generally considered safe. If the medicine is proved to be safe, then it is referred to as the null hypothesis. To reject the null hypothesis, we should prove that the medicine is unsafe. If the null hypothesis is rejected, then the alternative hypothesis is used.

Before you perform any statistical tests with variables, it's significant to recognize the nature of the variables involved. Based on the nature of the variables, it's classified into four types. They are categorical or nominal variables, ordinal variables, interval variables, and ratio variables. Nominal variables are ones which have two or more categories, and it's impossible to order the values. Examples of nominal variables include gender and blood group. Ordinal variables have values ordered logically. However, the relative distance between two data values is not clear. Examples of ordinal variables include considering the size of a coffee cup, large, medium, and small, and considering the ratings of a product, bad, good, and best. Interval variables are similar to ordinal variables except that the values are measured in a way where their differences are meaningful. With an interval scale, equal differences between scale values do have equal quantitative meaning. For this reason, an interval scale provides more quantitative information than the ordinal scale. The interval scale does not have a true zero point. A true zero point means that a value of zero on the scale represents zero quantity of the construct being assessed. Examples of interval variables include the Fahrenheit scale used to measure temperature and distance between two compartments in a train. Ratio scales are similar to interval scales in that equal differences between scale values have equal quantitative meaning. However, ratio scales also have a true zero point which give them an additional property. For example, the system of inches used with a common ruler is an example of a ratio scale. There is a true zero point because 0 inches in fact indicates a complete absence of length.

In this demo, you'll learn how to perform the hypothesis testing using SAS. In this example, let's check against the length of certain observations from a random sample. The keyword data identifies the input data set. The input statement is used to declare the aging variable and cards to read data into SAS. Let's perform a t-test to check the null hypothesis. Let's assume that the null hypothesis to be that the mean days to deliver a product is 6 days. So null hypothesis equals 6. Alpha value is the probability of making an error which is 5% standard and hence alpha equals 0.05. The variable statement names the variable to be used in the analysis. The output is shown on the screen. Note that the p-value is greater than the alpha value which is 0.05. Therefore, we fail to reject the null hypothesis. This concludes the demo on how to perform the hypothesis testing using SAS.

Let's now learn about hypothesis testing procedures. There are two types of hypothesis testing procedures: they are parametric tests and non-parametric tests. In statistical inference or hypothesis testing, the traditional tests such as t-test and ANOVA are called parametric tests. They depend on the specification of a probability distribution except for a set of free parameters. In simple words, you can say that if the population information is known completely by its parameter, then it is called a parametric test. If the population or parameter information is not known and you are still required to test the hypothesis of the population, then it's called a non-parametric test. Non-parametric tests do not require any strict distributional assumptions. There are various parametric tests. They are as follows: t-test, ANOVA, chi-squared, linear regression. Let's understand them in detail. t-test: A t-test determines if two sets of data are significantly different from each other. The t-test is used in the following situations: To test if the mean is significantly different than a hypothesized value. To test if the mean for two independent groups is significantly different. To test if the mean for two dependent or paired groups is significantly different. For example, let's say you have to find out which region spends the highest amount of money on shopping. It's impractical to ask everyone in the different regions about their shopping expenditure. In this case, you can calculate the highest shopping expenditure by collecting sample observations from each region. With the help of the t-test, you can check if the difference between the regions are significant or a statistical fluke. ANOVA: ANOVA is a generalized version of the t-test and used when the mean of the interval dependent variable is different to the categorical independent variable. When we want to check variance between two or more groups, we apply the ANOVA test. For example, let's look at the same example of the t-test example. Now you want to check how much people in various regions spend every month on shopping. In this case, there are four groups, namely east, west, north, and south. With the help of the ANOVA test, you can check if the difference between the regions is significant or a statistical fluke. Chi-squared: Chi-squared is a statistical test used to compare observed data with data you would expect to obtain according to a specific hypothesis. Let's understand the Chi-square test through an example. You have a data set of male shoppers and female shoppers. Let's say you need to assess whether the probability of females purchasing items of $500 or more is significantly different from the probability of males purchasing items of $500 or more. Linear regression: There are two types of linear regression: simple linear regression and multiple linear regression. Simple linear regression is used when one wants to test how well a variable predicts another variable. Multiple linear regression allows one to test how well multiple variables or independent variables predict a variable of interest. When using multiple linear regression, we additionally assume the predictor variables are independent. For example, finding the relationship between any two variables, say sales.

and profit is called simple linear regression. Finding the relationship between any three variables—say, sales, cost, telemarketing—is called multiple linear regression.

Some of the non-parametric tests are the Wilcoxon rank sum test and the Kruskal-Wallis H test. The Wilcoxon rank sum test is a non-parametric statistical hypothesis test used to compare two related samples or matched samples to assess whether or not their population mean ranks differ. In the Wilcoxon rank sum test, you can test the null hypothesis on the basis of the ranks of the observations.

The Kruskal-Wallis H test is a rank-based non-parametric test used to compare independent samples of equal or different sample sizes. In this test, you can test the null hypothesis on the basis of the ranks of the independent samples.

The advantages of parametric tests are as follows: They provide information about the population in terms of parameters and confidence intervals; they are easier to use in modeling, analyzing, and for describing data with central tendencies and data transformations; and they express the relationship between two or more variables. They don't need to convert data into rank order to test.

The disadvantages of parametric tests are as follows: They only support normally distributed data and are only applicable on variables, not attributes.

Let's now list the advantages and disadvantages of non-parametric tests. The advantages of non-parametric tests are as follows: They are simple and easy to understand; they do not involve population parameters and sampling theory; they make fewer assumptions; and they provide results similar to parametric procedures.

The disadvantages of non-parametric tests are as follows: They are not as efficient as parametric tests, and it is difficult to perform operations on large samples manually.

Welcome to this video on the use cases of mathematics in data science and Bayesian inference by Simplilearn. In today's video, we will embark on a journey to explore the intrinsic relationship between mathematics and data science. Mathematics serves as the backbone of data science, providing the essential tools and frameworks to extract valuable insights and make informed decisions from data. We will commence by examining various use cases where mathematics plays a pivotal role in data science applications, from predictive modeling to machine learning algorithms and data visualization techniques. We will uncover the mathematical principles that underpin these powerful tools. Furthermore, we will delve into the realm of Bayesian inference, a remarkable paradigm that has revolutionized the way we approach data analysis and decision-making. We will dissect the core principles of Bayesian inference, its applications across diverse domains, and the process of updating beliefs based on new evidence.

We will be discussing use cases of mathematics in data science. Then we have Bayesian inference. So Bayesian inference is used widely in different verticals. It's a popular data science technique used for businesses, especially for pricing decisions. So businesses can determine the prices for products based on field information like retail and wholesale prices, the size of the market, and the market share. So it can be used for new product development as a whole. Maybe one would look at the different risks by weighing uncertainties and determining if the project is worth it or not. But again, when it comes down to Bayesian inference and business decisions, the most common application relates to product ranking. So many systems have been developed to help customers have an ideal and customized shopping experience. So online shopping giants like Amazon use it to make ratings appear natural when searching for a product as opposed to displaying rankings in the order. The default option; it is used in terms of marketing. So again, we use it to look at previous campaigns and marketing information to improvise the existing campaigns or make new ones.

Hypothesis testing. Then we have hypothesis testing. So hypothesis testing is used in different scenarios. First of all, it is used in clinical trials as well. So many pharmacists and doctors use hypothesis testing for clinical trials. So the impact of a new clinical method, medicine, or procedure, or the condition of the patient is analyzed through hypothesis testing. For example, a pharmacist believes that the new medicine is resulting in the rise of blood pressure in diabetic patients. Now, to test this assumption, the researcher has to measure the blood pressure of the sample patients before and after the intake of new medicine for nearly a particular period, let's say 1 month. So then there can be two scenarios. We have the null hypothesis: the average blood pressure is the same after and before the conception of medicine. And then we can have the alternative hypothesis: the average blood pressure after the consumption of the medicine is less than the average blood pressure before the consumption of medicine. So if the p-value of the hypothesis test is less than the significance value, then the null hypothesis is going to be rejected. It is also used widely in testing the effectiveness of fertilizers on plants. So nowadays, it is highly used by farmers and those in the agricultural vertical. So suppose, let's say a researcher wants to check their assumption that a particular fertilizer may result in faster growth of the plant in a month than its usual growth—than what is observed, I suppose, in a couple of plants. Now, to verify this assumption, we will consistently give the fertilizer to the plant for nearly a month, and then it can be further divided into two hypotheses: null and alternative. Where the null will be like: the fertilizer does not have any influence on the growth, and the alternative can be: the fertilizer results in faster growth of the plant. And it is also used highly in the manufacturing sector as well. So it is used in finding the defects in any kind of manufactured product. For example, let's say manufacturing plant X decides to verify that a particular method results in an increase in the defective products per quarter, let's say the number can be 200. Now, to verify this, we can calculate the mean of the numbers of defective products produced before the start and end of the quarter. So again, we can divide them into two different scenarios: null and alternative, where the null can be the average number of defective products, and the alternative can be the average number of defective products before and after the implementation of the new method, and then we can see which one fits into our current criteria.

Then we have image and signal processing. So we have a number of use cases for image processing, and it is highly used in different verticals, including insurance, retail, banking, and pharmaceutical, especially in the healthcare community as well. So basically, it is used in reverse image search for different purposes. So, for example, if we have any image that we want to search on a webpage, if we use this image for performing a search, we can drag-and-drop search, and then any search engine will be able to use this image as a base and give us all the relevant links where we can find the image or whatever is present in the image. So this is called a reverse image search. So we can search by an image and find out the relevant details about it. So it uses artificial intelligence software and image recognition techniques to identify image content and then compare them with billions of images indexed on the web. So the image recognition algorithms help find out similar images, the origin of the image, and information about the owner of the image, websites using the same image, image plagiarism, and all other relevant information. In the past, reverse image search was only used for finding similar images on the web, but today we can use it for multiple other requirements as well.

Then we talk about linear regression. So linear regression is basically a technique through which we can model the relationship between different data pointers, and one of the most widely used use cases is weight loss reduction. So the problem of predicting the weight reduction, in the form of the number of kilos reduced hypothetically, could depend upon the input features that we have: age, height, the weight of the person, and the time spent on exercises. So weight reduction is going to be the function of age, height, weight, and time spent on exercises. So that's why we have a shoe size, which is going to be based on the height, weight, age, and time on exercise. So as a part of creating the above model, the goal will be to find the value of B1, B2, B3, B4, and B0, which would minimize the objective function. So the objective function would be the summation of the mean squared error, which is nothing but the sum of the square of the actual value and the predicted value for different values of age, height, weight, and time spent on exercises. That's how it's going to be. It is also used for forecasting the performance of students based on the marks based on the number of hours they have put in. Let's consider the problem of predicting the marks of a student based on the number of hours he or she puts into the preparation. Although, at the outside, it may look like a problem that can be modeled using simple linear regression, it could turn out to be a multiple linear regression problem depending upon multiple input features, and alternatively, it can also look like a problem that can be modeled using simple linear regression, but it can be a multiple linear regression problem. So let's assume, for the sake of understanding, that the marks of student M depends upon the number of hours he or she puts for the preparation. So this is going to be the formula that can represent this model: marks = (hours * m) + constant. So the best way to determine whether it is a simple linear regression problem will be to do a plot of marks versus hours. If the plot is going to be completely diagonal, then it is going to be a good fit for a linear regression setup.

So we have predictive modeling, which is basically used to model the probability distribution of variables in a given data set. So there are multiple use cases. First of all, we have churn prevention. So when a business loses a customer, it has to replace a loss of revenue by bringing in a new customer. So it proves to be expensive as the cost of acquiring a new customer is much higher than retaining the existing customer. So predictive analytics models help us in preventing the churn in our customer base by analyzing the dissatisfaction among our current customers and then identifying the customer segments at most risk for leaving. Our businesses can make the necessary modifications using predictive data to keep customers happy and satisfied, eventually protecting their revenue. And then it is also used for customer segmentation. So it allows us to group the customers by shared traits. In different businesses, they define the market differently depending upon the aspects that offer the most value to the company's products or services. So profound use of this analysis technique allows us to target markets based on accurate insight and indicators, and then we can analyze these segments of those most interested in what our company offers. So there are multiple verticals like banking, pharmaceutical, retail, insurance, and the automotive sector; they all make use of customer segmentation. And then, in terms of quality assurance, ineffective quality control will affect our customer satisfaction scale and ultimately impact the revenue and market share, also leading to more customer support calls, warranty issues, repairs for inefficient manufacturing, and so on. So using predictive analytics, we can help identify high-risk modules in our application where we can prioritize the critical areas. We can reduce the time to market through shift-left testing. And by using this, we are going to get better quality, and we will be able to retain customers for longer. Different verticals like pharmaceutical, manufacturing, automotive, logistics, and transportation all make use of this to keep on improving their current business models. And as we discussed earlier, there are multiple use cases of these in healthcare, like we use that in creating some kind of prediction modeling. So all the data sets from the demographics, clinical measurements, medical history, and mathematics are all used to create models so that we can make an accurate prediction of how the same data set is going to behave for any future references, how we can find the most probable condition before it actually happens. So they are used to analyze large data sets of health-related information so that we can prevent diseases before they happen. Like we have image scanning for the lungs. So based on previous lung cancer patients, we know what exactly is the pattern or development of cancer cells, and based on the current scanning which has already been completed, we can let the patient know if they are having a probability of developing the same lung cancer or not. So that since they will be having enough time, we can make the right decision at the right time so that the prevention of cancer can get started before it spreads. Then we have some clinical trials, like we discussed. So mathematics play a crucial role in the design and analysis of clinical trials. So we can make use of the models; we can make use of hypothesis testing to see if a given drug is effective or not. And then we can carry on with other hypothesis scenarios. It is used for medical imaging so that we can know how exactly we can identify tumors; we can identify cells; we can identify some kind of cancer cells before they actually start in the blood. So that we can analyze thousands of previous scans done, and based on the pattern seen in the actual patients, we can cross-verify that by using the current scan and simply get the best probability of developing cancer and so on. These all things can be easily verified there, and it is also used in precision medicine to ensure that we are able to give and use the right amount of medicine so that the possibility of them getting an overdose or getting less than what will be effective for their treatment can be avoided. So these things are all done by using machine learning.

Today we will learn the basics of what exactly Tableau is, the charts and graphs available in Tableau, the story reports, and the dashboard. Along with that, we will also understand some critical aspects like data planning in Tableau, joins, and data modeling in Tableau. Right now, for a better understanding, let's start with a simple use case so that you have a better idea of what the problem statement is and how exactly you can come up with a solution to that particular problem statement. Let's consider a simple use case for a real-time imports and exports dashboard for business growth. Let's imagine you're an ambitious entrepreneur who runs a successful import and export business dealing with multiple products across various states in your country. However, you recently started to struggle with a lack of real-time visibility into your operations. Tracking your imports and exports to specific states or monitoring product performance was a challenge, which often led to delayed business decisions. One day you discover Tableau, a powerful data visualization tool. And after implementing it, you're amazed at the clarity it provided. Tableau can offer real-time insights, showing trends, product demand, and country-specific data at a glance. With this new-found control, you can now make faster, smarter decisions, optimizing your supply chain, and growing your business with confidence. So that's how Tableau can change the world of business intelligence. Now, without further delay, let's get started with our topic for today, which is Tableau data visualization. So let's get started with the agenda for today's session. Firstly, what exactly is Tableau? Followed by that, we will have a simple two-step procedure to install Tableau in your Windows operating system. Then we will start Tableau. So we will have a quick walkthrough about what are the features in Tableau, how to import data. And followed by that, once we import the data, we will check into the joins in Tableau and data blending and data modeling in Tableau. Once we have all the data ready for processing for data visualization, then we will get started with building storyboards, and lastly, create a fully functional operational Tableau dashboard. I hope I made myself clear with the agenda. Now let's get started. So what exactly is Tableau? So basically, Tableau is just another business intelligence tool. Tableau is a visual analytics platform transforming the way we use data to solve problems, empowering people and organizations to make the most of their data. Now the process of installing Tableau in your Windows. So if you're using the Windows operating system, just open Google, and once you open Google, you just have to type "install Tableau," and then you will have the following webpage. So once you get into this webpage, you just have to get yourself registered by adding your first name, last name, all the details, business or personal email. If you're working for an organization, you can add that, the size of the company, department, job role, country, etc., phone number, and then you can download a free trial. Once you download the files, you just have to go through the installation procedures, and you will be fully ready with your Tableau desktop, which looks something like this. Right? We already have some data here. You don't have to worry. We will restart Tableau and try to import our data freshly. So there you go. Once you get started with Tableau, this is the overview you will have. So Tableau basically has some open-source data dashboards for your learning experience. So you also have some tabs here where you can get started with learning Tableau, and you also get to explore some sample data sets and everything. Now we will be dealing with this particular one: "Get started with a data source." So Tableau is just like any other data visualization tool which can access data from anywhere. You can get connected yourself through a database; you can get connected yourself to a website, social media platform, or anything. And if you want your data to be in Excel format, you can also get connected to a CSV or an Excel data file or anything. Right? So if it is data, then you can access it. Now we will be dealing with the Excel files, right? So our sales data is present in this particular data set. So we have product data, delivery data, customers data, vendor data. So what I have done is I've combined all those into one complete workbook, which happens to be our "sales combined." Let's have a quick overview of how it exactly looks like. So this is how our sales data looks like. The first tab, which happens to be the "vendors" tab, which is the person who is selling the product. So he has the order ID, order date, delivery date, traffic, leads, etc. Right? So before we have a quick walkthrough about this data, let's try to create a sample data. Right? So so we are making use of GPT here. So this is the prompt. So what we are writing is we want four data sets, each of 5,000 entries. So the four data set names are vendor, customer, order, and delivery. Right? So these are the data sets, and we are looking for these columns. Order ID, order date, delivery date, traffic, leads, orders, revenue, and here we have another column. So you can see "order" in all the four data tables. So there is a reason behind it. So this acts as a primary key, which helps us to create a join or create a relationship between the other tables, right? And here, if you see, we have order date and delivery date together, and here in the last table, last but one, delivery table, we have booking day and receive date, right? So these are one and the same, but we are maintaining a different name because this will help us in understanding what exactly is data blending, right? Data blending is nothing but the same data in between two different tables, shared among two different tables but with different column names. So sometimes, if Tableau is able to understand what exactly is happening, then it will automatically establish a relationship between two different tables. But in case if it doesn't understand by reading the column name, in case if there is a different column name in different tables, then it might not establish that connection. Right? So in such scenarios, you can have to go with the custom connection between two tables by linking the data columns. So in this case, we will link order date with booking date and delivery date with receive date. Right? And the last table is here, which will be having product ID, product category, and quantity. So we're also adding some category names here and product names and quantity. Right? So how many numbers of products have been ordered, right? And we are taking this data from a common man's POV, right? So we will quickly copy this prompt, and we will drop it on the window of ChatGPT. So we're using a premium version here, so quickly we should be receiving the response. There you go. So we have two different responses here, and here you can select any one of those and quickly download in the form of CSV or as a workbook. So we have already downloaded that particular data set and we have clubbed them into one. Right? So we have the same data, and we have distributed them in four different sheets. So these are the ones. So we have every column that we gave into the prompt. And here you can see we have revenue and company. We did not calculate profit, which happens to be revenue minus cost. So that we can do this practically on Tableau to understand the calculated fields. Now that we have a brief understanding about our data set, now let's begin with creating storyboards on our Tableau. So there you go. I'll select

the sales combined workbook and open it. And shortly we will be having the data set ready. So here we have it.

Now, as you can see, we have four different data tables. Now, to establish a data model or to establish a connection between two different columns or to join them, all you have to do is just drag the data set here and leave it. And after that, whichever table you want to join with the first table, just drag and drop. There you go. You have the first connection. This particular pipeline or a string type connection is nothing but a connection between the two different data tables, right? They have established it. And now let's connect this again. This one here. So, I think we drag delivery twice. Let's drag product. And the last one, which happens to be the delivery, uh, not the vendor, right. There you go. So there you go. We have a quick, good connection. So, uh, Tableau has identified the data. So, if you quickly go to the data vision sheet here. So, initially we had a booking day and uh delivery day, receive date, right? So it has understood the format and it has mentioned it as a calendar format here, and order date and uh booking day are recognized as one of the same, and we have a proper connection established here, right? Let's say I want to create a join between two tables. Let me quickly drag and drop all the tables here. Now, let's say I wanted to create a join. Now you double click this particular table and now drag delivery here, and here you can see there is a join, an inner join. If you want to modify that, you can also modify that to a full join, full outer join, right join, and left join. And here we can see Tableau has automatically recognized on what basis it should create a join, which is order ID from the first customer table and order ID from the second table, which happens to be the delivery table, right? Now we don't want to proceed with the join for this particular data set. We will just simply go ahead with data modeling, right? Let's quickly model the data back. I have customers data here, delivery data here and product data here, and lastly the vendor. There you go. Our data model is fully prepared now.

Now, what we can do is go back to sheet one and get started with creating story lines. Now, let's say I want to identify the total sales based on regions. So what we can quickly do is you can add regions to columns. You can add order ids to rows. There you go. So what we did is we quickly changed the aggregation of order ID so that it counts the total number of orders present in that particular region, right? So we have four different regions here. So it has calculated total orders that we received from all the different regions. So to improvise this, you can also add data tables or data labels. So what you can do is uh drag and drop order ids to label, and here the aggregation you can also do that. So that tool tip, you you just have to add it to the tool tip, should be sum. There you go. Now you have the count of orders from three different, four different regions. And in case if you want to make it a little more colorful, you can also add region to color so that each and every region has a different color. So that's the first story that we made, a region-wise sales, so that you can understand which is the highest uh profit-making region in your country. So here I can see uh northeast region is giving us the highest sale here, which happens to be 31,000, no 31 lakh 96,296 sales.

Now let's create a new sheet and now let's understand the sales based on category or product. All right. Now, let me add category to columns and the number of orders that we received based on those particular categories. Add all members, and you might have to change the aggregation. So here it is, measure to sum, and you have u what do you say, the bar graph here, but in case if you don't want the bar graph, you choose to have something else than a bar graph, then you can go with the pie chart here, and here you can change the standard size to entire view, and you have it here, and just like we discussed before, you can add order ID to label and change the aggregation. Yeah, it already changed. Okay. Yeah, there you have. So, change the aggregation to sum or count, and you have the total number of orders from a different category. And if you want the category name also to be popped up, you can also add a category into the table. Now you can see we have medical category has so many number of orders, furniture has so many, auto parts have so many, and electronics has so many number of orders. And you can change the sheet name to category wise sales.

Right now you can create a new story again. So since you're dealing with American states, so you can also do a state-wise sales report. So u where you have the sales number. Yeah, you have the sales uh detail here. You can add it to the columns and uh number of orders. So let's say we will go with traffic. Let's see which country has the highest number of traffic. Now since we have, you can see that the state thing is visualize a geographic location. You can choose a map here, right? You can choose this type of map or this one. So this is one of the recommended ones. I'll go with this one. And it might take a little while to read the data and give us the numbers. And you can also add a state to text label here and uh also some of the traffic. That's okay. It might take a little while to reflect on this particular map. So that's how you do the statewide sales.

And let's go through the data set once again and see what kind of reports we can move ahead and create. Right now we have a product reports here. Let's go with the product sales as well. You can drag and drop product rows and uh the leads and products here. And you can choose a map chart map. There you go. So you can also add leads to label. Just drag and drop it here. And now you can see we have uh medical. Okay. We will also add category, right? Drag and drop. And you can add it to rows to split the category in four different formats. Or you can drag and drop categories into columns to make it look like this. Right now you can see medical is the highest one which is giving you the total highest number of leads. So you have 34,000 number of highest leads from medical category. In a total it is more than about 65 or 67,000, right? That's how you can do it. Now let's keep it this way. Now we can term it as category wise sales, right?

Now let's quickly create a new sheet. Now as discussed, let's try to also venture into the calculated fields. Now if you quickly take a look at the data, we have revenue. We have orders, leads and everything, and we also have cost of the company and revenue. But if you closely observe, we don't have something called as profit, right? You might be spending around 10 rupees to buy a product and 15 rupees to sell it. So you made about 5 rupees of profit. Now, did you see the column named as profit here? No. So how to create that uh new column header which is termed at profit. Now uh let's quickly double click this and double click this one as well and uh go to a table type appearance. Now you can see what's the cost and revenue. Now I wanted to create a new column which is termed as profit. Now click this particular option which is a small arrow kind of thing. And here you can see a create option. Go to a field called as calculated field. Right now we want this to be named as profit, and uh if you are unsure about the functionalities it is about to offer you, can have all this here, right? It you can work with different data types. Let's say I have, since we already have delivery date and order date, we don't know how many days it took to deliver the product. Let's say you wanted to find out that, you wanted to find out the difference between the delivery date and the order date to know what are the number of days a particular vendor or a particular delivery guy is taking to deliver your product from your end to the customer. You can also do that, right? For now, we want to go with the number one, right? So here we have uh multiple options. You can go with max, minimum, power, radians, and you can also go with trigonometry things, everything. Right now I want to calculate. You can uh you know access this using this small uh menu icon here, right? You can do that. So if you don't want, you can just quickly continue. Now here we want to do something uh called as uh sum of uh revenue. Here you have it. You can scroll and press tab minus sum of cost to company, and your formula is done. So in case if uh there are errors, it will say uh the calculation contains errors. So we might have the error over here. Closing parenthesis. Okay. Yeah. So we did not close the parenthesis to this one, right? Uh this is because we manually typed it. But in case if you had taken the recommendations from table, it have got something like this which already has a closing parenthesis over here. Now we want to subtract revenue minus cost to company, right? So again scroll down and press tab to select, and there you go. Now the calculation is valid as you can see here. And now you just press apply and okay or okay anything. And you will have a new column called as profit. Now let's understand, let's remove these both from here. Let's remove this one as well. Now we have a new sheet, fresh sheet. Now we want to identify which category, which vendor. Yeah. Let's say we want to identify that one product which is giving us maximum profit. Now what you can do is you can uh take the product. Let's quickly check the data once. So here we have the product data. Now here, here we have category and product, right? So we want this particular one, the product, and want to find out which is that one single product which is giving us the maximum profits, right? Now let's go to the product uh table. Here we have the product. Select the product and drop it around the columns or rows wherever you want. And now we have recently created a new calculated field which happens to be the profit. Search for it. There you go. We have it here and drop it right here. There you go. Now if you would like to go with this particular one which is bubble chart, you can also do that. And you can also have the aggregation of profit to the text label here and change aggregation to sum. There you go. We already have the table here. So you can see tablets and syrups as we have this here category by sales. So the category was tablets and syrup. The medical category was the highest selling one. So here obviously we have the highest selling ones are these two ones, right? And something from furniture is not performing well, right? So here we have it, 48,000 happens to be the least one and so on, right? Now let's quickly do it uh product, product-wise profit. So far so good, so we have about four to five uh sheets and now we can quickly go with the dashboard with some KPIs. Okay.

Now, what you can do is just quickly drag and drop those sheets onto your dashboard and make some adjustments to it. You can close show me, and here you can choose um standard and the size of your dashboard. You can also program or make your dashboard such a way that it can be visible in phone as well. I don't want the phone now. I want to go with the default one. And here you have size of the canvas. You can increase the width and height of your canvas. I decrease it. There you go. So far we have u installed region wise seal. You can also, you know, uh change the size of your charts here. Drag and drop state wise sales and you can drag and drop category wise sales, product wise profits as well. So there you go. So after making some adjustments right in the width-wise or the length-wise, you can adjust u the dashboard and one thing which is missing here is the KPIs. So we created a new uh sheet here and added a KPI termed as leads. Similarly add another sheet and this will be your revenue. Add revenue to rows or columns and you can choose a table here just like we did before. And you can add um to tool tip. And u similarly you can add orders and graphics as well. And this will be named as orders. This will be named as traffic. So drag and drop traffic to the rows or columns. Select table and add traffic to tool. Similarly orders. Drag and drop orders to rows and this should be a card and an orders to total. There you go. Again, we will go to dashboard and here you can drag and drop the cards that you recently created. So, I want the leads to be here. Don't worry, we can adjust uh the cards. There you go. So after making few adjustments to the title, we can get the last one, apply this particular one as well. Let's keep it as 10 so that the numbers are visible. Lastly this one, so you have the KPIs, region wise sales, state-wise sales, product-wise sales, category wise and product wise as well. So this was supposed to be uh product wise sales. Here we have power product wise and category wise both, not a problem. And here you have, if you close the show me icon. So here we have the filters and slicers. So in case if you wanted to check for the sales happening in only Middle East from category of chairs and uh product chairs and a category as furniture. There you go. So you have the total information on that front. And if you can just quickly release all the filters, you have that. And if you wanted to know the sales of dining table, here you have it with all the related information to that particular item. So that's how you create a completely interactive dashboard in Tableau.

In today's session, we will look at data analytics for beginners. In this video, we will discuss what data analytics is and the need for data analytics. Then we will look at the different ways in which data analytics can be used, followed by the various steps involved in the data analytics process. After that, we will get an idea about the different tools used in data analytics and the companies using data analytics. Moving forward, we will see a case study on how Walmart uses data analytics for better customer service. And finally, we will perform a regression analysis in R to predict sales based on their advertising expenditure from three mediums: TV ads, radio ads, and newspaper advertisements.

So what is data analytics? Companies around the world are generating vast volumes of data every hour. This data could be in the form of log files, web server and transactional data, as well as various customer-related data. Also, data is being generated at a rapid rate from social media websites and applications such as Facebook, Instagram, Twitter, and WhatsApp. Companies want to use this data to derive value out of it and make business decisions. That's where data analytics comes into use. Data analytics is the process of exploring and analyzing large data sets to find hidden patterns, unseen trends, discover correlations, and valuable insights to make business predictions. Data analytics improves the speed and efficiency of your business. A few years ago, a business would have gathered information manually, performed statistical and complex analytics, and unearthed information that could be used for future decisions. But today that business can identify insights on the fly for immediate decisions. Most organizations have big data, and many understand the need to harness that data and extract value out of it. So they use a lot of modern tools and technologies to perform data analytics. Some of the tools I will discuss in detail later in this tutorial.

Now that we have looked at at what data analytics really is, let us understand the ways in which you can use data analytics. First is improved decision making. Data analytics eliminates a lot of guesswork and manual tasks from choosing the right content, planning marketing campaigns, and developing products. Organizations can use the insights they gain from data analytics to make informed decisions, leading to better outcomes and customer satisfaction. It gives you a 360-degree view of your customers, which helps you understand their behavior completely, enabling you to better meet their needs. Second is better customer service. Data analytics provides you with more accurate insights of your customers, allowing you to tailor customer service to their needs, provide more personalization, and build stronger relationships with them. Your data can reveal information about your customers' communication preferences, their interests, their concerns, and more. It helps you give better recommendations for products and services. Next is efficient operations. Data analytics can help you streamline your processes, save money, and boost production. When you have an improved understanding of what your audience wants, you waste less time in creating ads and content that don't match your audience interests. This helps you optimize your campaigns, create better content strategies, and hence improve results. And finally, we have effective marketing. When you understand your audience better, you can market to them more effectively. Data analytics also gives you useful insights into how your campaigns are performing so that you can fine-tune them for optimal outcomes. You also find out the probable customers who are the most likely to interact with the campaign and convert into leads.

Now let's discuss the various steps involved in the data analytics process. As you can see on the screen, there are five process steps. Now let me make you understand each of this one by one. So the first step is to understand the problem. Before starting with the analysis, you need to understand the business problem and define your goals. Asking questions at the outlet is vital because this would address issues such as how can we reduce production costs without sacrificing quality? What are some of the ways to increase sales opportunities with our current resources? Do customers view your brand in a favorable way? Answers to these questions will help you build a clear road map with lucrative solutions. Also try to find out the key performance indicators and consider the metrics to track along the way. The second step in the process is data collection. After you have finalized your goals, it's time to start looking for your data. Data collection is the process of gathering information on targeted variables identified as data requirements. The emphasis is on ensuring accurate and right data is collected. Data collection starts with primary sources, which are also known as internal sources. This is typically structured data gathered from CRM software, ERP systems, marketing automation tools, and others. These sources contain information about customers, finances, gaps in sales, etc. Under external sources, you have both structured and unstructured data. So if you're looking to perform a sentiment analysis towards your brand, you would gather data from various review websites or social media apps. The next step is to clean the data. The data which is collected from various sources is highly likely to contain incomplete, duplicate and missing values. So you need to clean these unwanted redundant data to make it ready for analysis. So to generate accurate results, analytics professionals must identify duplicate and anomalous data and other inconsistencies that could skew the analysis. According to a report, 60% of data scientists say most of the time is spent cleaning the data, while 57% of data scientists say it's their least enjoyable task. Now the fourth step in the process is data exploration and analysis. Once data is cleaned and ready, you can go ahead and explore the data using data visualization and business intelligence tools. You can also use various data mining and predictive modeling techniques to analyze the data and build models. You can use different supervised and unsupervised algorithms such as linear regression, logistic regression, decision tree, KN&N, C means clustering, and lots more to build prediction models for making business decisions. And the final step is to interpret the results. This part is important because it's how a business will gain actual value from the previous four steps. Interpreting the results will help you find unseen trends and patterns in the data and gain insights. You can have a validation check if the results are answering your questions. These results can be shown to your clients and stakeholders for better understanding and business collaboration.

Now that we have looked at the various steps involved in data analytics, let's now see the different tools that can be used to perform the above steps. So as you can see, we have seven tools including a few programming languages that will help you perform analytics better. Now let's discuss them one by one. First we have Python. Python is an object-oriented open-source programming language that supports

A range of libraries for data manipulation, data visualization, and data modeling. Python programmers have developed tons of free and open-source libraries that you can use. You can find many of them via the Python Package Index, which is PyPI, the repository of Python software. Python provides the default package installer called pip or pip3.

Python has libraries such as NumPy for numerical computation of data; Pandas to manipulate data on numerical tables and time series; SciPy for technical and scientific computations. It also provides scikit-learn, which is a machine learning library for creating classification, regression, and clustering algorithms. And finally, it also has PyTorch and TensorFlow for deep learning.

Up next, we have R. R is an open-source programming language majorly used for numerical and statistical analysis. It provides a range of libraries for data analysis and visualization. Some of these libraries are ggplot2, tidyr, plotly, dplyr, and carot.

Then we have Tableau. Tableau is a popular data visualization and analytics tool that helps you create a range of visualizations to interactively present the data, build reports and dashboards to showcase insights and trends. It can connect with multiple data sources and give hidden business insights and patterns.

Then we have a competitor of Tableau, which is Power BI. Power BI is a business intelligence tool developed by Microsoft that has an easy drag-and-drop functionality and supports multiple data sources with features that make data visually appealing. Power BI supports features that help you ask questions to your data and get immediate insights. You can also forecast your data for predicting future trends.

So the next tool is ClickView. ClickView provides interactive analytics with in-memory storage technology to analyze vast volumes of data and use data discoveries to support decision-making. It provides social media discovery and interactive guided analytics. It can manipulate huge data sets instantly with accuracy.

Up next, we have Apache Spark. Apache Spark is an open-source data analytics engine to process data in real time and carry out complex analytics using SQL queries and machine learning algorithms. It supports Spark Streaming for real-time analytics and SparkSQL for writing SQL queries. It also has Spark MLlib, which is a library that has a repository of machine learning algorithms, and then it has GraphX for graphical computation. And finally, we have SAS.

SAS is a statistical analysis software that can help you perform analytics, visualize your data, write SQL queries, perform statistical analysis, and build machine learning models to make future predictions. SAS empowers our customers to move the world forward by transforming data into intelligence. SAS is investing a lot to drive software innovation for analytics. Gartner has positioned SAS as a magic quadrant leader for data science and machine learning.

Moving on to the applications of data analytics. Data analytics is being used in almost every sector of business these days. Let's discuss a few of them.

First, we have retail. Customers expect retailers to understand exactly what they need and when they need it. Data analytics helps retailers meet those demands. Retailers not only have an in-depth understanding of their customers, but they can also predict trends, recommend new products, and boost profitability. Retailers create assortments based on customer preferences, invoke the most relevant engagement strategy for each customer, optimize supply chain and retail operations at every step of the customer journey.

The second application is in healthcare. Healthcare industries analyze patient data to provide life-saving diagnosis and treatment options. They also deal with healthcare plans and insurance information to drive key insights. Using analytics, they can discover new drugs and come up with new drug development methods. Advanced analytics allows healthcare companies to improve patient outcomes and experience. Cancer cells and diabetic retinopathy can be discovered using medical imaging.

At number three, we have manufacturing. For manufacturers, problem-solving is nothing new. They fight with difficult problems and situations on a daily basis. From complex supply chains to motion applications to labor constraints and equipment breakdowns, they deal with such problems on a regular basis. Using data analytics, manufacturing sectors can discover new cost-saving and revenue opportunities.

The fourth application is related to the banking sector. Banking and financial institutions collect vast volumes of structured and unstructured data to derive analytical insights and make sound financial decisions. Using analytics, they can find out probable loan defaulters, customer churn rate, and detect fraudulent transactions immediately.

The final application is based on logistics. Logistics companies use data analytics to develop new business models that can ease their business and improve productivity. They can optimize routes to ensure delivery reaches on time in a cost-efficient manner. They also focus on improving order processing capabilities as well as performance management.

With that, now let's look at the companies using data analytics on a daily basis. So we have the e-commerce giant Amazon. Then we have Accenture, followed by the American healthcare service organization Sigma. Then we have the American supplier of health information technology solutions, services, devices, and hardware, Cerner, followed by Target and antivirus company McAfee. Next, we have Rapido, which is an Indian bike rental company based in Bangalore. After that, we have Flipkart and the world's largest retail company, Walmart.

With that, let's understand a case study from Walmart and how it uses data analytics to grow its business and serve its customers better. Walmart is an American multinational retail company that has over 11,500 stores in 27 countries worldwide, and it has e-commerce websites in 10 different countries. It has more than 5,900 retail units operating outside the United States with 55 banners in 26 countries, with more than 7 lakh associates serving more than 100 million customers every week. It has over 2.2 million employees around the world and 1.5 million employees in the United States alone. Walmart's e-commerce branch alone employs more than 3,000 technologists from Silicon Valley to India, England, and South America. More than 240 million customers shop at Walmart each week online and at its banner stores. Walmart.com sees up to 100 million unique visitors a month, according to comScore, and is growing every year. Walmart collects over 2.5 petabytes of data from 1 million customers every hour. That's really huge.

Now, to make sense of all this information, Walmart has created Data Cafe, a state-of-the-art analytics hub located within its Bentonville, Arkansas headquarters. Here, over 200 streams of internal and external data, including 40 petabytes of recent transactional data, can be modeled, manipulated, and visualized. Teams from any part of the business are invited to bring their problems to the analytics experts and then see a solution appear before their eyes on the nerve center's touchscreen smartboards. Walmart also constantly analyzes over 100 million keywords to know what people near each store are saying on social media to understand customer behavior on what they like and dislike. Walmart uses modern tools and technologies to derive business insights and improve customer satisfaction. Some of these tools include Python, SAS, NoSQL databases such as Cassandra and Hadoop.

Now, using all these technologies and data analysis techniques, Walmart can better manage its supply chain, optimize product assortment, personalize the shopping experience, give relevant product recommendations, and finally optimize and analyze transportation lanes and routes for its fleet of trucks.

With that, let's jump into our use case demo where we will predict the sales based on advertising expenditure using the linear regression model in R. The advertising expenditure has been made via different mediums such as radio, television, and newspaper. We will use the R programming software to implement the demo.

So why R? Well, R is a free and open-source software that can be downloaded from the CRAN website. It is easy to learn and use. Our language is built specifically for performing statistical analysis, data manipulation, and data mining using packages such as plyr, dplyr, tidyr, and lubridate. R supports data visualization with the help of packages such as ggplot2, googleVis, RColorBrewer, leaflet, and ggmap. And finally, the R software can be used in a wide range of analytical modeling including classical statistical tests, linear and nonlinear modeling, data clustering, time series analysis, and more.

Now, let's have a look at our data that we will be using for this demo. Here is our advertising CSV data set which has four columns. You can see there's TV ads expenditure. The next column is for radio ads. Then we have the newspaper ads. And the last column is our target column, that is the sales. So the data set has in total 200 rows.

Now to understand the data, let me give an example. So consider the second row. So suppose you spend around $230 on TV ads, then $37.8 on radio ads and $69.2 newspaper ads, you can expect to sell nearly 22 units of a particular product. Similarly, if you are spending $44.5 in TV advertising, $39.3 in radio ads and $45 in newspaper ads, you can sell around 10 units of a certain item. We will analyze this data using linear regression.

So linear regression is a supervised learning algorithm, which means the data has labeled columns and is used to predict numeric continuous variables. So our sales column here is the target column and it has continuous numeric variables.

Now let me go to the RStudio and start with the demo. So first I'll create a new file. Then I'll select R script. The next step is to install all the necessary packages that we need for this demo. If you already have the packages installed in your RStudio, you need not do it again. You can just call these packages using the library function and pass the package names.

So first I will install the dplyr package, which is used for data manipulation. I'll be using the install.packages function and I'll give the package name. So I'll type install.packages. If you hit tab, it will autocomplete. Then, under quotations, I'll write dplyr. I'm not going to run this because I already have it installed in my RStudio. The next step, I'll write I'll call this uh package using the library function. I'll give the package name dplyr. I'll run this. Then I'll install the broom package. It takes the messy output of built-in functions in R such as linear model or lm, then t-test, and turns them into a tidy data frame. So I'll copy the above code. I'll just paste it again. I'll change it to broom. Here also I'll change it to broom. I'll run this. Okay, then I'll be installing the car package, which will help us build our linear regression model. I'll paste the same code and I'll take car, and I'll call that using the library function. I'll run this. Now sometimes people face issues with installing this particular package. If you also face this problem, do visit the RStudio community page. Now let me show it to you. So this is the RStudio community page, and here they have the solution. You can just go through these two pages. All right. After this, I will install the ggplot2 package, which is a very popular package in R for data visualization. I'm not running install.packages because I have already installed all these packages before. If you have not, so you have to run install.packages first and then call the library function.

With that, let's now load the data set. For this, I will use the read.csv function and provide the path location where my data is located, followed by the data set name and the extension. I'll assign the loaded data set to a variable. Let me now go ahead and show you where my data set is located. So here is my advertising CSV data set, and this is the location. I'll copy this location. I'll move back to RStudio, and let me comment this line. Load the data set. So I'll take a variable name ads and then using read.csv function I'll pass the path location where the data set is present. Now one thing to note is we have to change all the backslash to forward slash, otherwise R won't accept it. And finally, I'll give the data set name which is advertising.csv. Let me run it. Okay, we have successfully loaded our data set. Now let us look at how our data set looks like using the head function. So I'll give a comment, display the head of the data set. I'll be using the head function and I'll pass ads. I'll run it. So you can see the head function has displayed the first six rows from the advertising data set. Let me now check the dimensions of the data set. So I'll use the dim function. Uh, it will give you the total rows and columns present in the data set. Give a comment. Check the dimensions. I'll use the dim function and I'll pass in the ads variable. You can see it has given the number of rows, which is 200, and the total columns, which is four. Now if you want to get a summary of the data set, you can use the summary function. So I'll directly type in summary and I'll give ads. Let me expand this. So actually, the summary function gives you information about a few statistics for each of the columns. So you can see the minimum value for each column, the maximum value for each column, the mean, the median, first quartile, and the third quartile values. The first quartile or lower quartile is the value that cuts off the first 25% of the data when it is sorted in ascending order. The second quartile is the median, which has the value that cuts off the first 50%. And the third quartile or the upper quartile is the value that cuts off the first 75% of data.

Moving ahead, let's do some data visualization. Now to visualize our data, since our data has only numeric values, using scatter plots would be the best option. So we will visualize our sales against each of the independent variables. For that, I will use the plot function and give sales in my x-axis and the independent variable names in the y-axis. Let me now do that. So I'll give a comment, data visualization. First I'll use the plot function, and then in x-axis using the dollar symbol I'll give sales. Then in the y-axis I'll give my independent variable. You can see R is automatically giving you the suggestions. I'll select TV. Then I'll take type = under quotes I'll give P, which stands for points, and I'll take the color as red. So you can see under plots we have our scatter plot. If I zoom in, you can see the red dots are pretty much aligned in one direction, which means if you are increasing the expenditure on TV ads, the units sold are also increasing equally. So the more you spend on TV ads, the more sales you can expect. I close it. Now let's look at how sales vary based on radio advertising expenditure. I'll copy this, paste it, and under y-axis I'll change it to radio and I'll take the color now as let's say blue color. I'll run it. If I to zoom in. Now if you look at the blue dots, it is not that linear compared to our previous graph. You can see there are a few data points like this that show the sales were not good even after spending decent money on radio ads. But still you can expect a decent amount of sales if you are willing to spend on radio advertising. Close it. Let's now look at how sales will vary based on the newspaper advertising expenditure. I'll change the radio to newspaper column and this time I'll take color as green. I'll run it. Let me zoom in. You can see the plots are very haphazardly present. The data is completely nonlinear, and there seems to be a low correlation between the sales and newspaper advertising expenditure. Now, if you want to look at these plots at a time, you can use the pairs function. So, I'll type pairs and then pass in my variable name, which is ads. I'll run it. Let me zoom in. So, this is our plot, and you can see this has all the visualizations. So you can see the TV sales. Now you can see the sales that were made with radio expenditure and with the newspaper expenditure as well. I'll close it.

Moving ahead. Let's check the correlation between the variables and see what insight we can get. We will use the cor function or co function and build a correlation matrix. First let me go ahead and install the corrplot package. So I'll give a comment, correlation analysis. For this I will have to install the corrplot package. I have already got it installed. Then I'll call this function using library. I'll run it. You can see corrplot, the version has been uploaded. Now I'll tell you how you can grab only the numeric columns. Now our data only has numeric columns. But still let me tell you how you can do it since correlations are based on numeric columns only. This can be done using the sapply function. So for that we have already installed the dplyr library. I'll give a variable name as num.cols, which is numeric columns. Then I'll pass in the apply function. I'll give the ads variable and I'll check if the variable is numeric or not. So I'll use is.numeric. Let me run it. And now let's display what's there in num.cols. You can see it says TV, it's true, which means TV has numeric values. Even radio has numeric values. Similarly for newspaper and sales also. Then I'll use the correlation function, which is cor, to display the correlations between the variables. So I'll give my variable name as cor.data and then I'll take the cor function, pass in the ads variable and I'll only filter out the numeric columns. So comma numeric columns means we need all the rows and the selected columns. Let me run it. And now to display, let me call cor.data again. So this is our correlation output. As you can see the correlation values are all above zero, which means there is a positive correlation between the variables and the change in one of the independent variables will have a positive impact on the sales numbers. TV ads have the maximum correlation with sales and the value is around 78. Then there is radio advertising which has a correlation of about 0.57 with sales and newspaper ads have the lowest correlation compared to the other two, which is at 0.22. Now you can also build a correlation matrix using the corrplot method. This will give you a visual representation of the correlation between the variables. So let's see how we can do that. I'll type corrplot and I'll give cor.data and I'll pass a method as color. If I zoom in, you can see this is our correlation matrix. On the right, you can see the scale. So -1 is for negative correlation. Then there's light red, 0, which is almost white color. Then there's light blue. And finally dark blue for the maximum positive correlation. The diagonals are dark blue, which represents the same variables as in rows and in columns. So it's dark blue. TV ads and radio ads have the next highest correlation while newspaper ads have the lowest correlation with sales.

With that, let's jump into the most important part of this analysis, which is building our regression model. First, we will look at a simple linear regression model where we will take one input variable, that is TV ads. I'll be using the lm function or the linear model function to build the model. So I'll give a comment, simple linear regression. I'll take a variable name as model.simple and then using lm function I'll give my target variable which is sales and using ~ I'll give my independent variable which is TV and data as ads. I'll run it. Now that we have built our linear regression model, let's check the summary. Take summary function and I'll pass in model.simple. Let me run it. If I expand this, you can see our intercept estimate is around 7.03. So when the TV advertising budget is zero, we can expect sales to be around 7,30 or 7030. Also remember we are operating in units of thousand, and for every $1,000 increase in the TV advertising budget we can expect the average increase in sales to be around 47 units. Now the same summary can be checked using the tidy function present in the broom package. So if I call tidy and I'll give the model name which is model.simple. I'll run it. So there you go. This gives us a tidy representation of the summary figures.

Now let's build a regression model with more than one input variable. So we'll build a multiple linear regression model. I'll take my variable name as model.multiple this time and I'll use the same lm function. I'll pass in the sales and using ~ I'll take all the column names TV, then I'll use an addition operator, then I'll take in newspaper followed by the radio column and then I'll take my data as ads. Let's run it. I'll follow the same drill. Let me now call the summary function over this newly created model. So I'll write summary and I'll select my model name as model.multiple. Let me run it. So the interpretation of our coefficients is the same as in the simple linear regression model. First we see that our coefficients for TV and radio advertising budget are statistically significant, since our P-value is less than 0.05, while the coefficient of newspaper is not, which is around 0.86. Thus, changes in the newspaper budget does not appear to have any relationship with changes in sales. However, for TV ads, our coefficient suggests that for every $1,000 increase in TV advertising budget, holding any other predictors constant, we can expect an increase in sales of 45 units on average. Similarly, the radio coefficient suggests that for every $1,000 increase in radio advertising, holding all the other predictors constant, we can expect an increase of 188 sales units on an average. Now you can also call the tidy

function over this multiple linear regression model. So let me do that. I'll call tidy, and I'll pass in model multiple. You can see it has given the output.

Now you can also find the coefficients of the model using another method. It's called the coefficient matrix. Here is how you can do that. So take a variable name, and I'll use the summary function. I'll pass in model multiple, and using the dollar symbol, I'll take the parameter as coefficient. Let me call coefficient now. So these are the coefficients of different variables.

Let me now show you another example of how you can train a linear regression model using the CA tools library. First, I'll take a seed value, a random seed value of say 101. Next, I will split the data into training and testing sets. I'll take 70% for training the data and 30% for testing the data. So I'll use a variable sample. Then I'll call sample.split, split, take ads, and then I'll use another parameter called split ratio, and I'll take the split ratio as .7, which is 70%. I'll run it, and then I'll use another variable called train and take the subset of the sample. Pass in my ads variable, and I'll select sample is equal to equal to true. Similarly, I'll take another variable called test. I'll use my subset function and given the same parameters, but this time I'll take sample is equal to equal to false, which means the test sample data set won't have any values that are present in train data set. I'll run it.

Now we will use the same lm function to create our model. So I'll take model as my variable and assign it to lm function. So I'll assign the linear model to the model variable. I'll take sales as my target column, use the tild followed by a dot, which means I'm taking all the variables in terms of the independent variables, and then I'll select my train data set with that. Let's check the summary as well, so this is the summary of our newly created model.

Now you can also check the residual collector from the trained model using the residuals function. Let me go ahead and assign a variable called for residual and I'll use the residuals function. Pass in my model. Then I'll convert the residuals into a data frame. So I'll use the as.data.frame function and pass in. You can check the residuals. So these are the residual values.

Now it's time to make our predictions using the test data set. I'll use the predict function for this. Let me take another variable called sales.predictions, and I'll use the predict function. Pass in my model followed by the test data set. Now I'll run it. Then let me call sales.prediction to display the values. As you can see, these are my predicted sales values.

Now let me combine these predicted sales values to our original sales for the test data. For that, I'll use the cbind function and pass the column names. I'll take another variable called results and use the cbind function. I'll take sales.predictions and I'll consider the sales column from the test data. Let me check the values now. So you have the predicted sales values and the original values of sales. But you can see the columns don't have any name assigned to them. So let me go ahead and assign the column names using the colnames function and convert it into a data frame to make it look better. So I'll use the colnames function and and pass in my results variable. Then I'll take a vector and give the column names as spread for predicted values and let's say real for the original values. Me run it now. So I'll convert this into a data frame. So I'll use as.data.frame and give my results variable. Now if I display results, you can see go on top you can see the columns have been assigned successfully. So on the left you have the predicted values, and on the right you have the real values.

So we have successfully built our linear regression model and predicted the sales values using linear regression in R. You can also go ahead and find the accuracy of this model to know how good your model is. We won't be covering that as part of this tutorial. I'll leave it for you and encourage you to do some research on how you can find the accuracy of a linear regression model. You will come across terms such as mean squared error, root mean squared error, and R squared value. If you are able to find the accuracy, please post the results in the comment section; or if you face any issues with it, please post your queries. We'll be happy to help you.

Data scientist, you possess the power to unravel hidden patterns, predict trends, and make data-driven decisions that transform businesses. But how do you showcase your expertise to the potential employers? The answer is data science projects. Your resume is your gateway to the new opportunities, and your data science projects are the shining stars that make you stand out from the crowd. A study by McKenzie Global Institute estimated that by 2025, the potential annual value of applying data science in retail alone could be between $410 billion to $1.2 trillion. Just imagine having a remarkable collection of projects that shows your incredible skills in predictive modeling, your talent for natural language processing, your expertise in image recognition, etc. Each project is a proof to your skills and a captivating story waiting to be told. In this video, we will cover various data science projects that you can include in your resume. By including these key components of data science projects in your resume, you will not only demonstrate your technical skills but also your ability to solve real-world problems. So, are you ready to showcase your data science progress and stand out from the competition?

In this session, we will discuss five data science projects that you can include in your resume. Now, before moving forward, let me ask you a quick question. So, which of the following is not a commonly used programming language in data science? Option A is Python, option B is R, option C is Java, and option D is MATLAB. Now you can pause this video and answer in the comment section below.

Now, let's start with the first project that is fake news detection. So fake news detection project is a significant undertaking in the field of data science. It involves developing algorithms and models that can identify and classify misleading or fabricated information from genuine news sources. The project typically involves techniques such as natural language processing, machine learning, and data mining to analyze the text, content, metadata, and contextual information. Including a fake news detection project in your resume is essential for several reasons. Firstly, it showcases your ability to apply advanced and analytical techniques to real-world problems. Secondly, fake news has become an issue in today's digital age, impacting individuals, communities, and even elections. By including a fake news detection project on your resume, you show your commitment to ethical practices and contributing to the societal well-being. Lastly, working on a fake news detection project reflects your understanding of complex data science concepts such as feature engineering, model evaluation, and model deployment.

Next up we have is sentimental analysis. Sentimental analysis is a vital project in the field of data science that involves using natural language processing and machine learning techniques to identify and understand the sentiment or emotion expressed in a piece of text. This project holds great significance in various industries as it enables organizations to gain valuable insights from large volumes of textual data such as customer reviews, social media posts, and survey responses. Including a sentiment analysis project on your resume demonstrates your expertise in utilizing advanced data science techniques to extract meaningful information from unstructured text data. It showcases your ability to process and analyze data at scale. Furthermore, sentiment analysis has numerous practical applications including brand monitoring, customer feedback analysis, reputation management, and market research. By showcasing your experience in sentiment analysis, you present yourself as a valuable asset to potential employers who are seeking professionals capable of utilizing the power of data.

Next we have is Parkinson's disease detection. Parkinson's disease detection project in data science is an essential project to include on a resume due to its significance in both healthcare and technological advancements. Early detection plays a crucial role in managing the disease and improving patients' quality of life. By developing a Parkinson's disease detection project using data science techniques, one showcases a strong understanding of data preprocessing, feature engineering, and model development, which are fundamental skills in data science. This project also highlights your understanding of the complexities involved in detecting a complex disease like Parkinson's and your proficiency in utilizing advanced machine learning algorithms.

Next project on our list is resume parsing in machine learning with Python. So resume parsing in machine learning refers to the process of extracting relevant information from a resume and structuring it into a standardized format. It involves using natural language processing techniques to analyze text and identify key details such as contact information, work experience, education, skills, and other relevant data. Including a project on resume parsing in your data science resume is important for several reasons. Firstly, it showcases your expertise in NLP and machine learning techniques, highlighting your ability to extract meaningful information from unstructured text. Secondly, you understand the challenges faced by HR departments in analyzing large volumes of resumes and automating the initial screening process. This shows potential employers that you possess the knowledge and skills to streamline their hiring procedures.

Next project to include in the resume is a face recognition system in Python using Facenet. Face recognition systems are essential tools in the field of computer vision and have gained significant popularity in recent years. One popular approach is using Facenet, a deep learning model, to extract high-dimensional features from facial images and create a face recognition system in Python. Facenet, developed by Google researchers, utilizes a CNN to generate compact numerical representations known as embeddings. These embeddings can then be compared using distance metrics such as Euclidean or cosine similarity to determine if two faces belong to the same person. Including a face recognition system using Facenet in your resume showcases your proficiency in computer vision and deep learning techniques, as you have implemented a complex model for facial analysis. Additionally, it highlights your ability to work with large datasets and handle image processing tasks efficiently.

Thumbs up. I hope it really helped you all. Thanks for watching. Stay safe and keep learning. Staying ahead in your career requires continuous learning and upskilling. Whether you're a student aiming to learn today's top skills or a working professional looking to advance your career, we've got you covered. Explore our impressive catalog of certification programs in cutting-edge domains including data science, cloud computing, cyber security, AI, machine learning, or digital marketing. Designed in collaboration with leading universities and top corporations and delivered by industry experts. Choose any of our programs and set yourself on the path to career success. Click the link in the description to know more.

Hi there. If you like this video, subscribe to the SimplyLearn YouTube channel and click here to watch similar videos. To nerd up and get certified, click here.