Data Science Python | Data Science Python For Beginners | Data Science Python Project | Simplilearn

hello everyone. Welcome to this full course video on data science with Python. In this video tutorial, we will cover everything from basics to advanced that will help you become a data scientist. Our experienced instructors, with immense industry experience, will take you through this video. You will first understand the basics of data science in 5 minutes from an animated video. You will then learn the important libraries in Python, such as NumPy, Pandas, and Matplotlib.

You will get an idea about the various data science concepts along with mathematics, statistics, and linear algebra. You will look at the essential algorithms like linear regression, logistic regression, decision trees, random forests, k-nearest neighbors, and k-means clustering. You will also learn about reinforcement learning and Q-learning techniques. Finally, you will learn how to become a data scientist in 2021. So let's begin.

Are you one of the many who dreams of becoming a data scientist? Keep watching this video if you're passionate about data science, because we will tell you how does it really work under the hood. Emma is a data scientist. Let's see how a day in her life goes while she's working on a data science project. Well, it is very important to understand the business problem first. In our meeting with the clients, Emma asks relevant questions, understands, and defines objectives for the problem that needs to be tackled. She is a curious soul who asks a lot of questions—one of the many traits of a good data scientist.

Now she gears up for data acquisition to gather and scrape data from multiple sources like web servers, logs, databases, APIs, and online repositories. Oh, it seems like finding the right data takes both time and effort. After the data is gathered, comes data preparation. This step involves data cleaning and data transformation. Data cleaning is the most time-consuming process as it involves handling many complex scenarios. Here Emma deals with inconsistent data types, misspelled attributes, missing values, duplicate values, and what not. Then in data transformation, she modifies the data based on defined mapping rules. In a project, ETL tools like Talend and Informatica are used to perform complex transformations that help the team to understand the data structure better.

Then understanding what you actually can do with your data is very crucial. For that, Emma does exploratory data analysis. With the help of EDA, she defines and refines the selection of feature variables that will be used in the model development. But what if Emma skips this step? She might end up choosing the wrong variables, which will produce an inaccurate model. Thus, exploratory data analysis becomes the most important step. Now she proceeds to the core activity of a data science project, such as data modeling. She repetitively applies diverse machine learning techniques like KNN, decision tree, naive Bayes to the data to identify the model that best fits the business requirement. She trains the models on the training data set and tests them to select the best performing model. Emma prefers Python for modeling the data; however, it can also be done using R and SAS. Well, the trickiest part is not yet over.

Visualization and communication. Emma meets the clients again to communicate the business findings in a simple and effective manner. To convince the stakeholders, she uses tools like Tableau, Power BI, and QlikView that can help her in creating powerful reports and dashboards. And then finally, she deploys and maintains the model. She tests the selected model in a pre-production environment before deploying it in the production environment, which is the best practice, right? After successfully deploying it, she uses reports and dashboards to get real-time analytics. Further, she also monitors and maintains the project's performance. Well, that's how Emma completes the data science project. We have seen the daily routine of a data scientist is a whole lot of fun, has a lot of interesting aspects, and comes with its own share of challenges.

Now let's see how data science is changing the world. Data science techniques, along with genomic data, provide a deeper understanding of genetic issues and reaction to particular drugs and diseases. Logistic companies like DHL, FedEx have discovered the best rules to ship, the best-suited time to deliver, the best mode of transport to choose, thus leading to cost efficiency. With data science, it is possible to not only predict employee attrition but to also understand the key variables that influence employee turnover. Also, the airline companies can now easily predict flight delay and notify the passengers beforehand to enhance their travel experience.

Well, if you're wondering, there are various roles offered to a data scientist like data analyst, machine learning engineer, deep learning engineer, data engineer, and of course, data scientist. The median base salaries of a data scientist can range from 95,000 to 165,000. So that was about the data science. Are you ready to be a data scientist? If yes, then start today. The world of data needs you. With promising career opportunities, it's a great time to become a data scientist. The field of data science is seeing abundant opportunities. Companies now know the importance of data. The main job of a data scientist is to collect, process, and analyze the data.

One fundamental thing that a data scientist must know is statistics. Before learning any other programming language, you must have a thorough understanding about statistics. A data scientist needs to have an understanding about variance, mean, and mode. There is no requirement to learn complex statistics; start with basic statistics. Along with this, a data scientist must have an idea about statistical optimization. On top of that, you should have a basic understanding about linear algebra and calculus. Data analytics: it focuses on summarizing data in a meaningful way. This area of data science makes use of reducing methods to extract data, anticipate the future, and identify trends. Both machine learning and traditional statistics are included in this. Tools data analytics includes feature engineering and also data wrangling.

Programming languages: you need to know some important programming languages as a data scientist. The time period required to learn a programming language may be from one month to two months, but it is mandatory to know some of the important programming languages that I am mentioning below. Python is the best programming language for any data scientist. It is easy to learn as well. Python is implemented in developing machine learning algorithms, sort and categorized data. R is a prominent language in the field of data science. R enables the design of several statistical models. Statisticians use them to perform regression. Java is a favorite programming language for any developer. Java machine learning is used to implement machine learning algorithms. Popular Java libraries are used in data mining, machine learning, and deep learning. We need to use different IDEs for programming. The best IDE for Python is PyCharm and also Visual Basic Code. The best one for R is RStudio, and for Java, you can go for Eclipse, NetBeans, etc. Please install all the software in your computer system and start practicing.

Data visualization: the graphic display of information and data is called data visualization. You must know certain tools involved in data visualization that includes Tableau. Tableau, one of the widely used software for data visualization. Tableau, CBON can also use for data visualization. Web scraping: web scraping is the process of extracting data from websites. Data scientists apply this technique to extract data from different websites. You can scrape data from any e-commerce website like Amazon, Flipkart, Mantra, etc., and use it for your projects. A data scientist must have a general understanding about it. Tools like Scrapy. Scrapy is an open-source platform and it is used to extract data from a website. It is easy, quick, and compatible. This is something that every data scientist must be familiar with. Next comes ParseHub, base used by data scientists for research. It is a powerful visual-based web scraping tool which enables everyone to create their own data extraction workflows. Next comes Beautiful Soup. Beautiful Soup is not a tool; it is a library to extract data from different sources. It is a Python package used to parse HTML and XML text.

Machine learning: automatically generates and evaluates enormous amounts of data without involvement of human beings. It automates data analysis and generates predictions in the present. As a data scientist, you need to learn every machine learning algorithm thoroughly and understand the importance of it. Five major steps involved in machine learning are: data collection—it is the first step. It is crucial to get accurate and pertinent data because the quantity and quality of the data have a direct impact on how well your machine learning model performs. Data preparation: the second step is data preparation. Data preparation is a crucial step in getting suitable data for analysis. Data preparation ensures that there are no incorrect or inaccurate data points in the data sets. Training the model: the output value is predicted using the training data set. Next comes evaluation of the model. Model evaluation is the process of using different evaluation metrics to understand a machine learning model's performance as well as its strength and weaknesses. Lastly, prediction. Now that your model is trained and evaluated, it doesn't mean it is perfect and it is ready to deploy. The model is further improved by using different parameters. For better understanding how to use all this subject we can take up a use case. We can take up a use case of taxi vehicles. A taxi service company like Ola can use GPS to track the position of their vehicles in order to perform optimization of parking based on the number of vehicles. Ola cabs can redistribute the cars in advance. Each new ride is assigned to different vehicles. This is implemented using statistical analysis. For data extraction, we can use their site and implement web scraping with different tools like Scrapy and libraries like Beautiful Soup. We can take up a data set of vehicles and perform data analysis. Also, for data visualization, we can use Matplotlib or Tableau. In order to cluster, machine learning algorithms can be implemented in this process.

Python is the most widely used programming language today when it comes to solving data science tasks and challenges. Python never ceases to surprise its audience. Most data scientists out there are already leveraging the power of Python every day. Hi, I'm a baker from Simplilearn, and well, after some thought and a bit more research, I was finally able to narrow down my choice of top Python libraries for data science. What are they? Let's find out. So let's talk about this amazing library, TensorFlow, which is also one of my favorites. So TensorFlow is a library for high-performance numerical computations with around 35,000 GitHub comments and a vibrant community of around 1500 contributors, and it's used across various scientific domains. It's basically a framework where we can define and run computations which involves tensors, and tensors we can say are partially defined computational objects again where they will eventually produce a value. That was about TensorFlow. Let's talk about the features of TensorFlow. So TensorFlow is majorly used in deep learning models and neural networks where we have other libraries like Torch and Theano also, but TensorFlow has hands-down better computational graphical visualizations when compared to them. Also, TensorFlow reduces the error largely by 50 to 60 percent in neural machine translations. It's highly parallel in a way where we can train multiple neural networks and multiple GPUs for highly efficient and scalable models. This parallel computing feature of TensorFlow is also called pipelining. Also, TensorFlow has the advantage of seamless performance as it's backed by Google. It has quicker updates, frequent new releases with the latest of features. Now let's look at some applications. TensorFlow is extensively used in speech and image recognition, text-based applications, time series analysis and forecasting, and various other applications involving video detection. My favorite thing about TensorFlow is that it's already popular among the machine learning community and most are open to trying it, and some of us are already using it.

Now let's look at an example of a TensorFlow model. In this example, we will not dive deep into the explanation of the model as it is beyond the scope of this video. So here we're using the MNIST dataset, which consists of images of handwritten digits. Handwritten digits can be easily recognized by building a simple TensorFlow model. Let's see how. When we visualize our data using the Matplotlib library, the inputs will look something like this. Then we create our TensorFlow model. To create a basic TensorFlow model, we need to initialize the variables and start a session. Then after training the model, we can validate the data and then predict the accuracy. This model has predicted 92% accuracy, let's see, which is pretty well for this model. So that's all for TensorFlow. If you need to understand this tutorial in detail, then you can go ahead and watch our deep learning tutorial from Simplilearn as shown in the right corner. Interesting, right? Let's move on to the next library.

Now let's talk about a common yet a very powerful Python library called NumPy. NumPy is a fundamental package for numerical computation in Python. It stands for Numerical Python. As the name suggests, it has around 18,000 comments on GitHub with an active community of 700 contributors. It's a general-purpose array processing package in a way that it provides high-performance multi-dimensional objects called arrays and tools for working with them. Also, NumPy addresses the slowness problem partly by providing these multi-dimensional arrays that we talked about and then functions and operators that operate efficiently on these arrays. Interesting, right? Now let's talk about features of NumPy. It's very easy to work with large arrays and matrices using NumPy. NumPy fully supports an object-oriented approach. For example, coming back to ndarray once again, it's a class possessing numerous methods and attributes. Ndarray provides for larger and repeated computations. NumPy offers vectorization. It's more faster and compact than traditional methods. I always wanted to get rid of loops, and vectorization of NumPy clearly helps me with that. Now let's talk about the applications of NumPy. NumPy, along with Pandas, is extensively used in data analysis, which forms the basis of data science. It helps in creating the powerful n-dimensional array. Whenever we talk about NumPy, the mention of the array we cannot do it without the mention of the powerful n-dimensional array. Also, NumPy is extensively used in machine learning when we are creating machine learning models, as in where it forms the base of other libraries like SciPy, scikit-learn, etc. When you start creating the machine learning models in data science, you will realize that all the models will have their basis in NumPy or Pandas. Also, when NumPy is used with SciPy and Matplotlib, it can be used as a replacement of MATLAB.

Now let's look at a simple example of an array in NumPy. As you can see here, there are multiple array manipulation routines like there are basic examples where you can copy the values from one array to another. We can give a new shape to an array from maybe one dimensional; we can make it as a two-dimensional array. We can return a copy of the array collapsed into one dimension. Now let's look at an example where this is a Jupyter notebook, and we will just create a basic array. For detailed explanation, you can watch our other videos which target on these explanations of each library. So first of all, whenever we are using any library in Python, we have to import it. So now this np is the alias which we will be using. Let's create a simple array. Let's look what is the type of this array. So this is an ndarray type of array. Also, let's look what's the shape of this array. So this is a shape of the array. Now here we saw that we can expand the shape of the array. So this is where you can change the shape of the array using all those functions. Now let's create another using arrange functions. If I give arrange 12, it will give me a 1D array of 12 numbers like this. Now we can reshape this array to 3, 4, or we can write it here itself. So this is how our range function and the reshape function works for NumPy.

Now let's discuss the next library, which is SciPy. So this is another free and open-source Python library extensively used in data science for high-level computations. So this library, as the name suggests, stands for Scientific Python, and it has around 19,000 comments on GitHub with an active community of 600 contributors. It is extensively used for scientific and technical computations, also as it extends NumPy. It provides many user-friendly and efficient routines for scientific calculations. Now let's discuss about some features of SciPy. So SciPy has its collection of algorithms and functions which is built on the NumPy extension of Python. Secondly, it has various high-level commands for data manipulation and visualization. Also, the ndimage function of SciPy is very useful in multi-dimensional image processing, and it includes built-in functions for solving differential equations, linear algebra, and many more. So that was about the features of SciPy. Now let's discuss its applications. So SciPy is used in multi-dimensional image operations. It has functions to read images from disk into NumPy arrays, to write arrays to disk images, resize images, etc. Solving differential equations, Fourier transforms, then optimization algorithms, linear algebra, etc. Let's look at a simple example to learn what kind of functions are there in SciPy. Here I am importing the constants package of the SciPy library. So in this package, it has all the constants. So here I am just mentioning c, or e, or g, and this library already knows what it has to fetch, like speed of light, Planck's constant, etc. So this can be used in further calculations.

Data analysis is an integral part of data science. Data scientists spend most of the day in data munching and then cleaning the data. Also, hence mention of Pandas is a must in data science life cycle. Yes, Pandas is the most popular and widely used Python library for data science, along with NumPy and Matplotlib. The name itself stands for Python Data Analysis, with around 17,000 comments on GitHub and an active community of 1200 contributors. It is heavily used for data analysis and cleaning as it provides fast, flexible data structures like data frames, series which are designed to work with structured data very easily and intuitively. Now let's talk about some features of Pandas. So Pandas offers this eloquent syntax and rich functionalities like there are various methods in Pandas like dropna, fillna, which gives you the freedom to deal with missing data. Also, Pandas provides a powerful apply function which lets you create your own function and run it across a series of data. Now forget about writing those for loops while using Pandas. Also, this library's high-level abstraction over low-level NumPy, which is written in pure C, then it also contains these high-level data structures and manipulation tools which makes it very easy to work with Pandas like their data structures and series. Now let's discuss the applications of Pandas. So Pandas is extensively used in general data wrangling and data cleaning. Then Pandas also finds its usage in ETL jobs for data transformation and data storage as it has excellent support for loading CSV files into its data frame format. Then Pandas is used in a variety of academic and commercial domains including statistics, finance, neuroscience, economics, web analytics, etc. Then Pandas is also very useful in time series specific functionality like date range generation, moving window, linear regression, date shifting, etc.

Now let's look at a very simple example of how to create a data frame. So data frame is a very useful data structure in Pandas, and it has very powerful functionalities. So here I'm only enlisting important libraries in data science. You can explore more of our videos to learn about these libraries in detail. So let's just go ahead and create a data frame. I'm using Jupyter Notebook again, and in this, before using Pandas, here I am importing the Pandas library. Let me go and run this. So in data frame, we can import a file, a CSV file, Excel files; there are many functions doing these things, and we can also create our own data and put it into a data frame. So here I am taking random data and putting in a data frame. Also, I'm creating an index and then also giving the column names. So pd is the alias we've given Pandas. Random data of 6x4, index which is taking a range of six numbers, and column name I'm giving as abcd. Now let's go ahead and look at it. So here it has created a data frame with my column names abcd, my list has six numbers zero to five, and a random data of six by four. So data frame is just another table with rows and columns where you can do various functions over it. Also, I can go ahead and describe this data frame to see. So it's giving me all these functionalities where count and mean and standard deviation, etc. Okay, so that was about Pandas.

Now let's talk about the next library and the last one: Matplotlib. For me, it's the most fun library out of all of them, why? Because it has such powerful yet beautiful visualizations. We'll see in the coming slides. Plot and Matplotlib suggests that it's a plotting library for Python. It has around 26,000 comments on GitHub and a very vibrant community of 700 contributors. And because of such graphs and plots that it produces, it's majorly used for data visualization and also because it provides an object-oriented API which can be used to embed those plots into our applications. Let's talk about the features of Matplotlib. The pyplot module of Matplotlib provides a MATLAB-like interface. So Matplotlib is designed to be as usable as MATLAB with an advantage of being free and open source. Also, it supports dozens of backends and output types, which means you can use it regardless of which operating system you're using or which output format you wish. Pandas.

itself can be used as wrappers around matplotlib's API, so as to drive matplotlib via cleaner and more modern APIs. Also, when you start using this library, you will realize that it has a very little memory consumption and a very good runtime behavior.

Now let's talk about the applications of matplotlib. It's important to discover the unknown relationship between the variables in your data set. So this library helps to visualize the correlation analysis of variables. Also, in machine learning, we can visualize 95 percent confidence interval of the model, just to communicate how well our model fits the data. Then mat modeler finds its application in outlier detection using scatter plot, etc., and to visualize the distribution of data to gain instant insights.

Now let's make a very simple plot to get a basic idea. I've already imported the libraries here. So this function `matplotlib inline` will help you show the plots in the Jupyter Notebook. This is also called a magic function. I won't be able to display my plots in the Jupyter Notebook if I don't use this function. I am using this function in NumPy to fix random state for reproducibility.

Now I'll take my n as 30 and will assign random values to my variables. So this function is generating 30 random numbers. Here I'm trying to create a scatter plot, so I want to decide the area. Let's put this. So this is multiplying 30 with random numbers to the power 2, so that we get the area of the plot, which we will see in just a minute. So using the scatter function and the alias of matplotlib as plt, I've created this. If I don't use this and I have very small circles as my scatter plot, it's colorful; it's nice. So that's one very easy plot. I suggest that you explore more of matplotlib, and I'm sure you will enjoy it.

Let's create a histogram. So I'm using my the style as ggplot and assigning some values to these variables, any random values. Now we are assigning bars and colors and alignment to the plot, and here we get the graph. So we can create different types of visualizations and plots and then work upon them using matplotlib, and it's just that simple.

So that was about the leading Python libraries in the field of data science. But along with these libraries, data scientists are also leveraging the power of some other useful libraries, for example, like TensorFlow. Keras is another popular library which is extensively used for deep learning and neural network modules. Keras drafts both TensorFlow and Theano backends, so it is a good option if you don't want to dive into details of TensorFlow. Then scikit-learn is a machine learning library; it provides almost all the machine learning algorithms that you need, and it is designed to interpolate with NumPy and SciPy. Then we have Seaborn, which is another library for data visualization. We can say that Seaborn is an enhancement of matplotlib as it introduces additional plot types.

Let's start with what is NumPy. NumPy is the core library for scientific and numerical computing in Python. It provides high-performance multi-dimensional array object and tools for working with arrays. And I'll go a step further and say there are so many other modules in Python built on NumPy. So the fundamentals of NumPy are so important to latch on to for the Python, so you can understand the other modules and what they're doing. NumPy's main object is a multi-dimensional array. It's a table of elements, usually numbers, all of the same type, indexed by a tuple of position integers. In NumPy, dimensions are called axes. Take a one-dimensional array, or we have—remember dimensions are also called axes—you can say this is the first axis: 0, 1, 2, 3, 4, 5. And you can see down here it has a shape of six, why? Because there's six different elements in it in the one-dimension array, and they usually denote that as six comma with an empty node on there. And then we have a two-dimensional array where you can see 0, 1, 2, 3, 4, 5, 6, 7, and in here we have two axes or two dimensions, and the shape is 2, 4. So if you were looking at this as a matrix or in other mathematical functions, you can see there's all kinds of importance on shape. We're not going to cover shape today, but we will cover that in part two.

Did you know that NumPy's array class is called ndarray for NumPy data array?

Now we're going to take a detour here because we're working in Python, and two of my favorite tools in Python is the Jupyter Notebook, and then I like to use that sitting on top of Anaconda. And if you flip over to jupiter.org—that's j-u-p-y-t-e-r.org—you can go in here; you can install it off of here if you don't want to use the Anaconda Notebook. But this is the Jupyter setup; the documentation on the Jupyter. Jupyter opens up in your web browser; that's what makes it so nice is it's portable. The files are saved on your computer; they do run in IPython or IronPython, and you can create all kinds of different environments in there, which I'll show you in just a minute. I myself like to use Anaconda—that's www.anaconda.com. If you install Anaconda, it will install the Jupyter Notebook with the Anaconda separate, and you can install Jupyter Notebook, and it'll run completely separate from Anaconda's Jupyter Notebook.

And you can see here I've now opened up my Anaconda Navigator. What I like about the Navigator, and this is a fresh install on a new computer, which is always nice, I can launch my Jupyter Notebook from in here; I can bring other tools. So the Anaconda does a lot more, and under environments I only have the one environment, and I can open up the terminal specific to this environment. This one happens to have Python 3.7 in it, the most current version as of this tutorial, and the open terminal. If you're going to do your pip installs and stuff like that for different modules, you can also create different environments in here, so maybe you need a Python 3.6, Python 3.5. You can see we're having a nice framework; like Anaconda really helps so you don't have to track that on your own.

In the Jupyter Notebook, in your different Jupyter Notebook setups, we'll go ahead and launch this Jupyter Notebook, and then I've set my browser window for a default of Chrome, so it's going to open up in Chrome. And you can see here this opens up a folder on my computer. We have a couple different options on here. Remember, I set the environment up as Python 3.7. You would install any additional modules that aren't already installed in your Python on this, and it keeps them separate, so you do have to, for each environment, install the separate module so they match the environment on there. And in here we have a couple things we can look up what's running; you have your different clusters. Again, this is—I just installed this on a new machine, so I just have the one; a couple things in here that were run on here recently. And what we go on here is we then have on the upper right, new, and from the pull-down menu you'll see Python, Python 3, and this will open up a new window, and now we're in Jupyter Python. So this is a Python window, and we'll just do a print, and this, of course, is Hello World, and we'll run that, and it prints out Hello World. In the command line, there's a couple special things you have to know; we're not going to do today, which is on graphics. If you've never seen this, one of the things you can do, you can also do a = Hello World, and if you just put the a in there, now if you do a bunch of these, we have a = Hello World, b = Goodbye World, and you put a, b, a, and return b, it'll only run the last one, but you can see here if you put the variable down here, it will show you what's in that variable, and that has to do with the Jupyter Notebook inline coding. So that's not basic Python; that's just Jupyter Notebook shorthand, which you'll see in a little bit.

So back to our NumPy. NumPy array versus Python list; Python list being the basic list in your Python. Why should we use NumPy array when we have Python list? Well, first, it's fast. The NumPy array has been optimized over years and years by multiple programmers, and it's usually very quick compared to the basic Python list setup. It's convenient, so has a lot of functionality in there that's not in the basic Python list, and it also uses less memory, so it's optimized both for speed and memory use. And let's go ahead and jump into our Jupyter Notebook, since we're coding. Best way to learn coding is to code, just like the best way to learn how to write is write, and the best way to learn how to cook is cook. So let's do some coding here today. And just like any modules, we have to import NumPy; we almost always import it as np; that is such a standard, so you'll see that very commonly. We can just run that, and now we have access to our NumPy module inside our Python. And then the most common thing, of course, is to go and create a number array, and in here we can send it a regular list, and so we'll go ahead and send this a regular array. Let's do 1, 2, 3 to make it simple, and then I'm just going to type in a, and we'll run this. As you can see down here, the output is an array of 1, 2, 3. And we could also do print, just a reminder that this is an inline command, so that wouldn't work if you're using a different editor. You can see that it's an array 1, 2, 3, but we'll go and leave it as a kind of a nice feature so you can see what you're doing really quick in the Jupyter Notebook. And just like all your other standard arrays, I can go a[0], which is going to be a value of 1. Of course, we do a[1]; you go all the way through this; I have 1 has a value of 2 in it. So whether you're using the NumPy array or the basic Python list, that's going to be the same; that should all look pretty familiar and be pretty straightforward. Remember, the first value is always 0, and when we set on there.

So let's take a look why we're using NumPy, because we went over the slide a little bit, but let's just take a look and see what that actually looks like and what we want to look at is the fact that it's fast, convenient, and uses less memory. So let's take a glance at that in code and see what that actually looks like when we're writing it in Python and what the differences are. And to do this, I'm going to go ahead and import a couple other modules. We're going to import the `time` module so we can time it, and we're going to import the `sys` module so that we can take a look at how much memory it uses. And we'll go and just run those, so those are imported. So we'll do b = [0 range(1000)], yeah, 1,000 is fine, and so that's going to create a list of 1,000, 0 to 999. Remember, it starts at 0 and it stops right at the 1,000 without actually going to the 1,000. And let's go ahead and print, and we want `sys.getsizeof()`, and we'll pick any integer because we have, you know, 0 to 1,000, we'll just throw 1 in there, 5, it doesn't matter because it's gonna—whatever integer we put in is going to generate the same value because we're looking at the size of how—how much memory it stores an integer in, and then we want to have the length of the b; that's how many integers are in there. And if we go ahead and execute this and run this in a line, we'll see—oops, I did that wrong—comma. If we multiply them together, we'll see it generates 28,000. So that's the size we're looking at is 28,000; I believe that's bytes, that sounds about right. So let's go ahead and create this in NumPy, and we'll go with c = np.arange(), so that's the NumPy command, do the same thing that we were just doing in a list, and we'll also use the same value on there, the 1,000. And then once we've created the c value of c for np.arange(), let's go ahead and print, and we can do that by doing c.size * c.itemsize, when it's very similar we did before; we did get the size of, so the c.size is the size of the array, and each item size just reversed, so it's the size of an integer. 5 itemsize, it's going to be the integers and c.size. Now let's just take a look and see what that generates, and wow, okay, we got 4,000 versus 28,000. That's a significant difference in memory, how much memory we're using with the array. And then let's go ahead and take a look at speed. Let's do um—oh, let's do size; we tried this with lower values, and it would happen so fast that the np.arange kept coming up with 0 because I just rounded it off. So size, and let's create an l1 = list(range(size)), and we'll do an l2 = list(range(size)) on there. There we go. And then we can do a1 = np.arange(size), and then let's do an a2 = np.arange(); we'll keep it the same size. And what we're going to do is we're going to take these two different arrays and we're going to perform some basic functions on them. But let's go ahead and just load these up now. We'll go ahead and run this, so those are all set in memory except for the typo here; quickly fix that. There we go. So these are now all loaded in here. And let's do start = time.time(), so it's just going to look at my clock time and see what time it is. And then we'll do result = , and let's do—oh, let's say we got an array, and we're going to say let's do some addition here: x + y for x, y in zip(l1, l2). There we go. So here's our two different arrays; we're going to multiply each of the individual things on here, l1, l2. There we go. So that should add up each value, so l1 + l2, each value in each array. Then we want to go ahead and print, and let's say Python list took, and then we'll do time.time(); we'll just subtract the start out of there. So time—oops, I messed up on some of the quotation marks on there—okay, there we go—time - start, and we'll convert that to seconds, so we'll go—because in milliseconds or times 1,000—and let's hit the run on there. This is kind of fun because you also get a view while we're doing this of some ways to manipulate the script. And as you can see, also my bad typing. There we go. Okay, so we'll go ahead and run this, and we can see here that the Python list took 34—actually, I have to go back and look at the conversion on there—but you can see it takes roughly 0.34 of a second. And we go ahead and print the result in here too; let's do that. We'll run that just so you can see what the—what kind of data we're looking at, and we have the 0, 2, 4, 6, 8, so it's just adding them together; it looks pretty straightforward on there. And if we scroll down to the bottom of the answer again, we see Python list took 46, a little different time on there depending on what core, because I have—this is on an eight-core computer, so it just depends on what core it's running on, what else is pulling on the computer at the time. And let's go back up here and do our start time, paste that into here, and this time we're going to do result = , and this is really cool; notice how elegant this is, so straightforward. This is a lot of reason people started using NumPy is because I can add the two arrays together by simply going a1 + a2; it makes a lot of sense, both looking at it, and it's just very convenient. Remember that slide we're looking at: fast, convenient, and less memory. So look how convenient that is, really easy to read, really easy to see, and I don't know if we don't need to print the result again, so let's just go ahead and print the time on here, and we'll borrow this from the top part because I really am a lazy typer, and this isn't the Python list; this is the NumPy list or number array. And let's go ahead and see how that comes out, and uh, we get 2.99. So let's take a look at these two numbers: 46 versus 2.99. So we'll just round this up to 3. That's a huge difference; that's—that's like more than 10 times faster; that's like 15 times roughly at a quick glance; I'd have to go do the math to look at it, and it's going to vary a little bit depending on what's running in the background, the computer obviously. So we've looked at this, and if we go back here, we found out it's much faster; yes, there's different—going to be different speeds depending on what you're doing with the array, very convenient, easy to read, and it uses less memory. So that's the core of the NumPy; that's why a lot of people base so many other modules on NumPy and why it's so widely used.

So we did glance at a couple operations when we were looking at speed and size. Let's dive into a little bit more into the basic operations, and these are always nice to see. I mean, certainly you want to go get a cheat sheet if you're using it for the first time; you know, look things up; Google is your friend. We did this with the most basic np.array(), and we'll go ahead and create an array. Let's do pairs: [1, 2], and then let's do [3, 4], and if we're gonna do that, let's do [5, 6]. There we go. And if we go ahead and take this and run this, I can go ahead and do our a down here, so it's inline, and I'll print that out. You can see it makes a nice array for us. So we have a, and if you look at that, we have three different objects, each with two values in them, and hopefully you're starting to think, well, how many dimensions or indexes is that? And you'll see 3 x 2. So let's go ahead and take a look, and let's go—how about a.ndim? Speaking of which, we'll run that, and we have two dimensions for each object. And then we can do the itemsize, so a.itemsize; we saw this earlier; we looked up how many items it was up here where we wanted to multiply itemsize times the actual size of the object, so the memory is being used versus the itemsize, and we should see 4 there; memory is compressed down; that's always a good thing. And then the shape; the shape is so important when you're working with data science and you're moving it from one format to another. So we have our shape; we just talked about that; we have 3 x 2, three rows by two objects in each one. Generally, I don't look too much at the size, but the dimensions I'm always looking up. This is nice; you can automate it, so you might be converting something; you might need to know how many dimensions are going into the next machine learning package so that you can automatically just have it send that information over. So we looked at a.shape; let's go and create a slightly different array: np.array([1, 2, 3, 4, 5, 6], dtype=np.float64). And so what we've done is converting all of these into a float, and we've typed in a, and now instead of having 1, 2, 3, 4, 5, 6, you see they're all float values: 1.0; there's no actual 0 in there; just here's the 1., the 1.2, 3., 4., 5., 6.. And this again, data science—I don't know how many times I've had to convert something from an integer to a float so that's going to work correctly in the model I'm using. So very common features to be aware of and to be able to get around and use. And we'll also do—let's just curiosity—itemsize; we'll go ahead and run that, and we see that it doubled in size. So it's not a huge increase—well, doubling is always a big increase in computers—but it's not a

Huge increase compared to what it would be if you're running this in the Python list format. And then we did the shape earlier without having it set to float64. Let's go ahead and do a shape with it set to 64, and it should be the same: 3, 2, so it all matches. So we've gone through and remember, if you really—if this is all brand new to you—according to the Cambridge study at Cambridge University, if you're learning a brand new word in a foreign language, the average person has to repeat it 163 times before it's memorized. So a lot of this you build off of; so hopefully you don't have to repeat it 163 times, but we did manage to repeat it at least twice here, if not a little bit more.

Let's go ahead and take this; we're going to look at one more setup on here, and let me just take this last statement here on converting our properties of our data. Instead of float64, let's do complex. Let's just see what that looks like, and let's go and print that out and run it. And so we now have a complex data set up, and you'll see it's denoted by the 1. + 0.j. And if we flip over here and do a basic search for NumPy data types—better to go to the original web page—but pull up a bunch of these, you can see there's a whole list of different NumPy data types. Shorthand complex, we have complex, complex64, complex128; complex number represented by 2 64-bit floats, real and imaginary components; one option on there, float16, float32, float—shorthand for float64, most commonly used—and of course, all the different ones that you can possibly put into your NumPy array.

So we covered a basic addition up there; we're comparing how fast it runs, but some very basic components: how to set up a NumPy array, how many dimensions it has, item size, data type, item—again, we want item size—and there's also the shape, probably one of the more used. I used shape all the time; very commonly used. And then down here you can see where we actually created a NumPy complex data type.

So let's look at some other features in NumPy. One of them is you could do numpy.zeros, and we're going to do 3, 4. There we go, and we'll go ahead and run this, and you can see if I do np.zeros, I create a NumPy array of zeros. This is really important. I was building my own neural network, and I needed to create an array where I initialized the weights, and I want them all to be the same weight. In this case, I wanted them to start off as zero for the particular project I was working on. And there's other options that you can do: numpy.ones, and we'll do the same thing: 3, 4. We'll run that, and you can see I've created an array of NumPy ones. In this case, it comes out as a float array, and this is interesting to note because we have—let's go back to our Python and do `lrange(5)`, and we'll print the `l`, so as our list. And if I run that, it doesn't create the range until after the fact, until you actually execute it. That's an upgrade. Python 2.7 actually created the array 0, 1, 2, 3, 4. This one actually creates the script, and then once it's used, it then actually generates the array. And if we do that in NumPy, `arange`—remember that from before—and if we do a NumPy `arange(5)`, and let's do `l =` or we can just leave it as NumPy, that's fine. There we go, just run that. You can see there we actually get an array 0, 1, 2, 3, 4 for the value. The NumPy `arange(5)` generates the actual array.

For part one, we're going to do just one more section on basic setup, and we're going to concatenation—do a concatenation example. There we go. We're going to do strings. Let's take a look at strings; what's going on with there. And let's do—oh, let's see—`print`, let's do an `np.char`—something new here—and we're going to add, and then here's our brackets for what we're going to add. Oh, and let's say—let's do `hello`, `,` `hi`, and in the brackets on there, let's create another one, and this one's going to be `abc`, and we'll do `xyz`. So we're just creating some—randomly making some up on here—and then we'll go ahead and just print this. If we run that and come down here, and of course, make sure all your brackets are open and closed correctly, and then you can see in here when we concatenate the example in NumPy, it takes the two different arrays that we set up in there, and it combines the `hello` with the `abc` and the `hi` with `xyz`. And if we can also do something like `print`, oh, let's do `np.char.multiply`. So there's a lot of different functions in here; again, you can look these up; it's probably good to look them all up and see what they are, but it's good to also just see them in action. Let's do `hello `, `,` `3`, and we'll run this one, and run that without the error. You'll see it does `hello hello hello`, so we multiplied it by 3. And we can also—let's just take this whole thing here instead of retyping it—and we can do `char.center`. So instead of `multiply`, let's do `center`, and over here keep our `hello` going, take the space out of there, and let's do `center` at 20, and `fillchar =` we'll fill it with dashes. So if we run this, you can see it prints out the `hello` with dashes on each side.

And we keep going with that; we can also, in addition to doing the fill function, we can play with `capitalize`, we can `title`, we can do `lowercase`, we can do `uppercase`, we can `split`, `splitlines`, `strip`, `join`. These are all the most common ones, and let's go ahead and just look at those and see what those look like, each one of them here. We're going to do the `hello world`—all-time favorite of mine; I would like to say `hello universe`. And you can see here we did a capital H with the world, but so we want to `capitalize`. So `capitalize` is the first one in the array, so we get `hello world` on there. And we can also take this, and instead of `capitalizing`, another feature in here is `title`, and let's just change this to `How are we doing? How are you doing?` Instead of `do you`, let's run that, and you can see here, because we created as a title, it capitalizes the first letter in each word. And in this one, we're going to do `char.lower`. Two different examples here: we have an array, we have `HELLO WORLD` all capitalized, and we have just `hello`, and you can see that one is an array and one is just a string. If we run that, you get a an array with `hello world` lowercase and `hello` lowercase. And if we're going to do it that way, we can also do it the opposite way. There's also `upper`, and let's paste those in there, and you can see here we have `char.upper`. Opposite there: `Python.data`, and that will do `PYTHON IS EASY`. Hopefully you're starting to get the picture that most of the Python and the scripting is very simple; it's when you put the bigger picture together and starts building these puzzles, and somebody asks you, hey, I need the first letter capitalized unless it's the title, and then we have you start realizing that this can get really complicated. So NumPy just makes it simple, and we like that. And so in this case, we did `python data`; it's all uppercase: `PYTHON IS EASY`—like shouting in your messenger: `PYTHON IS EASY`! And then if you're ever processing text and tokenizing it, a lot of times the first thing you do is we just split the text. And we're just going to run this: `np.char.split('Are you coming to the party?')`. If we do that, it returns an array of each of the individual words: `Are`, `you`, `coming`, `to`, `the`, `party?`, splitting it by the spaces. And then if we're going to split it by spaces, we also need to know how to split it by lines. Just like we have the basic `split` command, we also have `splitlines`. `hello\nhow are you doing?`. And you'll see here the scoop in for our new line. And when we run that, if you're following the split part with the words, you should see `hello`, `how are you doing?`; the two different lines are now split apart.

Let's just review three more before we wrap this up: commonly used string variable manipulations. We have `strip`. And in this case, we have `nina admin anita`, and we're going to strip `a` off of there. Let's see what that looks like. And then you end up with `nin diminish it`. Basically, it takes up all leading and trailing letters. In this case, we're looking for a—more common would be a space in there—but it might also be punctuation or anything like that that you need to remove from your letters and words. And if we're going to strip and clean data, we also need to be able to reformat it or join it together. So you see here we have a `char.join`. We'll go ahead and run this, and it has—on the first one, it splits these letters up by the colon, and the second one by the dash—and you can see how this is really useful if you're processing—in this case, a date—we have `day:month:year`, `year-month-date`. Very common things to be have to always switch around and manipulate depending on what they're going into and what you're working with. And finally, let's look at one last character string. We're going to do `replace`. If you're doing misinformation, this is good—pulling news articles—`replace` is. And what, in this case, we're just doing: "He's a good dancer", and we're going to replace `'s'` with `'was'`. And you can see here: "He was a good dancer." Hopefully that's not because he had a bad fall; he just was from, like, you know, the 1920s and has gotten old. So there we go.

We covered a lot of the basics in NumPy as far as creating an array—very important stuff here when you're feeding it in—how do we know the shape of it, the size of it, what happens, we convert it from a regular integer into a float value, as far as how much space it takes. We saw that that doubled it: `itemsize`. You have your `n` dimensions, and probably the most used is `shape`. And we'll cover more on `shape` in part two, so make sure you join us on part two; there's a lot of important things on shaping in there and setting them up. We also saw that you can create a zeros-based array, you can create one with ones. If we do a range, you can see how it is a lot easier to use to create its own range or a range as it is in NumPy. You saw how easy it was to add two arrays; we saw that earlier—just a plus sign—then we got into doing strings and working with strings and how to concatenate. So if you have two different arrays of strings, you can bring them together. We also saw how you can fill, so you can add a nice headline: `---`. We saw about `capitalize` the first letter; we saw about turning it into a title, so all the first letters are capitalized; doing lowercase on all the letters, upper for all the letters—just lower and upper—nice abbreviation. We also covered how to split the character set, how to strip it, so if you want to strip all the a's out from leading a's and ending a's or spaces, you can do that very easily. Also how to join the data sets, so here's a `char.join` option for your strings. And finally, we did the `char.replace`.

Pandas really is a core Python module you need for doing data science and data processing. There's so many other modules that come off of it; there, it actually sits kind of on NumPy. So if you've already had our NumPy array, hopefully you've already gone through the NumPy tutorial one and two. So today we're going to cover: What is Pandas? We'll discuss Series; we'll discuss basic operations on Series; then we'll get into a data frame itself, basic operations on the data frame, file-related operations on a data frame, visualization, and then some practice examples. Roll up our sleeves and get some coding underneath there.

Let's start with just some real general: What is Pandas? Pandas is a tool for data processing which helps in data analysis. It provides functions and methods to efficiently manipulate large data sets. Now this is a step down from, say, using Spark or Hadoop in big data, so we're not talking about big data here, but we are talking about Pandas when there is some connections—there's like an interface going on with that—so there is availability, but you really should know your Pandas because if you're working in big data, you'll know there's data frames. Well, Pandas is a data frame primarily; it has a couple different pieces we'll look at here. And if you've never worked with data frames before, a data frame is basically like an Excel spreadsheet. You have rows and columns; you can access your data either by the row or the column when you have an index and different that kind of setup. And we'll dig more into that as we get deeper into Pandas, but think of it as like a giant Excel spreadsheet that's optimized to run on larger data on your computer. And then I said that it's a data frame, so the data structures in Pandas are Series—one-dimensional arrays—and then we have data frame—two-dimensional array—and it really centers around the data frame; the Series just happens to be part of that data frame.

Here's a closer look at a Pandas Series. Series is a one-dimensional array with labels. It can contain any data type, including integers, strings, floats, Python objects, and more. So it's very diverse. If you remember from NumPy, we studied—they had to be all uniform—not in Pandas. And Pandas, we can do a lot more, and Pandas actually kind of sits on NumPy, so you really need to know both of those. If you haven't done the NumPy tutorials, and you can see here we have our index: 1, 2, 3, 4, 5, and then our data: a, b, c, d, and e. Very straightforward; it's just two columns, and we have a nice index label and a column label for the data. And then a data frame is a two-dimensional data structure with labels. We can use labels to locate data, and you can see here we had—if we go back one—we had our index 1, 2, 3, 4, 5. So in each one of these Series, they would share the same index over there, the row index. So you have your row index: `df.index`, and then you have a column index: `df.columns`. And this should look—like I said—this would be really familiar if you've done any work with spreadsheets—Excel—so it kind of resembles that. This does make it a lot easier to manipulate data and add columns, delete columns, move them around—same thing with the rows—so you have a lot of control over all of this.

Now we're, of course, going to do this in our Jupyter Notebook. You can use any of your Python editors, but I highly suggest—if you haven't installed Jupyter and haven't worked with it—it is probably one of the best ways for easily displaying a project you're working on. I skip between a lot of different user interfaces or IDEs for editing my Python, and it's just simply jupiter.org (j-u-p-y-t-e-r.org). And then I always let mine sit on Anaconda (anaconda.com). And just real quick, we'll open that up for you. Oops, offline mode—don't show me that again—but you can see here that I have different tools that I can actually install in my Anaconda, including the Jupyter Notebook, which comes by default, and then I have access to the environments. And again, that's anaconda.com—named after the very large—one of the largest world's largest snakes—and then Jupyter Notebook—in this case, jupiter.org. And when we're in our—I'm going to go in here to our Jupyter Notebook—and we're going to go ahead and just do New and a Python 3. And this will open up a Python Python3 untitled folder.

Diving right in, let's go ahead and give this a title: `Pandas Tutorial`. And we'll go up to Cell, and we'll change the cell type to Markdown, so it doesn't execute it as actual code—one of those wonderful tools when you have Jupyter Notebooks—you can do demos with this—and let's go ahead and import Pandas. And usually people just call it `pd`; that has become such a standard in the industry. So we'll go ahead and run that. Now we have our Pandas has been imported into our Jupyter Notebook, and then we can go ahead and—let me do the Control +—since it's Internet Explorer, I can enlarge it very easily, so you have a nice pretty view. Oops, too big. There we go. And whenever you're working with a new module, it's good to check your version of the module. In Pandas, you just use the—in this case—`pd.__version__`. That's actually pretty common in most of our Python modules; there's different ways to look up the version, but that's one of the more common ones. And we'll go ahead and run that. We get 0.23.4. And if we go to the Pandas site, we see 0.23.4 as the latest release. And of course, a reminder that if you're going to environment, you need to install it, so you'll need to `pip install pandas` if you're using the pip installer. We're going to close out of that.

The first thing we want to do is we're going to work with Series. A lot of stuff you do in Series you can then do on the whole data set. We need to do what? Create one; we need to manipulate it, take pieces of it, so query it, query it, delete—so you can delete different parts of it—so we want to do all those things with the Series, and we'll start with the Series, and then almost all the code—in fact, all the code—does transfer right into the actual data table. So we go from a Series of a single list of one column, and then we'll take that and we'll transfer that over to the whole table. And we'll start by creating—let's put up there—creating a Series from list. And let's just call this `arr =`, and we'll do `[0, 1, 2, 3, 4]`. If you remember from our last one, we could easily do `r = range(5)`, which would be 0 to 4, but we'll do `r = [0, 1, 2, 3, 4]`, and we'll call this `s1 = pd.Series(arr)`. `Series` is capitalized; this one always throws me—is which letters do you capitalize on these modules? They're getting more and more uniform, but you got to watch that with Python. And we're just going to go ahead and do `arr`, so we're just going to take this Python list and we're going to turn it into a Series. And then because we're in Jupyter, we don't have to put the print statement; we can just put `s1`, and it'll print out this Series for us. And let's go ahead and run that and take a look, and you'll see we have two rows of numbers. So the first one is the index; now it automatically creates the index starting with 0 unless you tell it to do differently. So we get 0 index row 0, 0, 1, 1, 2, 2, 3, 3, 4, 4. And because it's a Series, it doesn't need a title for the column; there's only one column, so why title it? And this also lets you know that it's a data type of `int64`. So we print this out; this is our Series, our basic Series we've just created.

Let's do a second Series: `pd.Series(arr, index=order)`. And it helps if we actually give it an order, so we'll do `order = [1, 2, 3, 4, 5]`. So instead of starting with zero, we're going to give it an order starting with one. We're going to run that, and we'll go ahead and print it out down here: `s2`, and we'll see that we now have an index of 1, 2, 3, 4, 5, and that represents 0, 1, 2, 3, 4 in the Series, and we're still data type `int64`. And very common, as you're missing with NumPy arrays, is we can import our NumPy as `np`. Remember that from our NumPy tutorials. We can go ahead and create a NumPy out of random with the random numbers of five, and let's just see what that `end` looks like, so we see what our number looks like. So we have some nice random float values here: 2.33, so on. And that's from our...

Last tutorial, the NumPy tutorial 1 and 2. And instead of calling it order, let's call it index. We're going to set our index equal to a, b, c, d, and e. I want to show you that the index doesn't have to be an integer, so it can be something very different here.

Then let's go ahead and create our—we'll just use s2 again—and here's our np for NumPy series. Capital S and N is our np for NumPy, pd for pandas. There we go, switching my anachronisms. So we have pd.series of n, and we want to do our index equals our index we just created. Then let's go ahead and see what that looks like. s2 is a print. Let's run that, and we can see here we have a nice series going on: a, b, c, d, and e for our indexes. So instead of it being 0, 1, 2, 3, or 4, we can make this index whatever we want. You can see the numbers here going down that we randomly generated from the number array. So we use NumPy to create our Pandas series right here.

So continuing on with creating our series—this one I use so often—we create a series from a dictionary. So we have our dictionary; in this case, we went ahead and did a of one, b is two, c a three, d four, e of five. So each one of those is a key and then a value. And then we're gonna use—oh, let's use s3 equals pd for pandas series—and then we want to go ahead and just do d in here. Print out s3 here. Let's go ahead and run this, and you can see we got a is one, b is two, c is three, d is four, e is five. And it's still of integer 64 because the actual data is 1, 2, 3, 4, 5, and it's all integer 64 type 64.

And the last thing we want to do in the creating section of our series is to go ahead and modify the index because we're going to start modifying all this data. So let's start with modifying the index of the series. And if you remember, let's do a print this time. s1. I'll go ahead and run this. The reason I did print is because it only prints out the last variable. So if I put s1 up here and we're going to do another variable back down lower, it won't print the first one, just the last one. And we're going to go ahead and take s1, the index, and we're just going to set it equal to a new index. Obviously, the number of objects in our index has to equal the number of objects in our data. And then because it's a last variable, we can go ahead and just do an s1. Let's run that, and you can see how we went from 0 to 0, 0, 1, 2, 3, 4 as our index; we've now altered it to a, b, c, d, and e. So this would be much more readable or might be representational of a larger database you're working with. So cool tools we've covered: creating database based on our basic array, Python array. We've showed you how to reset the index. Then we showed you how to use a NumPy array, so you can put a NumPy array in there; it's all the same, you know, pd.series a NumPy array, and then we can set the index on there. The same thing with the dictionary, so it's very versatile how it pulls in data, and you can pull in data from different sources and different setups and create a new series very easily in the Pandas. And then we left on changing your index, so now we have a new index on here.

Then we want to go ahead and do some selection. Let's do some basic slicing—most common thing you'll probably do on here—and we'll just do s1. This notation should start to look really familiar again. This is going to put an output, so I'd usually—it doesn't change s1; this just selects it. So we might do a equals s1 and then print a, and you'll see that it just looks at the first three: 0, 1, 2. And we can do the same thing by not having the a in there. I'll go ahead and take that out, but just a reminder that it's not actually changing s1; it's just viewing s1. So simple slicing on here. And we can likewise do an append. Oops, before we do append, let's just do a quick kind of fun one. We'll do 2 -1, and you'll see it covers everything but the e. Of course, you can do -2 on this side, so one—another way to select it is to go how far from the end—and likewise we can do a 2 here, a cde to the end, so it starts at the second one. Another way we can do this is we can do a -2 over here, and that looks at just the last two in the slice. So you can see how easy it is to slice the data. And of course, there's no reason to do this, but you could select all of them if you wanted to view all of them on there. Helps 32; there's not 32, so it's just going to show the first three. There we go. And then we can also append, so I can take and—oh, let's create another series and append one to it—and if you remember we had s3, there's our s3, and we have our s1. We'll go ahead and do s1, and let's go ahead and do—oh, let's call it s4 equals s1.append s3. So we're just going to combine those two into s4. And if we go ahead and print s4 on here, you'll now see that we have a, b, c, d, e, a, b, c, d, e, 0, 1, 2, 3, 4, 1, 2, 3, 4, 5. Because we started the data at one, it's a very easy to append one series to the next. And if we're going to append one series to the next, we need to go ahead and drop or delete one, and drop is the keyword for that. Let's just do e or index e. And so if I run this, you'll see that it'll print it out, and a, b, c, d; there's no e. And remember all these changes—if I type in s4 again, you'll see that s4 still has e in it. So this change does not affect the series unless you tell it to. So I'd have to do like x s4 equals s4.drop e. And there's another way to do that, which we'll show you later on. Let me just cut this one out. There we go. Alright, so we've covered all kinds of cool tools here: we have appending, we have slicing, we did all the creating stuff earlier, as you can see here on the setup, how easy it is to manipulate the series.

Next, what we want to get into is we want to get into operations that happen on the series. Let me go ahead and change this cell to markdown. There we go, and run that. So series of operations: what can we do with the series? And let's start by creating a couple arrays. We'll call it one, and we'll do 0 through 7, and array two: 6 through 6, 7, 8, 9, 5. I don't know if we threw the five on the end. Let's go ahead and run those. So those load up into Jupyter, and uh, we'll do this a little backwards. We're going to do s5 equals a Pandas series of array two. So I'm doing this in reverse, and then when we do s5, you'll see that we have 0 to 4; it automatically assigned the index 6, 7, 8, 9 for our series. And let's go ahead and do the same, and we'll call this s6, and we'll set this equal to pd series for our first array. And if we do an s6 down here to print it out, we'll see something similar. I got 0 through 6: 0, 1, 2, 3, 4, 5, 7 for the data. So those are two series we just created: series six, five and six. And one of the first things we can do is we can add one series to the next. So I can do s5.add s6, and let's see what that generates. Just a quick thing: if you never use Pandas, what do you think is going to happen with the fact that this only has five different values in it and this one has seven values? So let's see what that does, and we end up with 6, 8, 10, 12, 9, and it goes—oh, I can't add this; there's nothing there—so it gives us a null return. Very different than the NumPy that would have given you an error; this instead tells you there's no value here because we couldn't generate one. So we can easily add s5.add s6, and likewise we can do s5 sub for subtract s6, and we'll run that. On the add, the subtract, and you guessed it, we're going to do multiply and divide next. Again, you can see there's the null values where it can't subtract the two; there's no values there to subtract. We can also do s5 multiply, and mul—they're all three letters on these—that's one of the ways to remember how they figured out the code for this. So remember these are all three letters: mole. We'll go ahead and run this, and you again you can see how they're multiplied together. And then we can also do the s5 div—three letters again—s6, and run that, and you'll see here this goes to infinity because we have zero in the wrong position, so it actually gives you a whole different answer here that's important to notice. And then in the null values because there's no data and it can't actually produce an answer off of an old—off of missing data. And since we're in data science, let's do s6 median. So let's look at the median data, which is simply—uh, median—sorry for those who are following the three letters because median is not three letters—and you can see an s6 is 3.0. And let's do a print here, and we'll do median or average s6, and let's print max, comma s6. Just like median, there's max value, and if we're going to have a max value, we should also have a minimum value. So let's pop in minimum. We'll go ahead and run this, and you're starting to see something that would be generated like, say, an R where you're starting to get your different statistics. We have a median value of 3, max value of 7, and a minimum value of 0. And what it does when it hits these null values—if there is no values in there—because we could still do that, we could actually—you know what, let's go up here and do—let's pick this one, we multiplied—let's go s7 equals. I'll go and print the s7 just so I keep it nice and uniform, so I still have my s7 down there and run it. And then I want to take the s7 because s7 now has null values and an infinity value, and let's see what happens. This is going to be interesting because I want to see what it is with infinity, and we end up with a median of 6, maximum of 27, and minimum of 0, which is correct; it drops those values. So when it gets to there and it doesn't know what to do with them, it just drops those values, and then it computes it on the remaining data on there. So it's important to know when you're making these computations, you're looking at min and max and median; you're not going to know that there's no values unless you double check your data for the null values. That's a very important thing to note on there. So just a real quick review on there: we've done our created our pd series, and we've gone ahead and done addition, subtraction, multiplication, division—all those are three letters—so sub, min, div, add—and then we looked at median, maximum, and minimum. So we're going to go ahead and jump into the next big topic, which is to create a data frame.

So now we're going to go from series, and we're going to create a number of series and bundle them together to make a data frame. There we go, cell type markdown. Let me go ahead and run that, so we have a nice title on there. It's always good to have a good title. All right, so our first data frame—we'll jump in with some stuff that looks a little complicated, but we'll break it down first. I'm going to create some dates, and you know what, let's just go ahead and do this. I want you to see what that looks like. What I'm creating here: I've created a series of dates, pd date range, and we're going to use these for the index, okay? So when you look at this, you'll see that it's just basically—it comes out kind of like a basic Python list or NumPy array, however you want to look at it—with our different dates going down, and we've generated six of them, and it's going to have whatever time it is right now on your—on the thing for the date for the time—that's that timestamp right there. And then you'll see we have 11 19 2008, 11 20, 11 19, and looking into the future there. So that's all this is: is generating a series of dates that we're going to use as our index. And this is a Pandas command, so we have a date range, which is nice; it's one of the tools hidden in there in the Pandas that you can use. And next we're going to use NumPy to go ahead and generate some random numbers. In this case, we'll do the np.random.random in 6, comma 4. You can look at this as rows and columns as we move it into the Pandas. And of course, you could reshape this if you had those backwards on your data, but we want the six to match the rows, and we have six periods, so our indexes should match along with the rows on there. And then, you know, before we do the next one, let's go ahead and just print out our NumPy array, so you can see what that looks like. Here we have it: 1, 2, 3, 4 by 1, 2, 3, 4, 5, 6; 4 by—so it's a nice little setup on there. And since working with data frames can be very visual, let's give our columns. We have four columns, and we're going to give them names: a, b, c, and d. So now we have columns on there also. And then let's put this all together in a data frame, and we can actually—you know what, let's do this—since I did it with everything else, let's go ahead and do columns, and you can see there's our columns on there. And we'll go ahead and do df1 equals pandas dot data frame, and note that the d and the f are capitalized. Series it was just the s, and I always highlight this because you don't know how many times these things get retyped when you forget what's capitalized down there. It's a minor thing; you'll pick it up right away if you do a lot of it. And the first thing we want to do is we want to go ahead and take our NumPy array because that's what we're going to create our data frame off of is the NumPy array, and then we want our index equal to our dates, so there's our index in there. And then we also have columns equals columns, and then finally let's see what that looks like. Now remember we had all the different data that just looked like a jumble of data. We have our column names and everything else; our NumPy array kind of just a jumble array over there, 4 by 6; you could sort of read it, but look how nice this looks. I mean, this is—you come into a board meeting, you're working with your shareholders—this is pretty readable; this is, you know, this is our date, this is our a, b, c, d, whatever it is. Maybe each one of these dates has your leads closures, lost leads, total dollar made, you know, whatever it is. If it's in a business, maybe it's measurements on some scientific equipment, whether searching material, you know, where this is like higher the temperature, low of the day, humidity of the date, whatever it is. So you can see that we can really create a nice clear chart, and it looks just like a spreadsheet. You know, we have our rows, and we have our columns, and we have our data in there. Now this one I use all the time. If we're going to create, we can create it like you saw here with our NumPy array; very easy to do that, and reshape it. You can also create it with a dictionary array. So here we have some data. Let me just go down a notch so you can see all the data on there. We have an animal, in this case, cat, cat, snake, dog, dog, cat, snake, cat, dog. We have the age, so we have an array of ages; we have the number of visits, and the priority: was it a high priority? Yes, no. And then we're going to take that; we're going to create some labels. We have a, b, c, d, e, f, g, h, i. And what I want you to notice on this is we have a title, animal, and then we have basically a Python list, and these lists, they don't necessarily have to be equal because we can have non-data, you know, np.nand NumPy array null value. But we want to go ahead and create labels that are equal to the number in the list, so a, the first cat, b, the second cat, c, the snake, d, the dog, and so on. So we'll go ahead and create our labels, which we're going to use as an index, and we'll call this df. Let's do it this way: we'll call this df2 equals pd for pandas data frame, and then we have our data just like we did before, and then we have our index equals labels. And if we're going to go from there, let's go ahead and print it out so we can see what that looks like: df2. So let's go ahead and run that. Another again, you have a nice, very clean chart to look at. We've gone from this mess of data here to what looks like a very organized spreadsheet, very visual and easy to read: animal, age, visits, priority, and then a through j, cats and all your different animals, so on and so on. And then when you do programming a lot of times, it's important to know what the data types are. So we can simply do df2.dtypes, and if we run that, we can see that our animal is an object because it's just a string, but it comes in as an object; age is a float 64, integer 64, and then priority again is just an object. And exploring this—this one's very popular—let's go df2.head, and if we print that out, the df2.head returns the first five, and we can change this; you don't have to do five. You might want to just look at the top two; maybe you want to look at—let's see—let's do six, so maybe we'll look at just the top six in the database in your data frame. And you can actually—this creates another data frame—so I could have a df3 equal to df2, and this now takes the df2 and just the first six values. So if we do df3, run, get the same answer. And if we do it the head of the data, we can also do the tail; it's the same thing: df.tail. You can look at the last—we'll just do the tail, which by default does five, the last five. And of course, you can just look at the last three of those real quick just to see what's at the end of the data. And this is like to tell—I love doing the tail of one because I'll have like the index or something like that, and it will just show me the last—whatever the last entry was—looking at stock values, and I might want to look at just the last five days of the stock values; I can do that with the data frame tail. And some other key things to look up are the index, so we can do df2.index, and I want you to notice that this isn't a call function, so if I put the brackets on the end, it'll give me an error because index is not callable; it's just an object in there. So we do df2.index. There's also columns, so we can go ahead and—let's do—uh—let's print this. Remember the first one is not going to show unless I print it, and then df2.column. So now we can see that we have our indexes and we have our columns listed here: df2.columns: animal, age, visits, priority; it tells you what kind of object it is or what kind of data type it is, and they're both object. And then finally, df2.values, and again there's no brackets on the end of df2.values because this is an actual object; it's not a callable function, so.

We'll go ahead and run that, and it creates just displays a nice array. A very easy way to convert this back to a NumPy array, basically.

So before I go into the next section, let's just take a quick look at what we covered so far with the data frame. We came up here; we created our data frame. We did it from a NumPy array first, setting the columns and the index. The index is setting it up; it's the same as when we set up the series, so that should look very familiar. So is the whole format: the NumPy array, the index dates, and the columns. Columns and remember in our NumPy array, we're looking at row, comma column, so six rows, four columns is how that reads in the data frame. And we went ahead and also did that from a dictionary. In this case, animal was the column name with all the date data underneath that column, and then age with that data, visits that data, priority that data, and then, of course, we added our labels in there for our index. So there's no difference in there, but it automatically pulled the column names. Important to know when you're dealing with a data frame and importing a data frame this way.

And then we did looking up dtype; we looked at head and tail, looking at your data really quick. We also did index and columns and values, and note these don't have the brackets on the end.

So the next thing we want to do is go ahead, since we're dealing with data science, is we want to go and describe the data. So we have df2.describe to do that, and we're going to manipulate it in just a minute, but let's just see what this generates. And you can see right here we have age and visits. So looking at our data from up above, let me just go all the way up here: animal, age, visits, priority, and it does a nice job generating your age versus visits, which has all the data: you have your count, your means, your standard deviation, your minimum value, 25th percentile, 50th percentile, 75th percentile, and your maximum value. So this should look familiar as a data science setup with your describe for a quick look at your data frame data.

So let's start manipulating this data frame, moving stuff around, and we'll start with transposing. And it is simply capital T for transpose. And when we run that, it flips the columns and the indexes. So now the indexes are all column names, and the columns are all indexes: animal, age, visits, priority. So if we had come in here with our data shaped wrong up above where we had a 4x6, we can quickly just swap it. If we had it backwards, not a big deal. And we can also sort our data—something that you can't do, which is more difficult to do with a lot of other packages—in the data frame; it's really easy to do. Take our data frame df2, and we're going to sort_values by equals age. And so when we run this, you'll see the default is ascending, so we have 0.5, 2, 2.5, 3, and everything else is organized. So if you look at your indexes, they've been moved around because each index, it moves a whole row, not just the one piece of data, is not being sorted. So very quick way to sort by age or different data in the data frame.

In addition to sorting it, we can also slice the data frame. So I could do df2, and this should look familiar from earlier. We'll just do one to three. So we're going to pull out—oops, it does help if I use a df instead of just d—and we're going to pull up just between 1 and 3. So we have 0, which is a; we have b, which is 1; and c, which is 2. So 1, 2, and then it does not include 3, which is the standard in Python. And we can even do something like this: we can combine them, which is always fun because remember this returns a data frame. So if I take df2.sort_values, and we'll do by equals age, this is just kind of fun, and then I'm going to slice it. There we go, double check my typing, and run it. And now you should see fa because fa are now 1 and 2 on there. So you can very quickly create a whole string on here, which narrows it, you know, that you can sort it, then slice it, and do all kinds of fun things with your data frame. We'll just go back to the original one. Run. There we go. And if we can slice it by row, we can also query the data frame. So we can do df2, and this is a little different because I'm going to create an array within an array, and in this case, we're going to look at—oh, let's do age, comma visits—so look at the different format in here. We have one to three, so we've done this by slicing by an integer value, and then on here I've done df2 age, comma visits in an array. And when I run this, you can see that we get just these two columns on here; we get age and visits. So it's a quick way to select just two columns or select a number of columns you're working with.

And if you stop there, we did the slicing. Almost identical to slice is iloc, which uses the integer location 1, 3. There's a push in pandas to move to this particular setup instead of doing just a regular slice, and that's because this can be confusing when we slice 1 to 3, and then we select age and visits. So there is a push to go ahead and move to iloc, which does the same thing. You can see here bc it's the same as up above. There's also a copy command, so we can do df3 equals df2.copy. We're just going to create a straight copy of it. And of course, if we do df3, it'll be the same as the df2 on there. So df3 equals df2.copy, and then let's do df3.isnull. So we're looking for null values, and this will return a nice map, and you'll see that everything is false except when you go up here under the cat or h; they had a null there. And so if we go to have a couple up here also underneath of—let's see—the dog. Okay, there's a bunch of nulls in here. There's d up here. So let's look at d down here, and you'll see false, true. There it is; there's our null value. So we can create a quick chart of null values. You can use this to do other things; we can leverage that null value to maybe take an average or something and fill those null spaces with data. And we can also modify the location. So here's our df3.loc, and notice this is loc, not iloc. Iloc has i for integer location; loc uses the, in this case, the variables on the left. And what we can do on here, and we'll just set this equal to 1, 5, and then let's—um—I'll pick a spot. Let's go back up here where we had—let's do f, a—let's see what are we looking at—oh, here we go—let's do f and h. And up here f is set to age of 2.0, and we find out that that's incorrect data. So we go ahead and switch to df3 equal, and then we're going to print out our df3, and if we go to f and age, it is now 1.5. So we're just changing the value in the df3, and this is changing the actual data frame. Remember, a lot of our stuff we do a slice and like it returns another data frame; this changes the actual data frame and that value in the data frame. So we've covered loc and iloc, isnull, making a copy; here's our iloc, which is equivalent of a slice, and also selecting columns.

So now we want to dive—just take a little detour here—and let's look at df3.mean(), and this is kind of nice because you can do this; you can either do this by—as you can select a single column here, by the way—you can just add the column selection right here like we did before. So we could have age look up the mean; that just creates a series. If I run that, there's our age. But if I take that out instead of selecting it, we can do the whole setup, and it has age and visits. So why doesn't it have priority or animal? Well, those are not integers, so it's really hard; they're non-numerical values. So what is the average? I guess you could do a histogram, which probably we'll look at that later on, but the only two things we can really look at is age and visits, and we have the average or the mean on the age is 3.375, and the mean on visits is 1.9. And let's do df3.visits; we'll go and steal the visits again. And remember all those different functions we looked at for a series? Well, we can do those here; we can do the sum. So if we run that, we'll see that these sum up to 19. We could also look up minimum, if you remember that from before; the minimum is one, max—so all that functionality is here. I'll just go back to summing it up and adding it all together.

So real quick, we've shown you how to take the series operations and put them into the data frame, and then we can actually—this is an interesting one—we can just do df3.sum(). Run, and you'll see the different summations on there. It just combines them. I like the way it just combines the strings on there for priority and animal. We've looked at isnull; we've also looked at copying along with the different slices which we talked about earlier. So let's talk about strings. Let's dive into the string setup on there, and let's go ahead and create a string series. string = pd.Series, and we just put it right in there. We have a, c, d, a, a, b, a, c, a; popped in a null value, cow, and al. I don't know why they picked cal and al; in the background, someone must like those animals. And of course, we can just do string. If we run that, you'll see—leave the r out—we'll get an error, but if we put it in there, you'll see that we have a simple series: 0, a; 1, c; 2, d; and it automatically indexes it 0 to 8. And then we can go string.str.lower. So when we're talking about our data frame in this case or our data series string in this case, we use the string function str, and we're going to make it lower. And if we go ahead and put the brackets on there, and you'll see that we've gone from capital A, capital C, so on to a, b, c, and b, a, c, a, c, b, a, cow, al; they were all lower case already. And of course, if you want to go lower, you can also do upper, and we'll go ahead and run that, and you can see we now have A, C, D, A, A, B, A, C, A; everything's capitalized except for the null value, which is still null. Alright, so we looked at a few basic string; you can see that string functions upper and lower. We're going to jump into a very important topic; I'm even going to give it its own header on here because it's such an important topic: what do you do with missing values? Pandas has some great tools for that, so we'll dive into those. We'll call—we'll work with df4, and if you remember the df.copy from above, we're just going to make a copy of df3. And let's just take a quick look at the data we're working with. Oops, df3; forgot the 3 on there. There we go. So here we have our cats, snakes, and dogs—hopefully not all in the same container, because that would be just probably mean to all of them. So we made a copy; we're going to be working with df4. And the reason we made a copy is we want to go ahead and fill the data, and we just simply do fillna, and then we're going to give it the value we want to put in there; we'll give it the value 4. So I can run in here, and you'll see now that df4 now has—where the NA was—is filled with a value of 4. Same thing down here. A lot of times we'll compute the mean first, so I might do mean_age equals df4, and then we want to go ahead and do age and .mean(), and then I'll do something like this: df4; I only want to select the age, and I want to fill that with the mean_h. And I run in there, and you'll see that our df4.age now has the means in there. Just a quick way of showing you how you can combine these. Let me go back to our original one. There we go, and run that. And keeping with good practices, df5 equals df3.copy(), and we'll print our df5, which should be the original one, and then on the df5 we can now drop our missing data. I'm going to simply dropna, and we're going to use how equals any. So I'm going to drop any row that has missing data in it, and you'll see we had d here with missing data and h, and then let's go ahead and see what df5 looks like when we do that. There we go, and there it is; d is gone, and so is h. So we create a new data frame off of this, missing those values. Now, if you have a lot of data, dropping values is a good way to take care of it because you don't miss some data. If you have not a whole lot of data you're working with, like the iris data set or something like that or something small, you want to start trying to find a way to fill that data in so you don't lose your computational power of the data you got. So just a quick look at processing null values or missing values: you can fill them, usually with the means; some people use median or the mode; there's different ways you can fill it. One way is means, and we can also just drop those rows. Those are the two main things we do with missing data.

Here we go; we're going to cover next—this is I so love data frames for this—file operations. It saved me so much time because they have so many different tools for bringing data in and saving data. So we're looking at the data frame file operations; it's really streamlined. I don't know how many times they'll go on to different data downloads, and they'll have pandas download standard on there just because it's so widely used. So let's start with the most common file is a CSV. So we have df3.to_csv("animal"), and let me just show you what the folder is going into right now. I have some untitled and a few things in here, but nothing labeled animal. So we go ahead and run this, and this is now saved the animal to my hard drive, and you can now see the animal folder up here. And if I—uh—let's do edit with a notepad—oh, let's open up with just a regular notepad. There we go, or Wordpad. If I open that up, you can see it's comma separated; our titles, they don't have an index on the categories on the top, and the index, comma, then all the different data is separated by commas, standard CSV file on there. And if we're going to send it to CSV, and notice the format is .2_csv, and it's just the name of the file we're sending it to. You can also put the complete path. By default, it's going to go whatever the active directory this program is running on; that's why those other folders are in there. So we have our df3.to_csv(), and then if we're going to put it in there, we want to also get it back out. And we'll call this one df_animal = pd.read_csv(). I always have to remember is to_csv and read_csv; I always want to do like a capital in there and not the underscore. We're going in here again; it's the active directory. So if I now do print out my df_animal, and let's just do the head; we only want to look at the first three lines. So if I go ahead and run this, we'll see the first three lines, and they should match up here what we saved to our CSV. So very easy to save and import from our CSV files on here. And it turns out df3 also has a .to_excel(); they actually have a lot of different formats, but you know, old-school Excel was real popular for so long, still is. We can go ahead and save it as animal.xlsx; we're going to call the sheet named sheet1, and then I can also do df—we'll call it animal2—animal2, and this one's going to come from—and the same format on here. There we go. So we still have our animal.xlsx, the sheet1; that's where it's coming from. index_col equals None, so we're not going to—we're going to suppress the indexing on the columns. na_values, and it'll just assign it 0 on up on your indexes. So if it says index_col equals None, that's what it does, and then we've added null values because there's no values in here, and we want to just make sure that they're marked as NA, and we'll go ahead and just print out the animal_animal2. There we go. And let's run that. Let's make this—let's just do the whole thing. So we'll go ahead and run that, and it probably doesn't help that I completely forgot the read, so animal2 = pd.read_excel(). There we go, excel. So now we go ahead and run it, and what we expect is happening here; we have the same data frame on here. And if I flick back to my folder, you can now see that we have the animal; one of these is in Excel, and one of these is a CSV on here. And so there's our two file types on there, and they have other formats; these are just the two most common ones used. And I don't know how many times I've had stuff from Excel I need to pull out. If you've ever played with Excel, it's a nightmare in the back end because of the way they do the indexing. So this just makes it quick and easy to pull in an Excel spreadsheet. So we looked at two different ways to bring data in and save it to files; we've looked at all kinds of different ways of manipulating our data set and slicing it and creating it for our data frame.

Let's get in there for your visualization; always a big thing at the end because, one, it lets you check to see what you did, make sure it looks right, and then also if you're going to show somebody else, it makes it very clear what's going on if they see something visual. So this is where a really important part of data science is. So let's go ahead and bring in our tools. We're going to do import numpy as np; we want to make sure we have our %matplotlib inline. This just lets Jupyter know that we're going to print it on this page. If you're using a different IDE, you don't really necessarily need that, but this does help; it displays correctly in Jupyter Notebook. And if you remember from earlier, we could create a—we're gonna call it ts—we're gonna create a pandas—which are cute, cuddly creatures—versus a panda, short for pandemonium—no. So we have ts = pd.Series(), and we're just gonna create a random setup; 50. We'll do an index; we'll set it equal to pd.date_range("today", periods=50). So the 50 should match, and I want you to notice something here: I did not import the Matplotlib library, why? Because it's already in there; pandas already has its built-in connection and interface with the Matplotlib library, so you don't have to import it. And we'll go ahead and do ts = ts.cumsum(); we're gonna do the cumulative sum. So a little reformatting there, and we'll go ahead and plot it, and let's take a look at what that looks like. So we have a nice graph here; we have the dates on the bottom; we set this up, so we have a nice range between, in this case, -4 to looks like about 2 maybe or 1; -4 and 1. So what we've done here, we plotted a basic series, just a single row of data, and we've set an index on there, but we can also do the whole data frame on there, and let's see what that looks like. So first, let's go ahead and create the data frame. We have here random numbers, so we're going to do 50 by 4, and then we'll go ahead and create columns a, b, x, and y, just because we can. index is the ts.index on there, so we're gonna use the

Same index as before, just to keep it nice and uniform. We've already generated the dates to go with it, and then we can do just like we did with the series; we can also do with the data frame: df = df.cumsum(). So we're going to sum the whole data frame, and then we'll do simply df.plot(), and let's put that in, and let's go ahead and run this. Look how easy and quick that was to generate a nice graph with all the different data on there. So we have our shared index, we have the shared columns, and then we have the different data from each one that we can easily look at and compare. So very quick way of displaying data. You can imagine if you were working in—oh, I think I mentioned stock earlier because they've been doing some analysis of stock lately—so you'd have your date down here, and then you would have stock A, stock B, stock X, Y, whatever it is, and you can put them all on one chart and see how they—what they look like next to each other. And this isn't too far off from what some of those graphs look like, and this is just randomly generated. So stock has a lot of randomness in it, which is one of the reasons I actually play with it for doing some of my models on for testing them out.

Now, there are a lot of features in pandas, so we're going to show you one more thing on here. There's some of the things like—I didn't go too deep—we looked at the top two for importing data from a CSV and from an Excel spreadsheet; showed you how to quickly plot the data. There's more settings in there you can do. We're going to do one more thing down here, and this is kind of a fun one: changes to a markdown and run that. So how would you remove repeated data using pandas? And this is where you have a data set that comes in and maybe it's a feeding from one location, and instead of noting that it's repeated the date like—oh, let's go back to stocks; that's a good visual—we have the stocks from the 23rd, and it adds another row, and it's the same row; it's importing the 23rd again and again. So now you have that data repeated three times, and you need to go back and figure out how to get rid of it. How do you track that down?

So let's start by creating a quick database—our data frame, not a database; I keep saying databases—a data frame. And we'll just make this data frame has—using our dictionary—going in this data frame only has one data series in it, which is fine. So if we do df.to_string() to print it out, you'll see a one, two, two, two, two, four, five, four, five, six, seven, so on. So how would you remove that? Well, there is a neat feature in data frames called `.shift()`, along with another feature that lets us select just certain date information, and we'll go with the `.loc` function, put that in brackets—remember that from above, `.loc`—and then in the `.loc`, let me just spread this out a little bit so it's really easy to read. In fact, I'm going to go upscale on that since we're doing some a little bit more complicated here. What you can see on this—on the `.loc`—is I have `df['a'].shift()`. This is going to shift up one by default. You can actually change this to two or three; you can even do a minus one, and it shifts the other way, but it's going to shift up by one by default, and it's going to say if that does not equal `df['a']`, then we want that. If you look down here, we had one, two, two, two, two, two. When we run this logic on here and we do the shift, it now gets rid of all the duplicates. So we went from one, two, two, two, two, four, four, five, whatever it was here, it is one, two, two, two, four, four, four, five, five, five, six, six, six, to one, two, four, five, six, seven, eight. And you'll see on the index it just deletes them out of there. So the index stays the same; obviously, you don't want the dates to change if you're working with an index-dated setup. So it just deletes those duplicates out of there. This is just a quick way to introduce you to one: the fact that you can add logic gates into here, and two: the `.loc` allows you to use `.shift()`. So there's the `.shift()` function, and then the `.loc` selects that based on true or false. Wow!

So we've actually covered a lot today in pandas. We've really covered the basics of selecting your different series out of your column, out of your data frame, how to index rows, how to slice, how to plot. Hopefully, you'll take this beyond that and start combining these different things, and you can create long strings and really explore your data, generate some nice graphs. If you're in Jupyter Notebook, it's a great demo to show others. And I didn't know this about Jupyter Notebook—you can do this in Jupyter Notebook, and then you can download—and I always—I never really look too closely at all the downloads—but you know, download as an HTML and post it to your blog. So it's got a neat feature in there. But any of this is really powerful tool—all of this is really powerful tools for doing your data science today. We're going to study the Matplotlib library and the Python code. So what's in it for you? What is Matplotlib library? Types of plots, plotting graphs and subgraphs, adding a graph inside a graph, graph parameters, title, label, legend, line graphs, line types, color and transparency, canvas, grid, and axis range, 2D plots, scatter, step, bar, fill between, radar chart, histogram, contour, image, 3D surface image, and then we'll hit a practice example: pie chart.

Let's start with what is Matplotlib library. Matplotlib library is an open-source drawing library which supports rich drawing types. It is used to draw 2D and 3D graphics, and there are so many packages in the Matplotlib library. We're going to cover the basics, and there are so many packages that sit on top of the Matplotlib library that we can't even cover them all today, but we'll hit the main one so you have a good understanding of what the Matplotlib library is and what the basics can do. You can understand your data easily by visualizing it with the help of Matplotlib library. You can generate plots, histograms, bar charts, and many other charts with just a few lines of code. And here we have some basic types of plots you can see here that we'll go into: we have the bar chart, the histogram—boy, I use a lot of histograms in my stuff—scatter plot, line chart, pie chart, and area graph. Let's start plotting them. And to do this, I'm going to be using Jupyter Notebook. You can use any of your Python interfaces for programming or scripting and running it. Of course, we here really like the Jupyter Notebook for doing basic—a lot of basic stuff—because it's so visual. And in our Jupyter Notebook, which opens up—in this case, I'm using Google Chrome—you can go up here to New, and we'll create a new Python 3 and set that up. If you're not familiar with Jupyter Notebook, we do have a tutorial that covers some of the basics of that. You'll look at any of our tutorials; I usually cover a number of them showing how to set up Jupyter and Anaconda. I myself use Jupyter through Anaconda. In fact, let's go ahead and open that up and just take a look at—see what that looks like. You can see your Anaconda Navigator; if you install it, it will automatically install the Jupyter Notebook. That also installs a lot of other things. I know some people like the Qt Console for doing Python or Spyder. I've never used them; I actually use Notepad++ as one of my editors, and then I use the Jupyter Notebook a lot because it's so easy to have a visual while I'm programming. And even simple script in Python, I'll take it from the Jupyter Notebook and then do a Save As. You can always go under File, and you can download as a Python program, so that will download it as an actual Python versus the IPython that this saves it as.

Let's go ahead and dive in and see we got going here, and let's go ahead and put Matplotlib library tutorial, and I'm going to turn this cell into a markdown so it doesn't actually run it. You can see it has a nice little title there; that's all Jupyter Notebook. And then from Matplotlib library, let's import Pyplot as plt. And then let's go ahead and just print—we'll go plt and the version. Let's go ahead and run this. So we're going to import our Pyplot module from the Matplotlib library, and we find out that we're in version 1.15.1. Always important to note the version you're in. Probably—I was reading an article that said the number one thing that Python programmers struggle with is remembering what version they're working in and making sure that they're going from one platform to the other with the same version. And if we're going to graph things, I think we need some data to graph, so we're going to import NumPy as np. Now, if you're not familiar with NumPy, definitely go back and check out our NumPy tutorial. There's so many different things you can do with it dealing with reshaping the data and creating the data. We're just going to use it to create some data for us. And there is a lot of ways to create data, but we're going to use the `np.linspace()`. So we're going to create a NumPy array, and the way you read this is we're going to create numbers between 0 and 10, and we're going to create 25 of these numbers, so we're just going to divide that equally up between 0 and 10. And if we have x coordinates, we should probably have some y coordinates, and we'll do something simple like x * x + 2. And let's just take a look; we're going to print x and print y. Let me go ahead and run this, and let's see what we got going on here. So we have our x coordinates, which is 0, 0.4, 0.83, etc., and you can look at this as an xy plot. So we have zero, we have two, we have point four one six, we have two point one seven. And just as a quick reminder, we're going to do `print(np.array((x,y)).reshape(25,2))`. And the reason I want to do this is I want to show you something here. A lot of times a program returns x, y, and it's an array of x, y, x, y, x, y, and so when you're working with the Pyplot, you have to separate it out and reshape it. So if I start off with pairs like this, I can reshape them. If I know there's 25 pairs in there, I can switch the 2 and the 25, and this is kind of goofy, but we'll do it anyways—reshape. So I'm going to reshape my 25 by 2 back to 2 by 25, and if I run that, you'll see I end up with the same output as the x, y, the two different arrays in here. And this is important that we want x and y separate again. That's all NumPy stuff, but it's important to understand that this is a format that Matplotlib library works with; it works with an array of x's, and they should match your array of y's, so each one has 25 different entities in it. And then for our basic plotting of this data, it only takes one command to draw a graph of this data, and so we use our from up here where we imported Pyplot; we take our Pyplot, and the key under there is `.plot()` for plotting a line, and then we want our x coordinates and our y coordinates, and we'll throw in 'r', and the 'r' simply means red, so we're going to draw the line in red. Let me go and run that. You can actually switch this around if you wanted to do different—there's 'b' for blue; we have a lot of fun—yellow—hard to see yellow—there we go—but we'll go ahead and stick with red. Run. And when you're doing presentations with these, try to be consistent. You know, if the business and the shareholders send you a spreadsheet and they have losses in red, use red for losses in your graph. Try to be consistent; use green for profit, for money. You don't have to necessarily use green, but it's whatever they're using—whatever the company is using—try to mirror that, that way people aren't going to be confused if you switch your data around every time. One graph has red for loss, and one graph has blue for loss; it gets really confusing. So make sure you're consistent in your graphs and your coloring. And something to know because we're going to cover this in a minute—this is your canvas size. So we have a canvas here, and what we're going to do next is we're going to look at subgraphs.

Okay, so let's take our Pyplot and create a subplot. And one of the things also to know when we're working with the Matplotlib library—I'm not setting—when I do this, this is my drawing canvas, the Pyplot. So once I've imported the Pyplot, I'm drawing my images on there—very important to know. And with the subplot, we're going to give it some different values, and we're going to represent by rows, columns, and indexes. And let's do 1, 2, 1. So it's going to be the first row, second column, and the index is like—you can stack your graphs and things like that. We don't worry too much about indexes, but rows and columns, we want to go ahead and use row one and column two. And if we're going to have one object, we should probably have two, but before we do that, we have to plot data onto the subplot. So the order is very important, and we're going to stick with our x, y, and let's do this: we're going to add in a third parameter here. Remember we did red; we're going to add '--' for dashed lines. So this plots the data into row one, column two. And if we're going to do that, let's do another one: `plt.subplot(1, 2, 2)`. And if we're going to do row one, let's do column two and index two, and this time we're going to add 'g' for green, and this denotes a style. And if we're going to set up our `plt.subplot`, there we go; `plt`—we got to go ahead and plot that—`plt.plot(y, x, 'g*')`. Oops, I messed up; this is in the wrong spot. There we go; we'll move that down here real quick because that goes in the plot part. So the subplot tells it the row, column, and index, and the `plt.plot` tells it what data—in this case, we switched them—and the color and then the style shorthand. Let's go ahead and run that, and you'll see it takes this canvas, splits it in two, and now we have two different graphs, and we have the red one with dashed lines, and we have the green one which is has a little stars going up. And if we take this and let's just—um—just for fun—let's change this and run that with an index of one. It puts them both on the same index and also gives me a warning because it's a strange way of doing two subplots. There's depreciated; there's another way to do it, but most people just ignore that warning because it's not going to go away anytime soon. Now that's using the same setup. What happens if we do—instead of this—let's change the column on here and find out what happens. And if we do the column, it didn't really like that on the setup; it just disappears. So let's keep our column as two, and let's change the row on the second one to two and run that, and you'll see again it kind of squishes everything together and causes some issues. So let's take the index. So these need a unique index, and you can see here where I made some changes; I said row two, and look what happens when I change to column two. So I now have row two, column two, index two; I squished it up here, so you could put another graph underneath is what that does. There's all kinds of different things; you really have to just play with these numbers till you get a handle on them because, you know, we have to repeat it 164 times according to Cambridge University if it's completely new to you. And you can see right here where we go three—run—there we go—but you can see it takes a little bit sometimes to play with these and get the numbers right. Hopefully, hit the wrong one; that's why—let's go three—there, three there—run—there we go. Now it's overlapping, so I have this doubled over here on the right. For now, we'll just go ahead and leave this with the where we have column and row two and the two different indexes so they appear nice and neatly side by side. And then as we just saw as we were flashing through them, we can put them on top of each other. And let me just highlight that and copy it down here, paste it down there. And here we have 1, 2, 1, and then we'll do 1, 2, 1 also for this one, and that puts the two subplots directly on top of each other, gives us that warning, and you can see we now have two different sets of data graphed on top of each other. And you can also see how it did the indexes since one of them is from 0 to 10—that's the green one—on the x axis, and the other one is from 0 to 10 on the y axis, so it took the greatest value of either one and then used those as a shared value.

Let's next look at operator description, and we'll go ahead and turn this cell into a markdown and run that so it looks nice. So `fig`, and you remember I talked about the canvas earlier; I briefly mentioned it. We're going to look a little bit more at the canvas later on, but that's what the figure is—`fig`. We're going to add axes, so we're going to initialize the subplot, add the subplot in rows and columns, and all kinds of different things with this you can do. Let's look at that code and see exactly what's going on. And I want you to notice that there's `fig`, which is the actual canvas in the Matplotlib library, and `ax` is commonly used to refer to the subplots. So we're creating subplots. You'll see `ax = plt.subplot()`. Earlier we did the Pyplot, so let's go ahead and import `pyplot` from Matplotlib and we're going to do it as `plt`. You'll see that a lot; that's really the standard in the industry is to call it `plt`, just like pandas as `pd` and NumPy array as `np`. Certainly, you can import it as whatever you want, but I would stick to the standards. And we're going to do the same graph as we did above with the Pyplot, but with the `plt`. So if it looks familiar, there's reason; we're doing this because we want to show you how the figure part works and working with the canvas goes, but we're going to do the same plot as we did before, and we'll call it `fig`, and we're going to set that equal to `plt.figure()`. So there's our figure or canvas on there. And let's create a variable called `axes`, and we're going to set that equal to `fig.add_axes([0.5, 0.1, 0.8, 0.8])`. And in this, we're going to control the left, right, the width, the height of the canvas from zero to one, and so we can go ahead—and I'm just going to put some stuff in there—I got 0.5, 0.1, 0.8. So when you're looking at this, this is a zero to one, or you could say fifty percent, ten percent, eighty, eighty percent, but it's a control; it's going to control your left and your right along with the width and the height. So the width and the height we're going to use eighty percent, and we're going to have like a little indent on the left and the right. And this should look familiar from above: `axes.plot(x, y, 'r')`. And then let's give it a color—how about red since we're recreating the same graph—let's keep it uniform. Oops, and it helps if I use `axes` instead of `axes`; I don't know where that came from. But this looks identical to the one we had up above. So here's our `axes.plot(x, y)` of red—same graph, same setup, but this time we've added a variable equal to the `figure.add_axes()`. So our `plt.figure()` is our canvas; our `axes` is what we're working in, and then our `axes.plot(x, y)`.

Again, we can draw subgraphs. Let me put that down here. [Music] Just like we did before, and a little different temptation here: we're going to fig, comma axes equals plt.subplots.

And in here, it's going to be the number of rows. We're going to do one row, and columns equals two. So if you remember before, that's what we did. We had one row with two different graphs on it. We're going to do the same thing, but know how we did this. Here's our figure, our canvas, and our axes. We're going to create actually two different axes. We're going to create row one, column two, and so axis is an array of information, so we can simply do four. Let's do x in axes. This will look familiar: x.plot. We're going to do x, comma y. We'll go ahead and make it red, keep everything looking the same. Remember nice uniform graphs; everything looks the same.

And if we go ahead and run this, you'll see we get two nice side-by-side graphs. So just as we had before, the same look, the same setup. And just for fun, let's change in columns to three. We'll run that, and now you see we'll have three on there. And let's see if we make it a little bit more interesting. We'll do in rows equals two, and you can see down here we're going to get in the tribute eriks is trying to scrunch everything together, so it does have a limit how much stuff you can put in one small space. That's important to know. You can fix that by changing the canvas size, which we'll look at in just a minute, and there's other ways to change it on here. But here we go. We can do in rows two and columns equals one. You can see two nice images right above each other. We'll go back to the original one row, two columns, side by side, left to right.

And we can also draw a picture or graph inside another graph. That's kind of a fun thing to do. It's important to note that we can layer our stuff on top of each other, which makes for a really nice presentation. So let's start by uh fig. We'll create another figure, so we're going to start over again with our canvas. We set that equal to plt.figure. So there's our new canvas. And let's do axes; we'll call it axes one and two. Axis one equals fig.add_axes. Remember this from earlier, and this here similar numbers we used before, saying how big this axis is, this figure and the axis is. So this is going to be the big axes. And let's do axes two equals another figure.add_axes, and then point two, point five, point four, point three. And if we're going to do this, they need data on them, so let's go ahead and plot some data on our axes. So axis one.plot, and we'll make this simply x, comma y, comma make it red. And then let's go axes2.plot, and let's reverse them: y, comma x, comma green. There we go, doing what I told you not to do. You shouldn't be swapping axes around and plotting your data in five different directions because it's confusing. Let's go ahead and run this and see what this looks like, and then let's talk a little bit about this. We talked about the 0.2, 0.5, 0.4, 0.3, and let me just grab the annotation for that. That's left, right, width, and height. So we have in here that this is going to be left, right. So here's our left is point one in point five, and we, you know what, let's just play with this a little bit. What happens when I change this to point one? Moves it way over to the left, so there's our point one. So we can make this point four. Run that. There we go. So you can see how you can move it around the branches on here. 0.2, 0.5 is the left, so that's our right. So see what happens when we do point oh. Let's make this point one, and that actually is they had it down at left, right. I thought this was wrong. It's actually how far from the bottom. Let me switch that on here: bottom. There we go. So we had here on this, we can go ahead and put that back to point five and run that, and this is point three. Let's make this point three also, and that is the width, and then of course there's the height. We can make that really tiny. Actually, let's do 0.2, and let's run that, and you can see it changes the height on there. We make it even smaller: 0.2 by 0.2, and as you can see, you can get stuck playing with this to make it look just right. It can sometimes take a little bit. Certainly, once you have the settings, if you're doing a presentation, you try to keep it uniform unless it doesn't make sense for the graph you're working on. Try to keep the same colors, the same position, and the same look and feel.

And I mentioned earlier we can adjust the canvas size than that, but whenever I'm doing professional graphics and printing them out on on something, 300 dots per inch is kind of the minimal on there. You can go a lot higher too, but keep in mind the higher you get, the more memory it takes, the more lag time, and the more resources you use. So usually 300 is a good solid number to use your dots per inch, and you can see it drills a nice it draws a nice large canvas here, which is 16 by nine, and then the dpi is 300 on here is a little higher quality. And just out of curiosity, I wonder how long it takes to draw something double that size, 600, and you can see here where at 600 dpi it's going to take a while. There it goes, just because it's utilizing a lot more graphics on there. And let me just go back to the 300. Now we'll actually do let's do a 100. You're not going to see a difference on this because it is web-based. Graphics are pretty low, and up here you saw I did this with the plot figure. This works the same if I do figure, axes, subplot, figure size, and then we'll go ahead and do axes.plot x, comma y, comma we'll stick to red. Let's go ahead and run this, and you should get almost the same thing here. Here's our axis on the subplot on here with the fixed size and the dpi. Let me take this all out. Let me just remove all that real quick. Run it again. There we go. Now we're back to our original figure.

And let's look at some of the other things you can do with this. One things we do is we can set a title for the axis. So axis.set_title. You'll see right here since I put this on the axis, it's the main title for the whole graph. And if you're going to have a title, you should also label. So we can label our x label, and we can set our y label. In this case, we're just going to call it x and y, keep it nice and uniform. And if we run this, you'll see that we've added a nice x label and y label. Whoops, where'd they go? And it turns out in this environment that you have to put it before the title. So let me go ahead and put it before the title, and there's our x, y, and then we run that, and of course we can also do upper size a little bit. You can see what's going on a little better. So here we have x label x. If you come down here, you'll see our x label and our y label. We can, of course, change this to x label. You can change this to y, maybe whatever you want on here, of course, and our title graph. There we go. Run. So here we have our title graph, our y label, and our x label all set up on our nice little plot.

And then before we move on to the next section, let's do one more thing on here. We have a thing called the legend, and we're gonna do we're gonna set our ax.legend label one, label two. Up here, that's a format for it, but let's go down here and actually use it. I'm gonna do two different plots. We're gonna have axes.plot x by x times x squared and x cubed. And if I run this, you'll see it puts two nice graphs on the setup on there, but it's nice to have a legend telling you what's going on. So for the legend, we actually do axes, since we have the two plots. Legend, and on here we've created an array, and we have y equals x squared, y equals x cubed. You can actually put this as whatever you want; those are just strings. And then location two, and let's go ahead and run this and see what that looks like. And you can see it puts a nice legend on the upper left-hand corner. Location two. We can do location three and run it, and it drops it down to the bottom. Location one, I can't remember where that's at. There we go, upper right. So each one of these is a number that refers to the different locations on the screen. Zero, kinda have to play with them or look them up to remember where they're at, but they do work. It just kind of moves around depending on where you want your legend out on there. So on this section, we cover the title of the graph, the y labels and legends. This is we're getting into some starting to look really fancy here, so we now have something we can actually put out. You'll see the title. The graph looks a little fuzzy, so I might in a web setup put the dpi up a couple notches, maybe put it at 200. 100 might work fine, just so you know, something to notice on here when you're playing with these different things. We had our subplots, dpi equals oh, let's do 200 to see what that looks like. So you can see now it's a lot clearer; it's also larger, so it's a nice little feature you can throw in there with your dpi, dots per inch.

So the next section is let's look at some graph features. We're going to look at line color, transparency, size, and a few more things on here. And oops, I forgot the main title. So we have our figure and our axis equals our plot and subplots, and I'm going to do a dpi equals 150, so the graph comes out nice and large and easy for you to see. Let's go ahead and do three plots on here. We'll do x by x plus one, so it's just going to be a straight line. Plot x plus x plus two, and axes.plot x, x plus three. This looks like we're doing nearest neighbor setup or showing how it uh located data, putting your lines on there between the nearest neighbors. There we go. So it draws a nice little graph with three lines on it. One of the things we can do is we can control the alpha on this. Oops, and you can actually see the um when they did these lines, it automatically pulls in different colors for your setup, so there's some automatic automatic things going on in there. And a lot of times we do that comma r where we're going to do color equals red. Another notation on here. Let's go ahead and run this. Now we have a bright red line down there. And with the matplot library, you're not limited to red. You can also use the one of many different color references as you see here with the pound sign one one five five dd, which just is just blue. And we can do the same thing with another color on here, which turns out to be green. I can just as easily do this green, blue. Oops, there we go, blue, and run that, and you'll see here we have red, blue, and green. And what I want to do is I want to make this. We're going to say what's called the alpha on this, and we're going to set this equal to 0.5, so this is halfway see-through. When I run this, and it's almost going to look pink because you can see through it. And let's change this just a little bit, just to make this kind of fun. Let's square it. There we go. Run it. So now we have this nice square that comes up, and you can see when it crosses it because I plotted these two lines after it, and they have no alpha. The red is behind those lines, or in this case pink because we did the alpha halfway through. So let's go ahead and do this alpha equals 0.5, and oh, you know what, instead of squaring it, let's take it to the 0.5 power. That'll be kind of interesting to see what that does. We'll just go to keep it squared. There we go, and run that, and let's go back and look at this where it crosses over. And the first thing you see right here is on the blue, it's kind of light blue now. You can see how the two colors add together; you get almost a purple on there. So I can clearly see where the red is, the blue line, and then the green just blanks it over because I didn't do any opaqueness, no alpha on there. So this is great if you have lots of data that crosses over, and you need to be able to track those lines better. And we'll go ahead and do this 0.5, and we'll run that. Oops, I did equals 0.5. Let me go ahead and run that, and so you can see right here now you can easily see the red line how it crosses the green and the blue down here. And if we want to, we can do this as the default is one; it's solid, so we can change this all to point eight. Let me just do that. Oops, 0.8. There we go. Run. Oops, I must have hit a wrong button there. Let me try that again. I actually get rid of a bracket, and let's go ahead and run that, and we come down here and look at this. You can still see where it passes behind them, but the green dominates, and the blue dominates because we're now at 80 percent instead of 50 percent. When you can do less, that's kind of fun, although at some point the lanes kind of fade, so 0.5 is usually the best setting on there. We have a nice pastel here at 0.3, and you can easily see where they cross over. And just like you can play with the colors, we can play with line width. And you know, let's do let's try dpi 100 and see what that looks like on my screen equals 100, and we'll go and just take our ax.plot. Let's do four of these lines just you can see how they look next to each other real quick here. There we go. And if I run this, they should all appear the same. It automatically does different colors on there. So let's do color equals blue. Forgot my quotation marks. There we go. And we'll go ahead and just make these all blue just for purposes of being nice and uniform. And then what I want to do is I want to do the line width, width equals 0.25, and let's just copy and paste that down here. Let's do equals one, about 1.5, and let's do one. Let's make this equal to two. Let's see what that looks like. And we do that, you can see it goes from a very thin line, a point five, a one, our one point five, and two, which is twice the width of the one. And if we're going to do different sizes, we had different colors, we had our alpha scheme. Let's take this whole thing here. Let's paste it down here and do another one, but instead of line width, let's look at styles. And something to note here, you can actually abbreviate this with lw, so line width can also be point. Let's just do everything 0.2, and let's set up a line style. We'll do the first one dashes, and let me just paste that down here, so I'm not doing a lot of extra typing. There we go. Take this out, so we have our dashed. We can do a dash dot. We'll just do the dash dot here and a colon here. There we go. And there's a lot of different options. We'll look at a few more as we go down for different ways of highlighting data. But when you look at this, we have everything as a line width of two, and now we have a straight line, we have a dashed line or a dot dash, and a dot dot dot line. And then another thing we can add on here is we're going to do here's our ax.plot, and we did x, let's do x plus four, so it goes right on the top, and do color black, line width 1.5, so it's a smaller line, and we're going to take the line and we're going to set dashes. So look, I've changed some of the notation here for my line and my ax.plot, so I can set my line comma equal to x.plot, and then I can change the line settings this way. And when I run this, let me run that on here, you'll see the 5, 10, 15, 10 creates a series of dashes that are varied in link link. In this case, they alternate between a short dash and a long dash. We can play with these numbers. Curiosity always has me what happens when you play with the numbers just to see what they look like. Let's do this. Let's paste this down here. I'll do two of these just because they're kind of fun to play with, and let's change this from 10 to 3, and we're going to change this one from 15 to 4, and let's run that, and you can see the differences in the lines. Oops, very a little bit confusing on there because I forgot to change the lines are all on top of each other. So let me change that really quick here, and let's run that, and now you can see here's our original dashed line alternating. When I change these numbers on the second one, the very end value to three, you can see now we have the dashes of five. Let's see, I'm going to guess this is a dashes of five, skip ten, dashes of 15, skip 3, and then it goes back to the beginning. Dash is 5 dashes, skip 10, 15 dashes, skip 3. And of course, the last one we just switched up a little bit. It looks a lot more uniform because I'm using two sets of 10. Or if I did something like this and changed it to 30, it really becomes pronounced as far as the distances between them. And instead of 4, let's go oh, let's put 30 here also, 30 by 30. There we go, really pronounced on that one.

And let's look at one more important group for plotting our data, and in this we're gonna here's our plot we started with with the x plus one, x plus two, x plus three, and did it in blue. On this one's three or four different blue lines, and this property we want to add the actual plots, so you can see where the plots are on the graph. And for that, we might have marker equals o, and if we run this, you'll see it puts a dot for each of these, and there's 25 dots because we have 25 x values. So we actually have zero, and each of the different values of x, y are then plotted here with the dots. And we don't want to just limit ourselves to dots. We can also do plus sign; that's another option. Dots is most common. I'll actually like the dots the best. Well, if we do the plus sign, you can see if it's a nice crosshairs or plus sign on there. And we can do a marker; there's a number of different markers you can use, and I think this one was it. S is another one, which is a nice square, and that's actually a good one that's for square, o for period. Okay, that's just kind of weird. So you can see that probably on these markers. Another one is uh number one. So if we run that, you'll see we now have these little hatch marks. And let's take oh, let's just go with the o on this one. By the way, this works with square really nicely. Some stuff we're going to do here on just a second. Let's do marker size equals two, and change that to five, and run that, and you can see here it's a nice little tiny dot versus uh the size dot here. This is interesting because it said two; I thought it would be bigger, but if you do 0.5, it gets even smaller. And let's just do 10 to see what that looks like. Run that. Looks huge. So marker size, a lot of these are dependent on the dpi and the setups. There's things that switch around as far as the way the size shows up. You got to be a little careful when you change one setting; it can change all the other markers. And then let's take our square on here, and we'll do we

Had marker size, so we also have marker face. We'll set that equal to red, of course. We, I mean, change the so it's up one notch. We'll run that. Whoops, must have mistyped something on here, and I did. It's marker face color equals red. And so when I run that, you can now see I have the squares on there with the marker face color.

Of course, we can mix and match these. Come down here and we'll make this instead of, let's make this plus seven, and we'll make this size 15. Marker face color equals, and we'll do what? Green, just because. There we go. Run. Very hard to actually see what's going on there. Still 25 dots; they kind of overlap, as you can see. They print them over each other. Of course, if we really wanted to make it look horrible, we could just make that really huge. Generally, though, you want something a little bit smaller and cuter. We'll just try doing it this way. There we go. That's too small to even see the face, so four. You can start to see the face on there around four and maybe an eight. Eight might be a good number for this. There we go, eight. Again, that all just depends on what you're trying to show and display.

So we've covered a lot of stuff here as far as our lines. We've covered opaque with our alpha setting on there, give us some nice pastels. You can see how they overlap and how they cross over. We covered the line width, different size on there, different formats for the line itself, and these are all, you can combine all these. So you can have our line width equals two, line style equals dash. You can bring this down here also to the markers, and then we added markers in, just entered a circle, a plus sign, the square, a little tick, which uses a one. Then we had a marker size and a marker color face, and we combine those. You see we get a nice different series of representations. We also briefly mentioned color where you didn't have to use, like in here we used color black someplace up here and have to find it. We use the actual number for the color as opposed to I changed it to red and blue. So you can get very precise on the color if you have a very specific color set that you need to match your website or whatever you're working on. All those are tools in the map plot library.

So we have one more piece to formatting the graph so we want to show you, and then we have two big sections we're going to go over: the different graphs that they have along with a challenge problem. So let's go to the last section we're going to look at: limits. We're going to limit our data. So this first primer is going to paste in there. We're going to create our subplots, one, two; so one row, two columns. We're gonna do a figure size of ten comma five. This should all look familiar now since we've done a number of them, and we're gonna go ahead and plot, and this is an interesting notation you should notice here: our axis zero, so one. We've used instead of, you can just iterate through them, but they're just an array. So it's an array of zero is still the axes of the first axes out of two, and we're going to plot x, x squared, x, x cubed, line width 2. So we're going to go ahead and just plot two graphs right on top of each other without doing multiple plots on here, and we'll set the grid equal to true, one here. Let's go ahead and run that, and you can see here our two plots with the x value going across. And I'm going to do something similar, and by the way, as you can just, if you look at it, you can see the grid on there; that's all that is, easier to spot the data going across. We're going to take the same data for axes one, so we have our plot of x, x squared, x, and x cubed, line width two, and this time we're going to take our axes one and do y limit, it's actually set_ylimit. This is the y axis, so it's going to be an array of two, two values, and we'll do 0 comma 60. I'm just making these numbers up; the guys in the back actually made them up; I'm just using their numbers. We're going to set the x limit, and we'll set the x limit as, don't forget our brackets there, two comma five. So it's the same data going in, and but we're setting a limit on it. Let's go ahead and run that and let's see what it comes out of, and here we have the y limit 0 to 60, so we're looking at just the lower part of this curve here up to here, and we have the x limit 2 to 5. So that starts right here at 2, and you can see very different graphs. This is kind of nice because you could actually put one of these on top of the other if you wanted to draw focus to one part of a graph. Remember how we did that earlier, one inside the other, but just a quick note: you can easily limit your graph and re-kind of reshape the way it looks quite easily, and we can also add that grid down there if you want a grid. We'll run that and add the grid in there. Oops, I just have to do the grid beforehand. Switch that. There we go. Sometimes the order on this is really important, so you may double check your order when you're printing these things out. It also helps if I change it to one. So in this case, might not be the order. I wonder if I'll go back here as one. There we go. So it doesn't matter the order and grid, but you can set the grid for easy viewing here. Nice setup on there, but you can see how we can limit the data.

So let's start looking at some other 2D graphs and make this cell a markdown, so we run it as a nice pretty title to it, and let's go ahead and create some data with an np array. We'll do zero to five on here. There we go. And let's look at uh four common graphs. We'll put them side by side, so we'll do a figure, our axes equals plot subplots one four columns, and then figure size, hopefully it'll fit nicely on here. It seems to do a pretty good on here, and I'll go and just run that since we're in there. Run. You'll see I have my four blank plots on here, and we'll start with axes of zero. Let's set title, and we want this to be a scatter plot. A scatter plot just means it has a bunch of dots on it. So here's our axes of zero dot scatter, easy to remember: scatter a bunch of plots on there. We'll do our n, or we can do x or n. There we go. And let's go ahead and do axes set title scatter. I've already did that; we're just gonna do scatter. That's how you do it on there. Notice how you create a scatter plot with simply with the scatter control, and we'll do, let's do the variable x, x plus, let's throw some randomness in here. Usually scatter plots are have a lot of random numbers connected to them; that's why they do them on there. And so the bigger the x gets, the bigger the randomness, so 0.25 times the randomness, and what we should end up doing here is with the scatter plot, and you can see as you go up it just kind of has some random numbers and moves up and down the line, but plus just the points. So if you remember from back up here where we did marker, this is plotting basically just the marker, so it's a scatter plot. Probably less used is a step plot. So for x is one, we'll go ahead and do a step plot, so you can see what that looks like, and this time we'll use our n value instead of x. We generated that n value up here, and so for this we have n, n times two, r n squared, n times two, n squared, line width equals two. And if we run that, it creates a nice step up. Let's see. So we've got a scatter plot, we've got a step plot. Let's do a bar plot, and we'll use the same formula, n, n squared, alignment centered, because you can have them left or right, with 0.5, and alpha, if you remember correctly, that's how opaque it is. Let's see what that looks like on there. So we have some nice, you can see here a nice bar plot. It should look very similar to the step plot, but colored in, and we can change the width. Let's see what happens we do 0.9. Run. And if we take width out completely, run that, you can see it starts coming together on there, and we can change the alpha. We can take the alpha out too and run that, so you know you have the solid colors. And if we take out the center and run that, everything you really can't see the shift on here because that's actually the default on this, but these are common settings for the bar graph. Let me just put them back in there. There we go, alignment center and alpha. Now I can't say I've used the step craft very much. There's certain other certain, I guess domains of expertise require a step graph, but the scatter plot and the bar graph, very common, especially the bar graph. And we'll look at histograms here in just a minute. So I use histograms a lot, especially in data science, but this is nice if you have very concrete objects, somebody, how many people are wearing yellow hats, that kind of thing. But if we're going to do that, let's go ahead and do the last one, which I see a lot more in the sciences, certainly using the data science, but more like for mapping. I saw publication on solar flares, and they're discussing the energy, and so filling in the graph gives it a very different look. So we're going to do the fill between, and it's just like you think it'd be; it's filled between, but with a underscore between them, and we'll do x and x squared and x and x cubed, and we'll do color green and alpha again. In case you had other data you want to plot on there, you can see it forms a nice squared coming up here, and also if you look at the bottom one is your squared value, the upper line is your cubed value, and then it fills in everything in between. If you remember from calculus, this would be if you had like a car, a motor, an efficiency; they would talk about the efficiency going up and the loss, and you're looking for the space or the area between the two lines, so it gives you a nice visual of that.

Now let's look at a few more basic two dimensionals. So we have our figure, figure size on here. We're going to do a radar chart. To be honest, I've never used a radar chart in business or in data science. I get to find a reason to use one now. So the first line for doing a radar chart, we have to add axes into the figure, and with this, this actually creates our, oh let's, let's run it so you can see what it creates. It creates a nice, looks like you're on a submarine and you're tracking the hunt for red october or something like that, and it needs all of these. The polar is the fact that we're doing polar coordinates. 0, 0.6, 0.6 has to do with the size. If you take out any of these things and run them, you get just a box. If you take out the other half, you pretty much get nothing in there. And if you change these numbers and change them a little bit, you can see it gets bigger. They had point six on here. I'll go ahead and leave it as one because that's just kind of fun. That's all about the size on here, the height and the width. And then let's create some data. T equals np line space, and this is zero to two times np times pi. So if you remember that is the distance across, and we're going to generate 100 points. So this is just the thing of data we're putting together. Then we simply do an ax dot plot, and in this case let's do t comma t, which would be a diagonal line on a regular chart, and we'll give it a nice color equals blue and line width equals three. Let's see what that looks like, and we can see here a spiral coming out. Remember this would be just a diagonal line on a regular chart. What happens if we take this and instead of t times 0.5? There we go. And you can see it slightly alters the way it spirals out. We could do t times two, spirals out a little quicker. So it's kind of just a fun, I've, like I said, I've never used a radar chart, it's a column, but you can always think of radar in a submarine, kind of looks like one of those, or in an airplane. And none of this would be complete if we didn't discuss histograms. Oh my gosh, do I use a histogram so much. And we'll use our numpy that we have set as np to generate, oh looks like we have a hundred thousand variables. We're going to set equal to n, and of course we create our figure in our from subplots 1 2, figure size 12 14. So we're going to look at two different variations of the histogram, and we'll set a title, default histogram. Set our title there, and then this is simply hist for histogram, and we'll just go ahead and put in our n in there. And let me run this and see what that looks like, and let's talk about what is going on here. So we generated an array here of data, 1000 random arrays. It looks like they're mostly between minus four and four, and then it adds up each one. It says zero, you have 35,000 that are zero, so that's what's most common on here, and we have 20,000 that are somewhere in this range right here, between the minus two and, well it looks like one and minus two and somewhere between zero and one, there's thirty thousand numbers. So all this is saying is this is how common these variables are, and this gives you this point in so many directions when you're looking at data science to go ahead and run your histogram. So you should always have your histogram, and you can always put limits and all the other different things on your array just like you did on the other graphs on there. And then we're going to do a cumulative detailed histogram, and all it is is a histogram. Let me just do that, and we set cumulative equal to true and bins equal 50. And I really want to highlight the, the cumulative equals true is important, but we can now choose how many bins we have. In the first one, it kind of selected them for us. In this case, let me go ahead and run this, and you'll see it has that prints the data out for us, and here's our, whoops, must have missed, oh there we go. It doesn't help that I put it over the old one. There we go. Okay, so now you have your default histogram, and then we have a cumulative histogram, and we should have 50 steps in there, and let's just find out if that's true, not so much by counting them. I'm not going to count them. If you want to, you can count them. Let's just change it to 10 and see what happens, and we see here we have now 10 counts of that, and we could set that for 5 and run that, and then we have our 5 on there, and we go ahead and take the cumulative equals true out just so you can see what that looks like, and let me run that on here too. That looks just like it did before. I think there's what, one, two, three, four, five, six, seven, eight; they have eight different bins on here is what the default came out of. Put that back in there. Run. And so now it should look almost identical, and it does. And then we can put the cumulative back in, see what that looks like with the cumulative, and run that, and we can see how that shifts everything over and has a slightly different look. Wait, it shifts it all to the right. No, it doesn't actually shift it to the right; it's cumulative, so it's the total of the different occurrences. And so what that means is like if you consider this like for the year of rainfall, we have like day one, you had a little bit of rain, day two we have more rain, and so if you look at the number, this is a hundred thousand, thirty-five thousand, so it's the cumulative detail, the histogram of the currents as it grows, and rainfall is a good one because that would be a cumulative histogram of how much rain occurred throughout the year.

And we're going to look at two more graphs. We've already looked at a bunch of them. We looked at our radar graph, we've looked at scatter, step, bar, fill in basic plots. We've looked at different ways of showing the data; you know, we can increase the size of the line, the look, the color, the alpha setting. So let's look at contour maps. Let's put that in there. There we go. Draw a contour map. And before we draw a contour map, we need to go ahead and create data for it, and if you have contours, your data is all going to have three different values. So let's go ahead and create the data here. We have our, you'd import your matplot library, your numpy. So we have our numbers array, and we'll import matplot.cm, and that's your color maps. So you have all these different color maps you can look at; there's like hundreds of color maps. So if you don't want to do your own color, you can do your own color map; they're pretty diverse. And of course our plt, we're going to our pi plot, and to generate our different data, we're going to create a delta 0.025, and we'll start with x. When we're going to create an array between -3 and 3 and delta increments of 0.025, and we'll have our y, we'll do something similar, and then we'll create our x y into a mesh grid. Again, these are all numpy commands, so if you're not familiar with these, you'll want to go back and review our numpy tutorial, and we'll do an exponential on here, minus x squared minus y squared for z1, we'll do a z2, so we have two different areas, and z equals z of 1 minus z2 times 2. So we've created a number of values here, and let me go ahead and run this, and let's plug that in so you can see where those values are going. So once we've set these, we're going to create our figure and our x from our plt subplots. We're going to create the variable cs, and this is going to be our contour. So right here, cs is our contour surface, and we're feeding it x, y, and z. If you remember x, y, we created as our x and y components using our mesh grid, and you know what, let's do this just because it's kind of good to see this. Let's go ahead and print x, and let's print y, and I always like to do this when I'm working with something that's either is really complicated, in this case is what we're looking at, or you don't understand yet. So we've created a mesh grid, we have x, y, and when we're done with this, we end up with, here's our x and this set of values, and our y. So those are x and y coordinates, and then we've also created z based on our x and y. So we have x, capital x, capital y, and capital z is our three components, x and y being the coordinates, well z is going to be our actual height since we're doing a contour map. So we created our contour map from our x, y, and z coordinates. We want to go ahead and put in a c label, maybe we want to go ahead and do a title on here. We'll put that in our set title, and this is a contour. There we go, contour map. And let's go ahead and run this and see what that looks like, and you'll see we generated a nice little contour map. There's different settings you can play with on this, but you can picture this being you're on a mountain climb, and here we have a line that's represents zero, maybe that's sea level, and then moving on up you have your contours of 0.5 and then minus one and different setups, little hills, I guess if it's minus, that's like a pit. So I guess you're going down into a pit at minus five and minus one, when on the other side you can see you're going up in levels. So here's a mountaintop, and here's like a basin of some kind, and in data

Science. This could represent a lot of things. This could also be representing two different values, and maybe profits and loss. I don't know if I'd ever really do that as a contour map, but I'm sure you can be creative and find something fun to do with a contour map.

And then we're going to look at one last map, which is the 3D map. And those are can be really important as a final product because it can show so much additional information that you can't fit on two-dimensional graphs. There we go, draw a 3D image. And so we're going to import from our mpl toolkits the implant 3D and the axis 3D. We're going to import axis 3D. This is what's going to let us work with the 3D image, and this should look familiar. We're going to create another figure just like we did before, figure size 14 by 6. That's a good fit on the screen. We'll go ahead and run that, so we have our figure.

And let's go ahead and take our x, and we're going to set that equal to fig.add_subplot. That should also be familiar from earlier, and we're going to work with this. Sets the settings for the projection. We're going to use one, two, one projection 3D, and we'll see what that looks like in just a minute. And we just created some three-dimensional data here before where we had x, y, and z, capital X, Y, and Z. So we're going to reuse that data. We're just going to use that since it's also this is also a three-dimensional image, so let's use that for a three-dimensional graph. And we simply do ax.plot_surface and our capital X, capital Y, capital Z. So there's our data coming in, and we're going to add some settings in here. We're going to do rstride 4, cstride 4, and linewidth 0. I'll show you what that is here in just a minute. Let's go ahead and run that so we can see our graph. And of course, it helps if I don't add an extra comma in there. And you can see it generates this really beautiful three-dimensional graph.

So let's take a little bit time to explore some of these numbers we have going in here. We have the rstride 4, the cstride 4, and the projection 3D. Projection 3D is the important one because that's telling us that this is a 3D graph here. So what are these first numbers, one, two, one? Let's just change one of these. I'm going to change this to five, and it's going to give me an error. Let's change it to one, and that didn't work. Let's change this middle one to three instead, and you're going to see how it starts reshaping the size and how it fits on the screen. And we'll change the first one to two. We'll run that one, and again it's changed the dimensions and the size and how it fits on here. Play with these numbers to get a nice look and feel for it. Part of it is the tilt and the angle. I'll do seven on this one. There we go. You can see it really shifted it there, but again that changes the size. Now fits on the canvas, but we'll leave it at the one, two, and just so you get a good look at what we're talking about here. This is column width and index from before. If we do one, one, one, you can see that it now spreads it out all the way across, uses the whole setup on there. So this has to do with the size and how big you want it to be.

Now there's one term that we didn't cover in this yet, but we've used it throughout the whole setup, and I'm just going to type that down here even though we're not going to go into detail, and that's the term heat map. You might see that it's kind of starting to lose ground as far as a common reference, but there sure are a lot of people who still talk about heat maps. What is a heat map? Well, it is simply a color map. That's all it is. So if you ever see the term heat map, that refers to the fact this is in different colors representing different heights. That one is in the heat map, but you can see up here we switched into let me go back up here. Here we go. This one has different colors for the different values. A lot of times you'll use like instead of x and y, you might do a heat map where you have a fourth value, and the fourth value represents the color. And so you'll see this 3D image and the nice colors represented by heat map. That's all it is. So if you see the term heat map, that only means we're plotting some of the data in color to make it stand out or to give it a fourth dimension in this case.

So we've covered a lot of things on map plot, and that brings us cover all the basics. So that brings us to practice example, and this is going to be the challenge for you. And let me go ahead and change our cell cell type markdown and run that so it looks pretty. Practice example: Write a Python program to create a pie chart of the popularity of programming languages. Okay, excellent. And if you're going to have a challenge, we need some data. And I'll just throw in our import our map library at the beginning. You should do that automatically. And so for our data to plot, we're going to have our languages. We're going to have Python, we're going to have Java, PHP, Javascript, C#, C++. So those are six categories. And then we have our popularity. Oops, misspelling there. Popularity. We'll give the first one 22.2 percent, Java 17.6, and I don't know if these are real numbers. They pulled my guess is that they might have just been made up because I don't know Python's really that much more popular than the other ones. Maybe specific to data science because Python is very popular in data science right now because it has so many options. The only other program that's highly used and exclusively for data science is R. So Python's big, and Python also does a lot more. It's a full programming language where R is primarily for data science. They didn't put R in here. So we have Python, we have Java, we have our PHP, and you can see the different values they've given it or different percentages. And I did add these up. Does not add up to 100. It adds up to 71 percent or something like that. And then we're going to give colors, and we've chosen these guys in the back brought in these colors. I'm not sure what these colors are. We'll find out in a minute, so I'll be exciting. But you can see they're using the actual color values. You can pull off of a color wheel or something like that. You could have just as easily done blue, red, green if you're too lazy to pick the exact colors. And then let's go ahead and solve this and see we got here. We're going to do something a little fancy just because we can. The first thing we're going to do is we're going to use a variable called explode, and you'll notice that there's six variables in here, so that matches our six different categories. And the first one we've done is point one and then zero, zero, zero, zero, zero, point one. When we put this in here under the explode in the plot, it will actually push that square out. So it's a really cool feature to highlight certain information on a pie chart. And this is simply plt.pie, and we're plotting popularity there. We go. And before we add in all the really cool settings for this, let's go ahead and run it, and you'll see we generate a nice flat pie. Not too exciting there. And then we'll go ahead and put in all the extras. I talked about explode. We can explode one of the values out. So here's our explode equals explode, labels as languages because we want to know what the different colors mean. Here's our colors equals colors, our autopct, and this is standard print format, so that's a Python setup on there. And that's just going to put the value on the pie slice. And then we're going to add shadow because it just looks cooler with a shadow. Gives a little 3D look. And we'll do a start angle of 140. Let's go ahead and run this and take a look and see what comes out of that. And look how that changes the whole setup. So here's our labels, there's our value we put on there, there's our slices pushed out, there's our shadow with 3D effect, and then we started at 140. We could also rotate this. Let's just do this angle 90, and if we run it, you'll see the blue pie slices moved up a little bit. We could actually do actually let's just take the whole starting triangle out and run it. It'll default to zero. This is what it looks like if it defaulted to zero. So depending on where you want the highlighted slice to appear, usually you want that appear on the left because people read left to right, and so it draws a focus onto in this case Python and how great Python is. I'm a little biased. We're teaching a Python tutorial, so it should be understandable that we're looking at Python.

And one last reference before we close. You can go over to the matplot library.pyplot setup, and if you go underneath there, the different functions on there, you can look this up on their website. You'll see a full list, and this is why it's so important to go through a tutorial like this because this list is just so massive. Trying to figure out like here's our bar plot, there's a barh. You can add barbs. There's a box plot. We didn't cover c labels, a totally different kind of for your contour plot you can set up in there. If you go down here, we have our figures we used on there. We showed you the basics how to do the figure. You'll see some closer references on those. There's a histogram down here, hist. There's also the hist2d makes a 2D histogram plot, hlines. All of this, these are all the different commands that are underneath of here, and you can see it's pretty extensive. We've covered all the basic ones so that you know have a solid ground to look at these different options. So when you come to these functions, some of them are going to look a little off or not off, will look unfamiliar, but you'll still have the availability to probably understand most of this and have a basic understanding of your matplot library.

What does a data scientist do? These are some of the activities: data acquisition, data preparation, data mining, data modeling, and then model maintenance. We will talk about each of these in great detail, but at a very high level, the first step obviously is to get the raw data, which is known as data acquisition. It can be all kinds of format and could be multiple sources, but obviously that raw data cannot be used as it is for performing data mining activities or data modeling activities. So the data has to be claimed and prepared for using in the data models or in the data mining activity. So that is the data preparation. Then we actually do the data mining, which can also include some exploratory activities. And then if we have to do stuff like machine learning, then you need to build a machine learning model and test the model, get insights out of it, and then if the model is fine, you deploy it, and then you need to maintain the model because over a period of time it is possible that you need to tweak the model because of change in the process or changing the data and so on. So that all comes under the model of maintenance.

So let's take a deeper look at each of these activities. Let's start with data acquisition. So the stage of data acquisition, basically the data scientist will collect raw data from all possible sources. So this could be typically an RDBMS, which is a relational database, or it can also be a non-RDBMS, or it could be flat files or unstructured data and so on. So we need to bring all that data from different sources. If required, we need to do some kind of homogeneous formatting so that it all fits into a looks at least format from a format perspective. It looks homogeneous so that may be requiring some kind of transformation. Very often this is loaded into what is known as data warehouse. So this can also be sometimes referred to as ETL or extract, transform, and load. So a data warehouse is like a common place where data from different sources is brought together so that people can perform data science activities like reporting or data mining or statistical analysis and so on. So data from various sources is put in a centralized place, which is known as a data warehouse. So that is also known as ETL. And in order to do this, there can be a data scientist can take help of some ETL tools. There are some existing tools that a data scientist can take help of, like for example, DataStage or Talent or Informatica. These are pretty good tools for performing these ETL activities and getting the data.

The next stage, now that you have the raw data into a data warehouse, you still probably are not in a position to straight away use this data for performing the data mining activities. So that is where data preparation comes into play, and there are multiple reasons for that. One of them could be the data is dirty. There are some missing values and so on and so forth. So a lot of time is actually spent in this particular stage. So a data scientist spends a lot of time, almost 60 to 70 percent of the time in this part of the project or the process, which is data preparation. So there are again within this there can be multiple sub activities starting from let's say data cleaning. You will probably have missing values. The data there is some columns, the values are missing or the values are incorrect. There are null values and so on and so forth. So that is basically the data cleaning part of it. Then you need to perform certain transformations, like for example, normalizing the data and so on, right? So you could probably have to modify a categorical values into numerical values and so on and so forth. So these are transformational activities. Then we may have to handle outliers. So the data could be such that there are a few values which are way beyond the normal behavior of the data for whatever reason, either people have keyed in wrong values or for some reason some of the values are completely out of range. So those are known as outliers. So there are certain ways of handling these outliers and detecting and handling these outliers. So this is a part of what is known as exploratory analysis. So you quickly explore the data to find out other so, and you can use visual tools like plots and identify what are the outliers and see how we can get rid of the outliers and so on. Then the next part could be data integrity. Data integrity is to validate, for example, if there are some primary keys that all the primary keys are populated, there are some foreign keys, and at least most of the foreign keys should be populated, and otherwise when we are trying to query the data, you may get wrong values and so on. So that is the data integrity part of it. And then we have what is known as data reduction. Sometimes we may have duplicate values. We may have columns that may be duplicated because they are coming from different sources. The same values are there and so on. So a lot of this can be done using what is known as data reduction, and thereby you can reduce the size of the data drastically because very often this could be written in data which can be removed and so on.

So let's take a look at what are the various techniques that are used for data cleaning. So we need to ensure that the data is valid and it is consistent and uniform and accurate. So these are the various parameters that we need to ensure as a part of the data cleaning process. Now what are the techniques that that are used for data cleaning or so we will see what each of these are in this particular case. And so what is the data set that we have? We have data about a bank and its customer details. So let's take an example and see how we go about cleaning the data. And in this particular example, we are assuming we are using Python. So let's assume we loaded this data which is the raw file.csv. This is how the customer data looks like, and we will see, for example, we take a closer look at the geography column. We will see that there are quite a few blank spaces. So how do we go about when we have some blank spaces? Or if it is a string value, then we put an empty string here, or we just use a space or empty string. If they are numerical values, then we need to come up with a strategy. For example, we put the mean value. So wherever it is missing, we find the mean for that particular column. So in this case, let's assume we have credit score, and we see that quite a few of these values are missing. So what do we do here? We find the mean for this column for all the existing values, and we found that the mean is equal to 638.6. So we kind of write a piece of code to replace wherever there are blank values. Nan is basically like null, and we just go ahead and say fill it with the mean value. So this is the piece of code we are writing to fill it. So all the blanks or all the null values get replaced with the mean value. Now one of the reasons for doing this is that very often if you have some such situation, many of your statistical functions may not even work. So that's the reason you need to fill up these values or either get rid of these records or fill up these values with something meaningful. So this is one mechanism which is basically using a mean. There are a few others. As we move forward, we can see what are the other ways. For example, we can also say that any missing value in a particular row, if even one column the value is missing, you just drop that particular row or delete all rows where even a single column has missing values. So that is one way of dealing. Now the problem here can be that if a lot of data has let's say one or two columns missing and we dropped many such rows, then overall you may lose out on let's say sixty percent of the data has some value or the other missing. Sixty percent of the rows, then it may not be a good idea to delete all the rows like in that manner because then you're losing pretty much sixty percent of your data, therefore your analysis won't be accurate. But if it is only five or ten percent, then this will work. Another way is only to drop values where or rather drop rows where all the columns are empty, which makes sense because that means that record is of really no use because it has no information in it. So there can be some situations like that. So we can provide a condition saying that drop the records where all the columns are blank or not applicable. We can also specify some kind of a threshold. Let's say you have 10 or 20 columns in a row. You can specify that maybe five columns are blank or null, then you drop that record. So again, we need to take care that such a condition, such a situation, the amount of data that has been removed or excluded is not large. If it is like maybe five percent maximum ten percent, then it's okay. But by doing this, if you're losing out on a large chunk of data, then it may not be a good idea. You need to come up with something better. What else we need to do next is the data preparation part is done. So now we get into the data mining part. So what exactly we do in data mining? Primarily, we come up with ways to take meaningful decisions. So data mining will give us insights into the data, what is existing there, and then we can do additional stuff like maybe machine learning and so on to get perform advanced analytics and so on. So one of the first steps we do is what is known as data discovery and uh which is basically like exploratory analysis. So we can use tools like Tableau for doing some of this. So let's just take a quick look at how we go about that. So Tableau is excellent data mining or actually more of a reporting or a BI tool, and you can download a trial version of Tableau at tableau.com, or there is also Tableau Public which is free, and you can actually use and play around. However, if you want

To use it for enterprise purposes, then it is a commercial software, so you need to purchase a license. You can then run some of the data mining activities. Say your data source—your data is in some Excel sheet—so you can select the source as Microsoft Excel or any other format, and the data will be brought into the Tableau environment. Then it will show you what is known as dimensions and measures. So dimensions are all the descriptive columns; and Tableau is intelligent enough to actually identify these dimensions and measures. So measures are the numerical values. So as you can see here, customer ID, gender, geography—these are all dimensions, non-numerical values—whereas age, balance, credit score, and so on are numeric values, so they come under measures.

So you got your data into Tableau, and then you want to, let's say, build a small model, and you want to let's solve a particular problem. So what is the problem statement? All right, let's say we want to analyze why customers are leaving the bank, which is known as exit, and we want to analyze and see if what are some of the factors for exiting the bank. And we want to, let's assume, consider these three of them, like let's say gender, credit card, and geography, these as a criteria and analyze if these are in any way impacting or have some bearing on the customer exiting or the customer exit behavior. Okay, so let's use Tableau, and very quickly we will be able to find out how these parameters are affecting. All right, so let's see. So this is our customer data. So from our Excel sheet, we have a data set about, let's say, 10,000 rows, and we want to find out what is the criteria. Let's start with gender. Let's say we want to first use gender as a criteria. So Tableau really offers an easy drag-and-drop kind of a mechanism, so that makes it really, really easy to perform this kind of analysis. So what we need to do is "exited" says whether the customer has exited or not, so it has a value of zero and one, and then, of course, you have gender and so on. So we will take these two and simply drag and drop. Okay, so "exited," and then we will put gender. And if we drag and drop into the analysis side of Tableau, all right, so here what we are doing is we are showing male, female as two different columns here, and 0 for people who did not exit and 1 for people who exited, and that is color-coded. So the blue color means people who did not exit, and this yellow color means people who did exit. All right. So now if we pull the data here, create like bar graphs, this is how it would look. So what is yellow? Let's go back. So yellow is uh who exited, and for the male, only 16.45 percent have exited. And we can also draw a reference line that will help us or even provide aliases. So these are a lot of fancy stuff that is provided by Tableau. You can create aliases and so that it looks good rather than basic labels, and you can also add a reference line. So you add a reference line something like this. From here we can make out that, on an average, female customers exit more than the male customers. Right, so that is what we are seeing here on an average. So we have uh analyzed based on gender. We do see that there is some difference in the male and female behavior. Now let's take the next criteria, which is the credit card. So let's see if having a credit card has any impact on the customer exit behavior. So just like before, we drag and drop the credit card. Has credit card a column? If we drag and drop here, and then we will see that there is pretty much no difference between people having a credit card and not having a credit card. Twenty point eight one percent of people who have no credit card have exited, and similarly, twenty point one eight percent of people who have a credit card have also exited. So the credit card is not having much of an impact; that's what this piece of analysis shows. Last, we will basically go and check how the geography is impacting. So once again, we can drag and drop the geography column onto this side, and if we see here, there are geographies like that; I think there are about three geographies like France, Germany, and Spain, and we see that there is some kind of uh with the geography as well. Okay, so what we derive from this is that the credit card is really—we can ignore the credit card variable or feature from our analysis because that doesn't have any impact—but gender and geography we can keep and do further analysis. Okay. All right. So what are some of the advantages of data mining? A bit more detailed analysis can help us in predicting the future trends, and it also helps in identifying customer behavior patterns. Okay, so you can take informed decisions because the data is telling you or providing you with some insights, and then you take a decision based on that. If there is any fraudulent activity, data mining will help in quickly identifying such a fraud as well, and of course, it will also help us in identifying the right algorithm for performing more advanced data mining activities like machine learning and so on.

All right, so the next activity—now that we have the data, we have prepared the data and performed some data mining activity—the next step is model building. Let's take a look at model building. So what is model building? If we want to perform a more detailed data mining activity, like maybe perform some machine learning, then you need to build a model. And how do you build a model? First thing is you need to select which algorithm you want to use to solve the problem on hand and also what kind of data that is available and so on and so forth. So you need to make a choice of the algorithm, and based on that you go ahead and create a model, train the model, and so on. We look at data science components: there's the mathematics, statistics, domain expertise, data engineering, data visualization, and machine learning. When we sum this up in data science, really uh this point in all of this you want to focus on first is domain expertise. Now, when you're taking your classes, obviously you want to go through the mathematics, the statistics, the data engineering, the data visualization, the machine learning, and learning all those tools, but it really in the job field the domain expertise comes in because if you're really good with business and you go into business with these tools, then you're able to ask the right questions and deal with it. When we talk about the tools in data science and where they sit, there is data analysis. Data analysis focuses more on statistics in mathematics where the data science side—where the data science side—focuses more on the machine learning, data visualization, and then there's database management, which really focuses on the data engineering part of it and how to manage that data in the data flow. You can see all these components come together, and then they form the data science. And we talk about tools in data science: we have SAS, we have MATLAB, we have SQL, you have your Apache Spark, your R, Tableau, Python. Right now, in the programming languages, Python is probably the most widely used, with R coming in a close second. The reason for that is these are both open source, and anybody can use them. We look at SAS and Tableau; those are two packages that are paid for, which have a lot of automation behind them. So a lot of the larger companies will pay for those packages because they help work out a lot of the the back end stuff before you even need to use them. And of course, we have Apache Spark. You almost have to know Apache Spark in the growing data lakes today; that's your big data. How do you distribute this if you're going to process it over a large amount of data? We look at data science applications: there's internet search—very common; you enter your Google thing, and it uses the data science algorithms to figure out what to return; voice assistance—analyzing the voice coming in and connecting it with whatever patterns they need to activate so you can ask it to turn your lights on in the living room; healthcare—biggest field growing right now; how do we analyze health data from a smart watch and figure out how to know when a disease is being spread across the world; healthcare—how do we identify cancer versus non-cancer so we know who to prioritize and get them in for surgery where other people might not need that immediate care; and even identifying problems somebody might have with their health long before they even happen—what a neat thing to do instead of waiting until you have a heart attack to know months ahead of time that you're having heart conditions; logistics—how do we get from point A to point B? If you ever look at some of the major distributors, they pre-ship things based on what they guess people are going to need. This way you get your stuff right in the mailbox the day after you order it; e-commerce—huge in there because you have to know what kind of marketing we're going to put out, who are we selling to, what's our target audience; robotics—one of the funner ones when you're coming up into the future is how do we automate the different tasks we have from self-driving cars to manufacturing lines to maybe even a self-robotic chef in the kitchen.

So we talk about solving problems with data science; we have to ask possible questions and the desired algorithms, and these really are kind of an overview of some of the terminology you'll want to know in data science. So we talk about that, how much or how many, we're talking about regression. Regression basically means a number. If you've ever seen stock charts, you see [Music] the guess at what the next sales price is or buy price is; that's all on your regression side. Is it A or B? Classification—is that a dog or a cat that my photo took a picture of? How is this organized? Clustering—this is where we take things that look alike and put them together so that we can then make predictions on it. How different is this? Just like you have clustering, you have the opposite: anomaly detection—how do we find things that don't fit in, where the outliers, what are things we might need to look at that aren't in our model, and what to do next? Reinforced learning—this is probably the newest market is how do we set something up whereas the data comes in, it learns from the new data as to what the next action is going to take. Most of these are combined. So we talk about regression, classification, clustering, anomaly detection, reinforced learning. A lot of times when you put your model together, you might have multiple parts of this. We might look at the clustering of data and then feed it into a classification: is this a bad loan to make at the bank, or is this a good loan to give? And from there we might start looking at when we start clustering these things as good and bad, and what kind of setup is—we then might run it through a regression model to say, hey, what is the amount—dollar amount—this person should be allowed to take a loan out. And you can see that all of these start coming in, and then a lot of the models—reinforced learning—we talk about like bank loans and things like that. Once a year they have to rerun those algorithms, and so that's like a human-run reinforced learning. Now we're starting to automate that so that as the data comes in real time, the models start updating themselves; they start figuring out a way to solve new problems as they occur. And of course, in robotics, reinforced learning is one of the biggest growing fields. Logarithms used in data science: linear regression, logistic regression. When you're looking at both of these, uh think numbers; we're talking regression models, and so linear regression figures out the best line through the data; maybe it's looking for a curved line. Logistic regression starts looking at data on a continuum so that it goes to an exponential; you have a spot where it might be one or the other, and then you know what it is as it goes to one side or the other. Decision tree—wonderful tool; if you used to call it a hack, now it's becoming mainstream to look at the back end—what made the decision—so it's really easy to trace back to how you arrived at the decision. Decision trees are kind of nice that way. Nearest neighbors—k-nearest neighbors is your clustering; k-means clustering, hierarchical clustering. So hierarchical starts dividing them and builds a hierarchy where the k-means and the k-nearest neighbor look for things that kind of connect each other—how close are they in data format? DBSCAN—another form of clustering your data. Principle component analysis—so we start breaking up and looking at the different components in our data coming in. These are all algorithms used in data science, and there are more; these are kind of the mainline ones, and each one of these has many settings which you can play with to build a better model to predict your data and where you want it to go.

Let's go ahead and take a hands-on demo and just see what does this look like. Now I'm going to go ahead and use the Anaconda Navigator and the which is your Jupyter Notebook, which will then open up in Google Chrome. This is one of the IDEs or ways of editing Python code. I like this one because it one protects my computer and creates its own kind of container, and two it's got a really nice visual demo for doing Python. Certainly you can use a number of other different tools like PyCharm; there's another real popular one. You can actually access PyCharm code through the Anaconda Navigator now, and there's a lot of other tools on there that each have their benefit. And we come into our Jupyter Notebook, go ahead and create a new Python; I've already done that, and we'll start with a lot logistic regression model just so you can see what this looks like in code. We'll actually do two of these as we go through this and take a look and see what that code looks like in Python. Now the first thing we want to go ahead and do is load up all our parts modules that we need in our Python code. I'll go and run that, and you'll see here I'm going to ignore warnings because some of the stuff we're doing uh we're just looking at it, and as you go from one version of Python to one version of whatever the site kit or in this case the SC package, you'll see some warnings come up; they're not really important right now. And then we're going to go ahead and use the German credit data CSV. And so when I run this, this is all loaded up now. Let's go ahead and just open up this CSV file so you can see what that looks like, and here it is. Here's our German credit data file. We'll go ahead and just open that up with—let's do WordPad—just fine because it's a word document. And when we open that up, you can see right here is just a bunch of comma-separated values. We have across the top; it actually has columns on here, so our first row is a bunch of columns, and then, of course, we look down here—oops, I just move stuff around, but that's okay; I'm not going to save it—and you can see the values: al1, al4, whatever that means, or a14 I guess, and then some sort of numbers on here. And we'll take a closer look when we get into the program and looking at the data. For right now, we've loaded that under credit df, and at this point I want to just kind of take a detour back to the life cycle of data science so we can see how that fits in there. And so we've already done our data discovery; we have a little data prep, but the data is pretty clean; it already comes in preset on there. We'll make do a little data preparation in here. We're going to start looking at exploring the data analysis and building a data model and then interpreting the results. So really we're focusing on the last three steps of this. The data discovery would be asking uh start asking questions of what do we want in there: is this going to be—is this going to be a good loan, a bad loan? What do we want to get out of this data? Where are we going with it? That would be part of the data discovery. A lot of times that might be a meeting you have with the management in the office or whatever. And then data preparation is, of course, cleaning the data up, uh making sure you know where the source came from, all those kind of fun things that go into making sure you have good data coming in. We're going to cover mathematics for machine learning, so today's agenda is going to cover data and its types. Then we're going to dive into linear algebra and its concepts, calculus, statistics for machine learning, probability for machine learning, hands-on demos, and of course, thrown in there in the middle is going to be your matrices and a few other things to go along with all this. Data in its types: data denotes the individual pieces of factual information collected from various sources; it is stored, processed, and later used for analysis. And so we see here just a huge grouping of information, a lot of tech stuff, money, dollar signs, numbers, and then you have your performing analytics to drive insights, and hopefully you have a nice—sure your shareholders gather it at the meeting, and you're able to explain it in something they can understand. So we talk about data types of data; we have in our types of data, we have a qualitative categorical—you think nominal or ordinal—and then you have your quantitative or numerical, which is discrete or continuous. And let's look a little closer at those. Data type vocabulary—always people's favorite is the vocabulary words. Okay, not mine, uh but let's dive into this. What we mean by nominal: nominal, they are used to label various—label our variables without providing any measurable value: country, gender, race, hair color, etc. It's something that you either mark true or false. This is a label; it's on or off; either they have a red hat on or they do not. So a lot of times when you're thinking nominal data labels, uh think of it as a true-false kind of setup. And we look at ordinal: this is categorical data with a set order or a scale to it, and you can think of salary range as a great one, movie ratings, etc. You can see here the salary ranges: if you have 10,000 to 20,000, the number of employees earning that rate is 150; 20,000 to 30,000, 100, and so forth. Some of the terms you'll hear is "bucket"—this is where you have 10 different buckets, and you wanna separate it into something that makes sense into those 10 buckets. And so we start talking about ordinal. A lot of times when you get down to the brass tacks again, we're talking true-false. So if you're a member of the 10-to-20k range, so forth, those would each be either part of that group or you're not, but now we're talking about buckets, and we want to count how many people are in that bucket. Quantitative numerical data falls into two classes: discrete or continuous. And so data with a final set of values which can be categorized: class strength, questions answered correctly, and runs hit in cricket. A lot of times when you see this, you can think integer, and a very restricted integer, i.e., you can only have 100 questions on a test, so you can—it's very discreet; I only have 100 different values that it can attain. So think usually you're talking about integers, but within a very small range; they don't have an open end or anything like that. So discrete is very solid, simple to count, set number. Continuous, on the other hand, continuous data can take any numerical value within a range: water pressure, weight of a person, etc. Usually we start thinking about float values where they can get phenomenally small in their in what they're worth, and there's a whole series of values that falls right between discrete and continuous. You can think of the stock market: you have dollar amounts; it's still discrete, but it starts to get complicated enough when you have like, you know, a jump in the stock market from 525.33 cents to 580.67; there's a lot of point values in there. It'd still be called discrete, but you start looking at it as almost continuous because it does have such a variance in it. Now uh we talked about—no, we did—we went over nominal and ordinal—almost true-false charts—and we looked at quantitative and numerical data, which were starting to get into numbers. Discrete—you can usually—a lot of times discrete will…

Be put into it. Could be put into true/false, but usually it's not. Uh, so we want to address this stuff, and the first thing you want to look at is the very basic, which is your algebra. So we're going to take a look at linear algebra. You can remember back when your Euclidean geometry, we have a line. Well, let's go through this. We have a linear algebra is the domain of mathematics concerning linear equations and their representations in vector spaces and through matrixes. I told you we're going to talk about matrixes. So a linear equation is simply 2x + 4y - 3z = 10. Very linear. 10x + 12.4y = z. And now you can actually solve these two equations by combining them, and that's we're talking about a linear equation in the vectors. We have a + b = c. Now we're starting to look at a direction, and these values usually think of an x, y, z plot, so each one is a direction and the actual distance of like a triangle, a + b = c. And then your matrix can describe all kinds of things. I find matrixes uh confuse a lot of people, not because they're particularly difficult, but because of the magnitude and the different things they're used for. And a matrix is a chart or a, you know, think of a spreadsheet, but you have your rows and your columns, and you'll see here we have a * b = c. Very important to know your counts. So depending on how the math is being done, what you're using it for, making sure you have the same rows and number of columns or a single number, there's all kinds of things that play in that that can make matrixes confusing, but really it has a lot more to do with what domain you're working in. Are you adding in multiple polynomials where you have like a x squared + b y + you know, you start to see that it can be very confusing versus a very straightforward matrix.

Let's just go a little deeper into these because these are such primary, this is what we're here to talk about is these different math uh mathematical computations that come up. So we're looking at linear equations. Let's dig deeper into that one. An equation having a maximum order of 1 is called a linear equation. So it's linear because when you look at this, we have ax + b = c, which is a one variable; we have two variable ax + by = c; ax + by + cz = d; and so forth, but all of these are to the power of one. You don't see x squared; you don't see x cubed. So we're talking about linear equations; that's what we're talking about in their addition. If you have already dived into, say, neural networks, you should recognize this ax + by + cz setup + the intercept, which is basically your your neural network, each node adding up all the different inputs, and we can drill down into that. Most common formula is your y = mx + c. So you have your y = the m, which is your slope, your x value + c, which is your y-intercept. You kind of labeled it wrong here, threw me for a loop, but the the c would be your y-intercept. So when you set x = 0, y = c, and that's that's your y-intercept right there. Uh, and that's they just had reversed value of y when x = 0 = the y-intercept, which is c, and your slow gradient line, which is your m. So it's your y = 2x + 3. And there's lots of easy ways to compute this this way. This is why we always start with the most basic one when we're solving one of these problems. And then, of course, the one of the most important takeaways is the slope gradient of the line. So the slope is very important, that m value. In this case, we went ahead and solved this: if y = 2x + 3, you can see how it has a nice line graph here on the right.

Matrixes: A matrix refers to a rectangular representation of an array of numbers arranged in columns and rows. So we're talking m rows by n columns here. a11 denotes the element of the first row in the first column. Similarly, a12, and it's really pronounced a11 in this particular setup. So it's a row one, column one; a12 is a row one, column two; uh, first row and second column; and so on. And there's a lot of ways to denote this. I've seen these as like a capital letter, a smaller case a for the top row, or I mean, you can see where they can go all kinds of different directions as far as the value. You just take a moment to realize there's a need to be some designation as far as what row it's in and what column it's in. And we have our basic operations. We have addition. So when you think about addition, you have uh two matrixes of two by two, and you just add each individual number in that matrix. And then when you get to the bottom, you have in this case the solution is 12; 10 + 2 = 12; 5 + 3 = 8; and so on. And the same thing with subtraction. Now, again, you're counting matrixes; you want to check your dimensions of the matrix, the shape. You'll see shape come up a lot in programming. So we're talking about dimensions; we're talking about the shape. If the two shapes are equal, this is what happens when you add them together or subtract them. And we have multiplication. When you look at the multiplication, you end up with a very uh a slightly different setup going now. If we look at our last one where uh we're like, why this always gets to me when we get to matrixes, they don't really say why you multiply matrixes. You know, my first thought is one times two, four times three, but if you look at this, we get 1 * 2 + 4 * 3; 1 * 3 + 4 * 5; 6 * 2 + 3 * 3; 6 * 3 + 3 * 5. If you're looking at these matrixes, uh, think of this more as an equation. And so we have, if you remember when we back up here for our multiple line equations, let's just go back up a couple slides where we were looking at a two variable. So this is a two variable equation: ax + by = c, and this is a way to make it very quick to solve these variables, and that's why you have the matrix, and that's why you do the multiplication the way they do. And this is the dot product of 1 * 2 + 4 * 3; 1 * 3 + 4 * 5; uh, 6 * 2 + 3 * 3; 6 * 3 + 3 * 5. And it gives us a nice little 14, 23, 21, and 33 over here, which then can be used and reduced down to a sample formula as far as solving the variables as you have enough inputs.

And then in matrix operations, when you're dealing with a lot of matrixes, now keep in mind multiplying matrixes is different than finding the product of two matrixes. Okay, so we're talking about multiplication; we're talking about solving for equations. When you're finding the product, you are just finding 1 * 2. Keep that in mind because that does come up. I've had that come up a number of times where I am altering data and I get confused as to what I'm doing with it. Uh, transpose: flipping the matrix over is diagonal, comes up all the time where you have you still have 12, but instead of it being 12, 8, it's now 12, 14, 8, 21. You're just flipping the columns and the rows. And then, of course, you can do an inverse, changing the signs of the values across this main diagonal, and you can see here we have the inverse a to the minus one and ends up with uh instead of 12, 8, 14, 12 is now -22, -12.

Vectors: Vector just means we have a value and a direction, and we have down four numbers here on our vector. In mathematics, a one-dimensional matrix is called a vector. So if you have your x plot and you have a single value, that values along the x axis, and it's a single dimension. If you have two dimensions, you can think about putting them on a graph. You might have x, and you might have y, and each value denotes a direction. And then, of course, the actual distance is going to be the hypothesis of that triangle. And you can do that with three dimensionals: x, y, and z. You can do it all the way to nth dimensions. So when they talk about the k-means for categorizing and how close data is together, they will compute that based on the Pythagorean theorem. So you would take the square of each value, add them all together, and find the square root, and that gives you a distance as far as where that point is, where that vector exists, or an actual point value. And then you can compare that point value to another one. It makes a very easy comparison versus comparing 50 or 60 different numbers.

And that brings us up to eigen vectors and eigen values. Eigen vectors: the vectors that don't change their span while transformation. And eigen values: the scalar values that are associated to the vectors. Conceptually, you can think of the vector as your picture. You have a picture; it's uh two dimensions, x and y. And so when you do those two dimensions and those two values or whatever that value is, um, that is that point, but the values change when you skew it. And so if we take and we have a vector a, and that's a set value, uh b is your is your you have a and b, which is your eigen vector; two is the eigen value. So we're altering all the values by two; that means we're uh maybe we're stretching it out one direction, making it tall. Uh, if you're doing picture editing, that's one of the places this comes in, but you can see when you're transforming your different information, how you transform it is then your eigen value. And you can see here: vector after line transit transition, we have 3a; a is the eigen vector; three is the eigen value. So a doesn't change; that's whatever we started with; that's your original picture. And three is skewing it one direction, and maybe b is being skewed another direction, and so you have a nice tilted picture because you altered it by those by the eigen values.

So let's go ahead and pull up a demo on linear algebra. And to do this, I'm going to go through my trusted Anaconda into my Jupyter notebook, and we'll create a new notebook called Linear Algebra. Since we are working in Python, we're going to use our NumPy. I always import that as np, or NumPy array, probably the most popular module for doing matrixes and things in. Given that this is part of a series, I'm not going to go too much into NumPy. We are going to go ahead and create two different variables, a for a NumPy array [10, 15], and b [20, 9]. We'll go ahead and run this, and you can see there's our two arrays: [10, 15], [20, 9]. And I went and added a space there in between so it's easier to read. And since it's the last line, we don't have to put the print statement on it unless you want. We can simply, but we can simply do a + b. So when I run this, we have [10, 15], [20, 9], and we get [30, 24], which is what you expect: 10 + 20, 15 + 9. You could almost look at this addition as being just adding up the columns on here coming down. And if we wanted to do it a different way, we could also do a.T + b.T. Remember that T flips them. And so if we do that, we now get them; we now have [30, 24] going the other way. We could also do something kind of fun; there's a lot of different ways to do this as far as a + b. I can also do a + b.T, and you're going to see that that will come out the same, the [30, 24], whether I transpose a and b or transpose them both at the end. And likewise, we can very easily subtract two vectors. I can go a - b, and we run that, and we get [-10, 6]. Now remember this is the last line in this particular section; that's why not to put the print around it. And just like we did before, we can transpose either the individual, or we can transpose the main setup, and then we get a [-10, 6] going the other way. Now we didn't mention this in our notes, but you can also do a scalar multiplication and just put down the scalar. So you can remember that what we're talking about here is I have this array here u, and if I go a * u, we'll take the value 2, we'll multiply it by every value in here. So 2 * 30 = 60, 2 * 15, and just like we did before, this happens a lot because when you're doing matrixes you do need to flip them; you get [60, 30] coming this way.

So in NumPy, uh, we have what they call dot product, and uh with this, this in a two-dimensional vectors, it is the equivalent of two matrix multiplication. Remember we were talking about matrix multiplication uh where it is the well, let's walk through it. We'll go ahead and start by defining two NumPy arrays. We'll have [10, 20], [25, 6] for our u and our v, and then we're going to go ahead and do if we take the values, and if you remember correctly, an array like this would be 10 * 25 + 20 * 6. We'll go ahead and print that; there we go. And then we'll go ahead and do the np.dot(u, v), and we'll find when we do this, we go and run this, we're going to get 370, 370. So this is a strain multiplication where they use it to solve linear algebra when you have multiple numbers going across. And so this could be very complicated; we could have a whole string of different variables going in here, but for this we get a nice value for our dot multiplication. And we did addition earlier, which is just your basic addition, and of course the matrix. You can get very complicated on these, or in this case, we'll go ahead and do let's create two complex matrixes. This one is a matrix of um, you know, [12, 10, 4], [6, 4, 31]. We'll just print out a so you can see what that looks like. Here's print a. We print a out; you can see that we have a 2 by 3 layer matrix for a, and we can also put together always kind of fun when you're playing with print values. We could do something like this; we could go in here; there we go; uh, we could print a; we have it end with uh = a; run. And this kind of gives it a nice look; here's your matrix; that's all this is; comma n means it just tags it on the end; that's all; all that is doing on there. And then we can simply add in what is a + b, and you should already guess because this is the same as what we did before; there's no difference. We do a simple vector addition; we have 12 + 2 = 14; 10 + 8 = 18; and so on. And just like we did the matrix addition, we can also do a - b and do our matrix subtraction. And we look at this; we have what 12 - 2 = 10; 10 - 8; um, where are we? Oh, there we go. 8 - confusing what I'm looking at; I should have reprinted out the original numbers, but we can see here 12 - 2 is, of course, 10; 10 - 8 = 2; 4 - 46 = -42; and so forth. So same as the subtraction as before; we just call it matrix subtraction; it's identical. Now if you remember up here we had a scalar addition; we're adding just one number to a matrix; you can also do scalar multiplication. And so simply if you have a single value a and you have b, which is your array, we can also do a * b. When we run that, you can see here we have 2 * 4 = 8; 5 * 4 = 20; and so forth. You're just multiplying the 4 across each one of these values.

And this is an interesting one that comes up, a little bit of a brain teaser, is matrix and vector multiplication. And so when we're looking at this, we are just doing regular arrays; it doesn't necessarily have to be a NumPy array. We have a, which has our array of arrays, and b, which is a single array. And so we can from here the .a, b, and this is going to return two values, and the first value is that it's you could say it's like uh we're doing the this array b array first with a and then with a second one. And so it splits it up, so you have a matrix of vector multiplication, and you can mix and match. When you get into really complicated uh backend stuff, this becomes more common because you're now you've got layers upon layers of data. And so you'll end up with a matrix and a set of bolt vector matrices, do you want to multiply. Now keep in mind that if you're doing data science, a lot of times you're not looking at this; this is what's going on behind the scenes. So if you're in the scikit, looking at sklearn, where you're doing linear regression models, this is some of the math that's hidden behind the scenes that's going on. Other times you might find yourself having to do part of this and manipulate the data around so it fits right, and then you go back in and you run it through the scikit. And if we can do up here where we did a matrix and vector multiplication, we can also do matrix-to-matrix multiplication. And if we run this, we have the two matrixes; you can see a very complicated array that of course comes out on there for our dot. And just to reiterate it, we have our transpose matrix, which is your .T. And so if we create a matrix a and we do transpose it, you can see how it flips it from [5, 10, 15], [20, 25, 30] to [5, 15, 25], [10, 20, 30], rows and columns. And certainly with the math, this comes up a lot; it also comes up a lot with xy plotting. When you put into pyplot, you have one format where they're looking at pairs and numbers, and then they want all of x's and all y's, so you know the transpose is an important tool both for your math and for plotting and all kinds of things. Another tool that we didn't discuss uh is your identity matrix, and this one is more definition. The identity matrix we have here one where we just did two, so it comes down as [1, 0, 0], [0, 1, 0], [0, 0, 1]. It creates a diagonal of one, and what that is is when you're doing your identities, you could be comparing all your different features to the different features and how they correlate. And of course, when you have feature one compared to feature one to itself, it is always one, where usually it's between zero one depending on how well correlates. So when we're talking about identity matrix, that's what we're talking about right here is that you create this preset matrix, and then you might adjust these numbers depending on what you're working with and what the domain is. And then another thing we can do to kind of wrap this up, we'll hit you with the most complicated uh piece of this puzzle here is an inverse a matrix. And let's just go ahead and put the um it's a lengthy description; let's go and put the description. This is straight out of the the website for NumPy. Uh, so given a square matrix a, here's our square matrix a, which is [2, 1, 0], [0, 1, 0], [1, 2, 1]. Keep in mind, three by three; it's square; it's gotta be equal; it's going to return the matrix a inverse satisfying .a, a inverse. So here's our matrix multiplication, and then of course it equals the dot; yeah, a inverse of a with an identity shape of a.dot.shape[0]. This is just reshaping the identity; that's a little complicated there. Uh, so we're going to have our here's our array. We'll go ahead and run this, and you can see what we end up with is we end up with an array [0.5, -0.5], and so forth, with our [2, 1, 1] going down [2, 1, 0], [0, 1, 0], [1, 2, 1]. Getting into a little deep on the math understanding when you need this is probably really is is what's really important when you're doing data science versus handwriting this out and looking up the math and handwriting all the pieces out. You do need...

To know about the linear algorithm inverse of A, so if it comes up, you can easily pull it up, or at least remember where to look it up. We took a look at the algebra side of it; let's go ahead and take a look at the calculus side of what's going on here with the machine learning. So calculus, oh my goodness, and differential equations—you got to throw that in there because that's all part of the bag of tricks, especially when you're doing large neural networks, but also comes up in many other areas. The good news is most of it's already done for you in the back end, so when it comes up, you really do need to understand it from the data science, not data analytics. Data analytics means you're digging deep into actually solving these math equations, and a neural network is just a giant differential equation.

Uh, so we talk about calculus; we're going to go ahead and understand it by talking about cars versus time and speed. So it helps to calculate the spontaneous rate of change. Suppose we plot a graph of the speed of a car with respect to time. So as you can see here, going down the highway, probably merged into the highway from an on-ramp, so I had to accelerate, so my speed went way up. Uh, stuck in traffic, merged into the traffic; traffic opens up, and I accelerate again up to the speed limit and maybe peter's off up there. So you can look at this as the speed versus time; I'm getting faster and faster because I'm continually accelerating, and if I hit the brakes, you go the other way. So the rate of change of speed with respect to time is nothing but acceleration—how fast are we accelerating? The acceleration is the area between the start point of x and the endpoint of delta x. So we can calculate a simple: if you had x and delta x, we could put a line there, and that slope of the line is our acceleration. Now that's pretty easy when you're doing linear algebra, but I don't want to know it just for that line in those two points; I want to know it across the whole of what I'm working with. That's where we get into calculus.

So we talk about the distance between x and delta x; it has to be the smallest possible, near to zero, in order to approximate the acceleration. So the idea is that instead of—I mean, if you ever did take a basic calculus class, they would draw bars down here, and you would divide this area up—let's go back up the screen—you divide this area of this time period up into maybe 10 sections, and you'd use that, and you could calculate the acceleration between each one of those 10 sections, kind of thing. And then we just keep making that space smaller and smaller until delta x is almost infinitesimally small, and so we get a function of a equals a limit as h goes to 0 of a function of a plus h minus a function of a over h. And that is you're computing the slope of the line; we're just computing that slope in smaller and smaller and smaller samples, and that's what calculus is. Calculus is the integral. You can see down here we have our nice integral sign; it looks like a giant S, and that's what that means is that we've taken this down to as small as we can for that sampling.

Uh, so we're talking about calculus; we're finding the area under the slope is the main process in the integration. Similar small intervals are made of the smallest possible length of x plus delta x, where delta x approaches almost an infinitesimally small space, and then it helps to find the overall acceleration by summing up all the links together. So we're summing up all the accelerations from the beginning to the end, and so here's our integral: we sum of a of x times d of x equals a plus c; that is our basic calculus here. So when we talk about multivariate calculus, uh, multivariate calculus deals with functions that have multiple variables, and you can see here we start getting into some very complicated equations: change in w over change of time equals change of w over change in z, the differential of z to dx, differential of x to dt; it gets pretty complicated, and it really translates into the multivariate integration using double integrals. And so you have the sum of the sum of f of x of y of d of a equals the sum from c to d and a to b of f of x y d x d y equals the sum of a to b sum of c to d of f x of y d y d x. Understanding the very specifics of everything going on in here and actually doing the math is usually calculus 1, calculus 2, and differential equations, so you're talking about three full-length courses to dig into and solve these math equations. What we want to take from here is we're talking about calculus; we're talking about summing of all these different slopes, and so we're still solving a linear expression; we're still solving y equals mx plus b, but we're doing this for infinitely small x's, and then we want to sum them up. That's what this integral sign means: the sum of a of x d of x equals a plus c. And when you see these very complicated multivariate differentiation using the chain rule, when we come in here and we have the change of w to the change of t equals the change of w dz, uh, and so forth, that's what's going on here; that's what these means. We're basically looking for the area under the curve, which really comes to how is the change changing; speeds going up, how is that changing? And then you end up with a multiple layer; so if I have three layers of neural networks, how is the third layer changing based on the second layer changing, which is based on the first layer changing, and you get the picture here that now we have a very complicated multivariate integration with integrals. The good news is we can solve this mathematically, and that's what we do when you do neural networks and reverse propagation.

So the nice thing is that you don't have to solve this on paper unless you're a data analysis and you're working on the back end of integrating these formulas and building the script to actually build them. So we talk about applications of calculus; it provides us the tools to build an accurate predictive model. So it's really behind the scenes; we want to guess at what the change of the change of the change is. That's a little goofy; I, I know; I just threw that out there; it's kind of a meta term, but if you can guess how things are going to change, then you can guess what the new numbers are. Multivariate calculus explains the change in our target variable in relation to the rate of change in the input variables. So there's our multiple variables going in there; if one variable is changing, how does the variable—and then in gradient descent, calculus is used to find the local and global maxima, and this is really big; we're gonna actually gonna have a whole section here on gradient descent because it is really—I mean, I talked about neural networks and how you can see how the different layers go in there, but gradient descent is one of the most key things for trying to guess the best answer to something.

So let's take a look at the code behind gradient descent, and before we open up the code, let's just do real quick gradient descent. Let's say we have a curve like this, and most common is that this is going to represent your error—oops, error—there we go—there—ah, hard to read there—and I want to make the error as low as possible, and so what I'm looking at it is I want to find this line here, which is the minimum value. So we're looking for the minimum, and it does that by sampling there, and then based on this, it guesses it might be someplace here, and it goes, hey, this is still going down; it goes here and then goes back over here, and then goes a little bit closer, and it's just playing a high low until it gets to that spot, that bottom spot. And so we want to minimize the error, and on the flip note, you could also want to be maximizing something; you want to get the best output of it; that's simply minus the value. So if you're looking for where the peak is, this is the same as a negative for where the valley is; looking for that valley, that's all that is; this is a way of finding it.

So the cool thing is, um, all the heavy lifting is done. I actually ended up putting together one of these a while back is when I didn't know about sidekick, and I was just starting—boys, a long while back—and uh, is playing high low; how do you play high low, not get stuck in the valleys, figure out these curves and things like that? Well, you do that, and the back end is all the calculus and differential equations to calculate this out. The good news is you don't have to do those. So instead, we're going to put together the code, and let's go ahead and see what we can do with that. So, uh, guys in the back put together a nice little piece of code here, which is kind of fun. Uh, some things we're going to note, and this is this is really important stuff because when you start doing your data science and digging into your machine learning models, you're going to find these things are stumbling blocks. The first one is current x: where do we start at? Keep in mind your model that you're working with is very generic, so whatever you use to minimize it, the first question is where do we start? Um, and we started at this because the algorithm starts at x equals three, so we arbitrarily picked five. Learning rate is, uh, how many bars to skip, going one where the other. I'm, in fact, I'm going to separate that a little bit because these two are really important. Um, if we're dealing with something like this where we're talking about—well, here's our, here's the function we're going to use—our gradient of our function: 2 times x plus 5; keep it simple. So that's a function we're going to work with. So if I'm dealing with increments of 0.1, it's going to be a very long time, and if I'm dealing with increments of 0.001, 0.1 is going to skip over my answer, so I won't get a very good answer. And then we look at precision; this tells us when to stop the algorithm. So again, very specific to what you're working on; if you're working with money and you don't convert it into a float value, you might be dealing with 0.01, which is a penny; that might be your precision you're working with. And then, of course, the previous step size, max iterations; we want something to cut out at a certain point; usually that's built into a lot of minimization functions. And then here's our actual formula we're going to be working with, and then we come in; we go while previous step size is greater than percent and it is less than max and max iters; say that 10 times fast. Um, we're just saying if it's, uh, if we're, if we're still greater than our precision level, we still got to keep digging deeper. Um, and then we also don't want to go past a thou, or whatever this is—a million or ten thousand—running that's actually pretty high; we almost never do max iterations more than like a hundred or two hundred; rare occasions you might go up to four or five hundred if it's, depending on the problem you're working with. Uh, so we have our previous equals our current that way we can track time-wise, uh, the current now equals the current minus the rate times the formula of our previous x. So now we've generated our new version; previous step size equals the absolute current previous; so we're looking for the change in x; errors equals iterations plus one; that's so we know to stop if we get too far; and then we're just going to print the local minimum occurs at x on here. And if we go ahead and run this, you can see right here it gets down to this point, and it says, hey, local minimum is minus 3.3222 for this particular series we created. This is created off of our formula here: lambda x: 2 times x plus 5. Now when I'm running this stuff, you'll see this come up a lot in with the sk learn kit, and one of the nice reasons of breaking this down the way we did is I could go over those top pieces; those top pieces are everything when you start looking at these minimization toolkits in built-in code. And so from—we'll just do—it's actually docs.scipy.org, and we're looking at the scikit—there we go—optimize minimize. You can only minimize one value; you have the function that's going in; this function can be very complicated; so we used a very simple function up here; it could be—there's all kinds of things that could be on there—and there's a number of methods to solve this as far as how they shrink down, and your x naught—there's your, there's your start value. So your function, your start value; there's all kinds of things that come in here that you can look at, which we're not going to—optimization automatically creates constraints, bounds; some of this it does automatically, but you really—the big thing I want to point out here is you need to have a starting point; you want to start with something that you already know is mostly the answer. If you don't, then it's going to have a heck of a time trying to calculate it out, or you can write your own little script that does this and does a high low guessing and tries to find the max value. That brings us to statistics; what this is kind of all about is figuring things out; a lot of vocabulary and statistics—ah, so statistics—well, I guess it's all relative; it's definitely not an edel class. So a bunch of stuff going on—statistics. Statistics concerns with the collection, organization, analysis, interpretation, and presentation of data; that is a mouthful. Um, so we have from end to end: where does it come from? Is it valid? What does it mean? How do we organize it? Um, how do we analyze it? Then you gotta take those analysis and interpret it into something that people can use; kind of reduce it to understandable, and nowadays you have to be able to present it; if you can't present it, then no one else is going to understand what the heck you did.

So we look at the terminologies; there is a lot of terminologies depending on what domain you're working in. So clearly, if you're working in, um, a domain that deals with viruses and T cells and, and how does, you know, where does that come from, you're studying the different people that you can have a population. If you are working with, um, mechanical gear, you know, a little bit different; if you're looking for the wobbling statistics to know when to replace a rotor on a machine or something like that, that can be a big deal. You know, we have these huge fans that turn in our sewage processing systems, and so those fans, they start to wobble and hum and do different things; if the sensors pick up at one point, do you replace them instead of waiting for it to break, in which case it costs a lot of money? Instead of replacing a bushing, you're replacing the whole fan unit—an interesting project that came up for our city a while back. Uh, so population: all objects or measurements whose properties are being observed; uh, so that's your population; all the objects; it's easy to see it with people because we have our population in large, but in the case of the sewer fans, we're talking about having the fan units; that's the population of fans that we're working with. You have a parameter: a matrix that is used to represent a population or characteristic. You have your sample: a subset of the population studied; you don't want to do them all because then you don't have a—if you come up with a conclusion for everyone, you don't have a way of testing it, so you take a sample; sometimes you don't have a choice; you can only take a sample of what's going on; you can't study the whole population. And a variable: a metric of interest for each person or object in a population. Types of sampling: we have a probabilistic approach: selecting samples from a larger population using a method based on the theory of probability, and we'll go into a little bit more deeper on these. We have random, systematic, stratified, and then you have a non-probabilistic approach: selecting samples based on the subjective judgment of the researcher rather than random selection; it has to do with convenience, trying to reach a quota or snowball, and they're very biased; that's one of the reasons you'll see this big stamp on it says biased; uh, so you gotta be very careful on that.

So probabilistic sampling, uh, when we talk about a random sampling, we select random size samples from each group or category; so we, it's as random as you can get. We talk about systematic sampling; we're selecting random size samples from each group or category with a fixed periodic interval; so we kind of split it up; this would be like a time setup or our different categories, and you might ask your question: what is a category or a group? Uh, if you look at—I'm going to go back a window—let's say we're studying economics of different—of an area; we know pretty much that based on their culture, where they came from, they might need to be separated, and so—and when I say separated, I don't mean separated from their place where they live; I mean, as far as the analysis, we want to look at the different groups and make sure they're all represented. So if we had like an 80 percent, uh, of a group that is Hispanic and/or Indian and also in that same area we have 20 percent, 20 percent who are—let's call our expatriates—they left America and they're nice and, uh, your Caucasian group, we might want to sample a group that is representative of both; uh, so we're talking about stratified sampling, and we're talking about groups; those are the groups we're talking about. And that brings us to stratified sampling: selecting approximately equal size samples from each group or category; this way we can actually separate the categories and give us an insight into the different cultures and how that might affect them in that area. So you can see these are very, very different; kind of depends on what you're working with as far as your data and what you're studying. And so we can see here just a little bit more; we'd have selecting 25 employees from a company of 250 employees randomly; don't care anything about them; what groups are in which office they're in, nothing; and we might be selecting one employee from every 50 unique employees in a company of 250 employees; and then we have selecting one employee from every branch in the company office; so we have all the different branches; there's our group or categories by the branch; the category could depend on what you're studying; so it has a lot of variation on there. You see this kind of grouping and categorizing is also used to generate a lot of misinformation; so if you only study one group and you say this is what it is, then everybody assumes that's what it is for everybody, and so you've got to be very careful of that, and it's a very unethical thing to kind of do.

So types of statistics: we talk about statistics; we're going to talk about descriptive and inferential statistics; there are so many different terms and statistics to break it up; uh, so we, so we're talking about a particular setup; so we're talking about descriptive and inferential statistics. The base of the word describe is pretty solid; you're describing the data; what does it look like? With inferential statistics, we're going to take that from the small population to a large population. So if you're working with a drug company, you might look at the data and say these people were helped by this drug; they did 80 percent better as far as their health or 80 percent better survival rate than the people who did not have the drug; so we can infer that that drug will work in the greater populous and will help people; so that's where you get your inferential; so we are predicting how it's going to affect the greater population. So descriptive statistics: it is used to describe the basic features of data and form the basis of quantitative analysis of data. So we have a measure of central tendencies; we have your mean, median, and mode; and then we have a measure of spread like your range, your interquartile range, your variance, and your standard deviation; and we're going to look at all these a little deeper here in a second, but one of them you can think of is how

The data: difference, differences. You know what's the max, min, range, all that stuff is your spread. And anything that's just a single number is usually your central, uh, tendencies, measure of central tendencies. So we talk about the mean; it is the average of the set of values considered. What is the average outcome of whatever's going on? And then your median separates the higher half and the lower half of data. So, where's the center point of all your different data points? So your mean might have some, a couple really big numbers that skew it so that the average is much higher than if you took those outliers out, where the median, by separating the high from the low, might give you a much lower number. You might look at and say, "Oh, that's that's odd. Why is the average so much higher than the median?" Well, it's because you have some outliers. Or why is it so much lower? And then the mode is the most frequent appearing value. This is really interesting if you're studying economics and how people are doing. You might find that the most common income, like in the U.S., was 1.24000 a year, where the average was closer to 80,000. And it's like, "Wow, what a difference!" Well, there's some people have a lot of money, and so that skews that way up. So the average person is not making that kind of money. And then you look at the median income, and you're like, "Well, the median income is a little bit closer to the average," so it does create a very interesting way of looking at the data. Again, these are all central tendencies, single numbers you can look at for the whole spread of the data.

And we look at the measure of central tendencies: the mean is the average marks of a students in a classroom. So here we have the mean: sum of the marks of the students/total number of students. And as we talked about the median, we have 0 through 10, and we take half the numbers and put them on one side of the line, half the numbers on the other side of the line; uh, we end up with five in the middle. And then the mode: what mark was scored by most of the students in a test? In a simple case where most people scored like an 82 percent and got certain problems wrong, easy to figure out; uh, not so easy when you have different areas where like you have like the um—oh, let's go back to economy—a little bit more difficult to calculate if you have a large group, the scores that makes thirty thousand, and a slightly bigger group that makes twenty-six thousand. So what do you put down for the mode? Uh, certainly there's a number of ways to calculate that, and there's actually different variations depending on what you're doing.

So now we're looking at a measure of spread; range: what's the difference between the highest and the lowest value? First thing you want to look at, you know, it's uh—we had everybody in the test score between 60 and 100, so we got 100, or maybe 60 to 90 percent—it was so hard that a lot of people could not get a hundred percent. You have your inter-quartile range: quartiles divide a rank ordered data set into four equal parts. Very common thing to do as part of all the basic packages, whether you're working in uh data frames with pandas, whether you're working in scala, whether you're working in R, you'll see this come up where they have range, your min, your max, and then it'll have your interquartile range: how does it look like in each quarter of data? Variance measures how far each number in the set is from the mean and therefore from every other number in the set. So you have like a how much turbulence is going on in this data. And then the standard deviation: it is to measure the variance or the dispersion of a set of values from the mean, and you'll usually see if I'm doing a graph, I might have the value graphed, and then based on the the error, I might graph, graph the standard deviation in the error on the graph as a background so you can see how far off it is; uh, so standard deviation is used a lot.

So measurement of spread; marks of a student out of a hundred, we have here from 50 to 63 or 50 to 90. So the range: maximum marks - minimum marks. We have 90 to 45, and the spread of that is 45 (90 - 45). And then we have the interquartile range, using the same marks over there, you can see here where the median is, and then there's the first quarter, the second quarter, and the third quarter based on splitting it apart by those values. And to understand the variance and standard deviation, we first need to find out the mean; uh, so here's our our, you know, calculating the average there, we end up approximately 66 for the average. And then we look at that the variance, once we know the means, we can do equals the marks minus the mean squared. Why is this squared? Uh, because one, you want to make sure it's you don't have like if you if you're putting all this stuff together, you end up with an error as far as one's negative, one's positive, one's a little higher, one's a little lower. So you always see the squared value, and over the total observations. And so the standard deviation equals the square root of the variance, which is approximately 16. And if you were looking at a predictable model, you would be looking at the deviation based on the error: how much error does it have? That's again really important to know if your if your prediction is predicting something, what's a chance of it being way off or just a little bit off.

Now that we've looked at the tools as far as some of the basics for doing your statistics, what we're talking about, let's go ahead and pull up a little demo and show you what that looks like in Python code, so you can get some little hands-on here for that. Let's go in back into our Jupyter notebook in Python. Now, almost all of this you can do in NumPy. Last time we worked in NumPy, this time we're going to go ahead and use pandas. And if you remember from pandas on here, this is basically a data frame: rows, columns. Let's just go ahead and do a print df.head and run that, and you can see we have the name: Jane, Michael, William, Rosie, Hanna, Sat, and their salaries on here. And of course, instead of having to do all those hand calculations and add everything together and divide by the total, we can do something very simple on this, like use the command mean in pandas. And so if I go ahead and do this: print df, pick our column salary because we want to find the means of that calorie, we want to find the means of that column, uh, and we go and print this out, and you can see that the average income on here is 71,000. Uh, and let's just go ahead and do this: we'll go ahead and put in means, and if we're going to do that, we also might want to find the median. And the median is very similar, except it actually is just median. We're used to means in average; it's kind of interesting that those are they use the two different words; uh, there can be in some computation slight differences, but for the most part the means is the average; uh, and then the median, oops, let's put a median here, do you have salary that way it displays a little better, we can see the median is 54,000. So the halfway mark is significantly below the average. Why? Because we have somebody in here makes 189,000. Darn you, Rosie, for throwing off our numbers! But that's something you'd want to notice: this is this is the difference between these is huge. And so is what is the meaning behind that when you're studying a populace and looking at the different data coming in. And of course, we also want to find out, hey, what's the most uh common income that people make in this little tiny sample. And so we'll go ahead and do the mode, and you can see here with the mode, uh, it's at fifty thousand. So this is this is very telling that most people are making fifty thousand; the middle point is at fifty-four thousand, so half the people are making more than that. What that tells me is that if the most common income is weight is below the median, then there's a few, there's a scale, there's a lot of high salaries going up, but there's some really low salaries in there. And so this trend, which is very common in statistics when you're analyzing the economy and different people's income, is pretty common. And the bigger difference between these is also very important when we're studying statistics. And when you hear someone just say, "Hey, the average income was," you might start asking questions at that point: why aren't you talking about the median income? Why aren't you talking about the mode, the most common income? What are you hiding? Uh, and if you're doing these analysis, you should be looking at these, saying, "Hey, why are these discrepancies? Why are these so different?" And of course, with any analysis, it's important to find out the minimum and the maximum. So we'll go ahead, it's just simply uh .min, it'll pull up your minimum, and then .max pulls up the maximum, pretty straightforward on as far as translating it and knowing which, you know, put the your lowest value and what your highest value is here, um, which you'll use to generate like a spread later on.

And real quick on no mode; note that it puts mode 0. Like I said, there's a couple different ways you can compute the mode, although the standard one is pretty good. We can of course do the range, which is your max minus your min. So now we have a range of 149,000 between the upper end and the lower end, and you might want to be looking up the individual values on all of these, but it turns out there is a describe feature in pandas. And so in pandas, we can actually do df salary describe. And if we do this, you can see we have that there's seven uh setups: here's our mean, our standard deviation which we didn't compute yet, which should just be a .std, and you've got to be a little careful because when it computes it, it looks for axes and things like that. We have our minimum value, and here's our quartiles, our maximum value, and then of course the name salary. So these are these are the basic statistics: you can pull them up and like just describe. This is a dictionary, so I could actually do something like in here, I could actually go uh count and run, and now it just prints the count. So because this is a dictionary, you can pull any one of these values out of here. It's kind of a quick and dirty way to pull all the different information and then split it up and depending on what you need. Now, if I just walked in and gave you this information in a meeting at some point, you would just kind of fall asleep; that's what I would do anyway; um, so we want to go ahead and see about graphing it here, and we'll go ahead and put it into a histogram and plot that graph on it of the salaries. And let's just go ahead and put that in here. So we do our matplot inline, remember that's a Jupyter's notebook thing; a lot of the new version of the matplot library does it automatically, but just in case I always put it in there: import matplot library; piplot is plt; that's my plotting; and then we have our data frame; I don't I guess I really don't need to respell the data frame; maybe we could just remind ourselves what's in it, so we'll go ahead and just print df that way we still have it; and then we have our salary df salary salary.plot history title salary distribution color gray; uh, plot ax v line salary the mean value, so we're going to take the mean value color violet line style dash; this is just all making it pretty; uh, what color dashed line line width of 2, that kind of thing; and the median. And let's go ahead and run this just so you can see what we're talking about. And so up here we are taking on our plot, so here's the data, here's our data frame, print it out so you can see it with the salaries; we'll look at the salary distribution, and just look at this, the way the salary is distributed: you have our in this case we did, let's see, we had red for the median, we have violet for our average or mean, and you can just see how it really, I mean here's our outlier, here's our person who makes a lot of money, here's the average, and here's the median. And so as you look at this, you can say, "Wow, based on the average, it really doesn't tell you much about what people are really taking home; all it does is tell you how much money is in this, you know, what the average salary is." So some of the things you want to take away in addition to this is that it's very easy to plot um an ax v line; these are these up and down lines for your markers. And as you just display the data, I mean you can add all kinds of things to this and get really complicated; keeping it simple is pretty straightforward. I look at this and I can see we have a major outlier out here; we can definitely do a histogram and stuff like that, but you know, a picture's worth a thousand words. What you really want to make sure you take away is that we can do a basic describe which pulls all this information out, and we can print any of the individual information from the describe because this is a dictionary. And so if we want to go ahead and look up the mean value, we can also do describe mean. So if you're doing a lot of statistics, being able to—it doesn't have the print on there, so it's only going to print the last one which happens to be the mean—you can very easily reference any one of these. And then you can also, if you're doing something a little bit more complicated and you don't need just the basics, you can come through and pull any one of the individual references from the from the pandas on here.

So now we've had a chance to describe our data; let's get into inferential statistics. Inferential statistics allows you to make predictions or inferences from data. And you can see here we have a nice little picture: movie ratings. And if we took this group of people and said, "Hey, how many people like the movie, dislike it, can't say," and then you ask just a random person who comes out of the movie who hasn't been in this study, uh, you can infer that 55 percent chance of saying liked, 35 percent chance of saying disliked, or a 10 or 11 percent chance of can't say. So that's real basics of what we're talking about is you're going to infer that the next person is going to follow these statistics. Uh, so let's look at point estimation; it is a process of finding an approximate value for a population's parameter like mean or average from random samples of the population. Let's take an example of testing vaccines for Covid-19 vaccines and flu bugs, all that; it's a pretty big thing of how do you test these out and make sure they're going to work on the populace. A group of people are chosen from the population; medical trials are performed; results are generalized for the whole population. So here's a protected; there's our small group up here where we've selected them; we run medical trials on them, and then the results work for the population. Nice diagram with the arrows going back and forth and the very scary Covid virus in the middle of one. And let's take a look at the applications of inferential statistics: very central is what they call hypotheses testing and the confidence interval which go with that. And then as we get into probability, we get into our binomial theorem or normal distribution in central limit theorem.

Hypothesis testing: hypothesis testing is used to measure the plausibility of a hypothesis, assumption by using sample data. Now, when we talk about theorem's, theory, hypothesis, keep in mind that if you are in a philosophy class, theory is the same as hypothesis, where theorem is a scientific statement that is something that has been proven, although it is always up for debate because in science we always want to make sure things are up to debate. So hypothesis is the same as a philosophical class calling a theory, where theory in science is not the same; theory in science says this has been well proven; gravity is a theory. So if you want to debate the theory of gravity, try jumping up and down. If you want to have a theory about why the economy is collapsing in your area, that is a philosophical debate, very important; I've heard people mix those up, and it is a pet peeve of mine. When we talk about hypotheses testing, the steps involved in hypotheses testing is: first, we formulate a hypothesis; we figure out the right test to test our hypothesis; we execute the test; and we make a decision. And so when you're talking about hypothesis, you're usually trying to disprove it. If you can't disprove it and it works for all the facts, then you might call that a theorem at some point.

So in a use case, uh, let's consider an example: we have four students; we're given a task to clean a room every day; sounds like working with my kids. They decided to distribute the job of cleaning the room among themselves; they did so by making four chits which has their names on it, and the name that gets picked up has to do the cleaning for that day. Rob took the opportunity to make chits and wrote everyone's name on it. So here's our four people: Nick, Rob, Emilia, Emilia, and Summer. Now, Rick, Emilia, and Summer are asking us to decide whether Rob has done some mischief in preparing the chits, i.e., whether Rob has written his name on one of the chit. For that, we will find out the probability of Rob getting the cleaning job on first day, second day, third day, and so on until 12 days. The probability of Rob getting the job decreases every day, i.e., his turn never comes up, then definitely he has done some mischief while making the chits. So the probability of Rob not doing work on day one is a three out of four; there's a 0.75 chance that he didn't do work. Two days: three fourths times three fourths equals point five six. Three days: you have three fourths, three fourths, three fourths, which equals 0.42. When you get to day 12, it's 0.032, which is less than 0.05. Remember this .05; that comes up a lot when we're talking about certain values when we're looking at statistics. Rob is cheating as he was consecutive days; that's a very high probability when on day 12 he still hasn't gotten the job cleaning the room. So we come up to our important, important terminologies: we have null hypothesis: a general statement that states that there is no relationship between two measured phenomena or no association among the groups. Alternative hypothesis: contrary to the null hypothesis; it states whenever something is happening, a new theory is preferred instead of an old one. And so the two hypotheses go hand in hand; uh, your null—this is always interesting in in when talking about data science and the math behind it—it's about proving that the things have no correlation. Null hypothesis says these two have zero relation to each other, where the alternative hypothesis says, "Hey, we found a relation; this is what it is." We have p-value: the p-value is a probability of finding the observed or more extreme results when the null hypothesis of a study question is true. And the t value: it is simply the calculated difference represented in units of standard error; the greater the magnitude of t, the greater the evidence against the null hypothesis. And you can look at the t values being specific to the test you're doing, where the p value is derived from your t value, and you're looking for what they call the five percent or the 0.05 showing that it has a high correlation.

So digging in deeper: let's assume that a new drug is developed with the goal of lowering the blood pressure more than the existing drug. And this is a good one because the null value here isn't that you don't have any drug; the null value here is that it's better than the existing drug. The new drug doesn't lower the blood pressure more than the existing drug. Now, if we get that that says our null hypothesis is correct; there is no correlation, and the new drug is not doing its job. The alternative hypothesis: the new drug does significantly lower the blood pressure more than an existing drug; yay, we got a new drug out there. And that's our alternative hypothesis, or the h1 or ha.

And we look at the p-value results from the evidence, like medical trials showing positive results, which will reject the null hypothesis. And again, they're looking for a 0.05 or 5 percent. And the t-value comparing all the positive test results and finding means of different samples in order to test hypothesis. So this is specific to the test: how what percentage of increase did they have?

And this leads us to the confidence intervals. A confidence interval is a range of values we are sure our true values of observations lie in. Let's say you asked a dog owner around you and asked them how many cans of food do you buy per year for your dog? Through calculations, you got to know that, on an average, around 95 percent of the people bought around 200 to 300 cans of food. Hence, we can say that we have a confidence interval of 200-300, where 95 percent of our values lie in that spring data spread. Uh, and this graph really helps a lot, so you can start seeing what you're looking at. Here we have the 95 percent; you have your peak. In this case, it's a normal distribution, so you have a nice bell curve, equal on both sides; it's not asymmetrical. And 95 percent of all the values lie within a very small range, and then you have your outliers, the 2.5 percent going each way.

So we touched upon hypothesis; we're going to move into probability. So you have your hypothesis; once you've generated a hypothesis, we want to know the probability of something occurring. Probability is a measure of the likelihood of an event to occur. Any event can be predicted with total certainty and can only be predicted as a likelihood of its occurrence. So any event cannot be predicted with total certainty; it can only be predicted as a likelihood of its occurrence: score prediction, how good you're going to do in whatever sport you're in; weather prediction; stock prediction. If you've studied physics and chaos theory, even the location of the chair you're sitting on has a probability that it might move three feet over, granted that probability is one in like, uh, I think we calculated as under one in trillions upon trillions. So it's the better the probability, the more likely it's going to happen. There are some things that have such a low probability that we don't see them.

So we talk about random variable. A random variable is a variable whose possible values are numerical outcomes of a random phenomena. So, uh, we have the coin tossed: how many heads will occur in a series of 20 coin flips? Probably you know the on average they're 10, but you really can't know because it's very random. How many times a red ball is picked from a bag of balls if there's an equal number of red balls and blue balls and green balls in there? How many times the sum of digits on two dice results are five each? Um, so you know there's how often you're going to roll two fives on your paradigms.

So in a use case, let's consider the example of rolling two dice. We have a random variable outcome equals y; you can take values 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12. So we have a random variable and a combination of dice. And instead of looking at how many times both dice for roll 5, let's go ahead and look at the total sum of five. And you have, as far as your random variables, you can have a one, four equals five; four, one; two, three; three, two. So four of those rolls can be four. If you look at all the different options, you have four of those random rolls can be a five. And if we look at the total number, which happens to be 36 different options, you can see that we have four out of 36 chance every time you roll the dice that you're gonna roll a total of five; you can have an outcome of five. And, uh, we'll look a little deeper as to what that means, but you could think of that: at what point if someone never rolls a five or they always roll a five, can you say, "Hey, that person's probably cheating"? We'll look a little closer at the math behind that, but let's just consider this is one of the cases: rolling two dice and gambling.

There's also binomial distribution. It is the probability of getting success or failure as an outcome in an experiment or trial that is repeated multiple times. And the key is is "bi" meaning two, binomial: passing or failing an exam; winning or losing a game; and getting either head or tails. So if you ever see binomial distribution, it's based on a true/false kind of setup: you win or lose. Let's consider a use case, and let's consider the game of football between two clubs, Barcelona and Dortmund. The teams will have to play a total of four matches, and we have to find out the chances of Barcelona winning the series. So we look at the total games, and we're looking at five different games or matches. Let's say that the winning chance for Barcelona is 75 percent or 0.75. That means that each game they have a 75 percent chance that they're going to win that game, and losing chances are 25 percent or 0.25. Clearly, 0.75 + 0.25 = 1, so that accounts for 100 percent of the game. The probability for getting k wins in n matches is calculated, and we we're talking like: so if you have five games and you want to know if I play, how many wins in those five games should I get? What's a percentage on those? And the probability for getting k wins in n matches is calculated by P(x = k) = nCk * p^k * q^(n-k). Here, p is the probability of success, and q is the probability of failure.

And so we can do total games of n = 5, where k = 0, 1, 2, 3, 4, 5; p, which is the chance of winning, is 0.75; q, the chance of losing, equals 1 - p, which equals 1 - 0.75, which equals 0.25. The probability that Barcelona will lose all of the matches can then just plug in the numbers, and we end up with a 0.0009765625; so very small chance they're going to lose all their matches. And we can plug in, uh, the value for two matches: the probability that Barcelona will win at least two matches is 0.0878. And of course, we can go on to the probability that Barcelona will win three matches: 0.26; and of course, four matches and so on. And it's always nice to take this information, um, and let's find the accumulated discrete probabilities for each of the outcomes where Barcelona has won three or more matches: x = 3, x = 4, x = 5. And we end up with the p = 0.264 + 0.395 + 0.237, which equals 0.896. In reality, the probability of Barcelona winning the series is much higher than 0.75, and it's always nice to put out a nice graph so you can actually see the number of wins to the probability and how that pans out with our binomial case.

Continuing in our important terminology: location. The location of the center of the graph depends on the mean value, and this is some very important things. So much of the data we look at, and when you start looking at probabilities, almost always has a normalized look like the graph in the middle, but you do have left-skewed where the data is skewed off to the left, and you have more stuff happening off to the left, and you have right-skewed data. And so when this comes up and these probabilities come up where they're skewed, it's really important to take a closer look at that. Mostly you end up with a normalized set of data, but you've got to also be aware that sometimes it's skewed data. And then the height: the height of the slope inversely depends upon the standard deviation. So you can see down here the standard deviation is really large; it kind of squishes it out. And if the standard deviation is small, then most of your data is going to hit right there in the middle; you can have a nice peak. And so being aware of this, that you might have a probability that fits certain data but it has a lot of outliers, so you're—if you have a really high standard deviation—if you're doing stock market analysis, this means your predictions are probably not going to make you much money, where if you have a very small deviation, you might be right on target and set to become a millionaire.

Which leads us to the z-score. Z-score tells you how far from the mean a data point is; it is measured in terms of standard deviations from the mean. Around 68 percent of the results are found between one standard deviation; around 95 percent of the results are found between two standard deviations. And you read the symbols; of course, they love to throw some Greek letters in there. We have μ - 2σ; μ is just a quick way—it's a kind of funky u—it just means the mean, uh, and then the σ is the standard deviation, and that's the o with the little arrow off to the right or the little waggy tail going up, the o with it with the line on it. So μ - 2σ: 95 percent of the results are found between two standard deviations.

Central limit theorem. This goes back to the skew. If you remember, we were looking at the skew values on this previous slide: have left-skewed, normalized, and right-skewed. When we're talking about it being skewed or not skewed, the distribution of the sample means will be approximately normally distributed—evenly distributed, not skewed—if you take large random samples from the population with the mean μ and the standard deviation σ with replacement. And you can see here, of course, we have our μ - 2σ and the spread down here, the mean, the median, and the mode. And so you're talking about very large populations; these numbers should come together, and you shouldn't have a skewed value. If you do, that's a flag that something's wrong. That's why this is so important to be aware of what's going on with your data, where your samples are coming from, and the math behind it.

And if we're going to do all this, we've got to jump into conditional probability. The conditional probability of an event A is the probability that the event will occur given the knowledge that an event B has already occurred. And you'll see this as Bayes theorem: B-A-Y-E-S, Bayes. And this is red—I mean, you have these funky-looking little P brackets (A|B). This is the probability of A being true while B is already true. And you have the probability of B being true when A is already true. So P(B|A): probability of A being true divided by the probability of B being true. And we talk about Bayes theorem, which occurred back in the 1800s when he discovered this. This is such an important formula, and it's really—it's not—if you actually do the math, you could just kind of do x * y = j * k, and then you divide them out, and you're going to see the same math, but it works with probabilities, which makes it really nice. And so if you have a set, you might have eight or nine different studies going on in different areas; different people have done the studies; they brought them together. If we look at today's COVID virus, the virus spread, certainly the studies done in China versus the studies, the way they're done in the U.S., that data is different in each of those studies. But if you can find a place where it overlaps, where they're studying the same thing together, you can then compute the changes that you need to make in one study to make them equal. And this is also true if you have a study of one group and you want to find out more about it. So this formula is very powerful, and it really has to do with the data collection part of the math and data science and understanding where your data is coming from and how you're going to combine different studies in different groups.

And we're going to go into a use case. Let's find out the chance of a person getting lung disease due to smoking. And this is kind of interesting, the way they word this. Let's say that according to a medical report provided by the hospital, states that around 10 percent of all patients they treated suffered lung disease. So we have kind of a generic medical report. They further found out by a survey that 15 percent of the patients that visit them smoke. So we have 10 percent that are lung disease, and 15 percent of the patients smoke. And finally, 5 percent of the people continued to smoke even when they had lung disease—not the brightest choice—um, but you know, it is an addiction, so it can be really difficult to kick. And so we can look at the probability of A: prior probability of 10 percent people having lung disease; and then probability B: probability that a patient smokes is 15 percent; uh, and the probability of B, if B then A: the probability of a patient smokes even though they have lung disease is 5 percent. And probability of A is B: probability that the patient will have lung disease if they smoke. And then when you put the formulas together, you get a nice solution here; you get the probability of A of B: probability that the patient will have lung disease if they smoke, and you can just plug the numbers right in, and we get a 3.33 percent chance. Hence, there is a 3.33 percent chance that a person who smokes will get lung disease.

So we're going to pull up a little Python code; I'm always my favorite: roll up the sleeves. Keep in mind we're going to be doing this, um, kind of like the back-end way so that you can see what's going on, and then later on we're going to create—we'll get into another demo which shows you some of the tools already pre-built for this. Let's start by creating a set. So we're going to create a set with curly braces. This means that our set has only unique values. So you have a list; you have your tuples, which can never change; and then you have, in this case, the the set. So {4, 7}; you can't create a {4, 7, 4}; it'll delete the four out; it's only unique values. And if you use dictionaries—quick reminder—this should look familiar because it is a dictionary—we have a value, and that value is assigned to or that key is assigned to a value. So you could have a key-value setup as a dictionary. So it's like a dictionary without the value; it's just the keys, and they all have to be unique. And if we run this, we have a set of {4, 7}. We can also take a list, a regular setup, and I'm going to go ahead and just throw in another number in here: four, and run it. And you can see here, if I take my list [1, 2, 3, 4, 4] and I convert it to a set, and here it is: my_set_from_list = set(my_list), the result is {1, 2, 3, 4}. So it just deletes that last four right out of there. And with the sets, you can also go in there and print: "Here is my set," my_set, "{3} is in the set," and then if you do 3 in my_set, that's going to be a logic function, uh, and 1 in my_set; 6 is not in the set, and so forth. If we run this, we get: "3 is in the set: True," "1 is in the set: False," because {3, 5, 7} is another one; "6 is in the set: False," "6 is not in the set: True," "6 not in my_set: True." You can also use this with the list; we could have just used [3, 5, 7], and it would have, um, the same response on there is three. And usually do if 3 is in, but 3 in my_set still works on just a regular list.

And we'll go ahead and do a little iteration. We're going to do kind of the dice one, remember: 1, 2, 3, 4, 5, 6. And so we're going to bring in the iteration tool and import itertools as product. And I'll show you what that means in just a second. So we have our two dice; we have dice_a, and it's going to be a set of values; you can only have one value for each one; that's why they put it in a set. And if you remember from range, it is up to seven, so this is going to be {1, 2, 3, 4, 5, 6}; it will not include the seven. And the same thing for our dice_b. And then we're gonna do is we're gonna create a list which is the product of a and b. So what's uh a + b? And if we go ahead and run this, it'll print that out, and you'll see in this case when they say product because it's an iteration tool, we're talking about creating a tuple of the two. So we've now created a tuple of all possible outcomes of the dice where dice_a is {1, 2, 3, 4, 5, 6} and dice_b is {1, 2, 3, 4, 5, 6}. And you can see (1, 1), (1, 2), (1, 3), and so forth. You remember we had a slide on this earlier where we talked about the different—all the different outcomes of a dice. We can play around with this a little bit; we can do: in_dice = 2; dice_faces = {1, 2, 3, 4, 5, 6}; uh, another way of doing what we did before. And then we can create an event space where we have a set which is the product of the dice_faces: repeat = in_dice. And we'll go ahead and just run this, and you can see here it just again puts it through all the different possible variables we can have. And then if we wanted to take the same set on here and print them all out like we had before, we can just go through: for outcome in event_space: outcome, and equals. So the event_space is creating a sequence. And as you can see here when we print it out, it stacks them versus going through and putting them in a nice line. And we'll go ahead and do something: let's go print—since we have the end printing with a comma, that just means it's just gonna—it's not gonna hit the return going down to the next line. And we'll go ahead and do the length of our event_space; that'll be an important variable we're going to want to know in a minute. And of course, if I get carried away with my typing of length, we'll print it twice and give me an error. So we have 36 different possible variations here. And we might want to calculate something like: what about the multiple of 3? What if we want to have the probability of the multiple of 3 in our setup? And so we can put together the code for the outcome in event_space: x, y = outcome; if (x + y) % 3 == 0: then it's a favorable outcome; we're going to pop that outcome on the end there, and we'll turn it into a set. So favorable_outcome = set(...). Not necessary because we know it's not going to be repeating itself, but just in case, we'll go ahead and do that. And if we want to print out the outcome, we can go ahead and see what that looks like. And you can see here these are all multiples of 3: 1 + 2 = 3; 5 + 4 = 9, which divided by 3 is 3, and so forth. And just like we looked up the length of the one before, let's go ahead and print the length of our f_outcome, so we can see what that looks like. There we go. And of course, I did forget to add the print in the middle because we're looping through and putting an end on the on the setup on there. So we're going to put the print in there. And if I run this, you can see we end up with 12. So we have 36 total options; we have 12 that are multiple that add up to a multiple of 3. And we can easily compute the probability of this, uh, by simply taking the length of our favorable_outcome over the length of the event_space. And if we print it out—let me put that in there—probability: last line. So we just type it in; we end up with the 0.3333 chance; that's roughly a third. And we want to make this look nice, so let's go ahead and put in another line there: "The probability of getting the sum which is a multiple of three is 0.3333." We can compute the same thing for five dice, and if we do this for five dice and go and run it—yeah, you can see we just have a huge amount of choices, so it just goes…

On and on down here, and we can look at the length of the event space. And we have over 7776 choices; that's a lot of choices. And if we want to ask the question like we did above, uh, what is the sum where the sum is a multiple of five but not a multiple of three? We can go through all of these different options. And then you can see here d1, d2, d3, d4, d5 equals the outcome. And if you add these all together, and the division by five does not have a remainder of zero, but the remainder is also of a division by three is not equal to zero, so the multiple of five is equal to zero, but the multiple three is not, we can just append that on here. And then we can look at that, uh, favorable outcome. We'll go ahead and set that, and we'll just take a look at this: what's our length of our favorable outcome? It's always good to see what we're working with. And so we have 904 out of 7776. And then, of course, we can just do a simple division to get the probability on here. What's the probability that we're going to roll a multiple of 5 when you add them, but not a multiple of three? And so we're just going to divide those two numbers, and you can see here we get 0.116255, or 11.62 percent. And so you can really have a nice visual that this is not really complicated math right here on probabilities; uh, it's just how many options do you have and how many of those are you possibly going to be able to come up with with the solution you're looking for.

This leads us to a confusion matrix. A confusion matrix is a table which is used to describe the performance of a classification model on a set of test data for which the true values are known. And so you'll see in the left we have the predicted and the actual, and we have a negative, false negative, positive, true positive, and then we have false positive and true negative. And you can think of this as your predicted model. What does that mean? That means if you divided your data and you use two-thirds of this to create the model, you might then test it against an actual case for the last third to see how well it comes out; how many times was it true positive versus uh false positive? It gave a false positive response. And you can imagine in medical situations this is a pretty big deal; you don't want to give a false positive, so you might adjust your model accordingly so you don't have a false positive, say with a coronavirus test. It'd be better to have a false negative and they go back and get retested than to have 30 percent false positives where then the test is pretty much invalid.

So in a use case like cancer prediction, let's consider an example where a cancer prediction model is put to the test for its accuracy and precision. Actual result of a person's medical report is compared with the prediction made by the machine learning model. And so you can see here, here's our actual predicted whether they have cancer or not; you know, cancer, a big one, you don't want to have a false positive—I mean a false negative. In other words, you don't want to have it tell you that you don't have cancer when you do, so that would be something you'd really be looking for in this particular domain; you don't want a false negative. And this is again, you know, you've created a model, you have hundreds of people or thousands of pieces of data that come in; there's a real famous case study where they have the imagery and all the measurements they take, and there's about 36 different measurements they take. And then if you run the a basic model, you want to know just how accurate it is; how many negative results do you have that are either telling people they have cancer that don't or telling people that don't have cancer that they do. And then we can take these numbers and we can feed them into our accuracy, our precision, and our recall. So accuracy, precision, and recall. Accuracy, a metric to measure how accurately the results are predicted, and this is your total true where you got the right results; you add them together, the true positive, the two negative, over all the results. So what percentage of them were accurate versus what were wrong? We talked about precision is a metric to measure how many of the correctly predicted cases are actually turned out to be positive. So we have a precision on true positive; again, if you're talking about like COVID testing with the viruses, you really want this to be a high number; you want this true that to be the center point where you might have the opposite if you're dealing with a cancer where you want no false negatives. So this is your metric on here; precision is your test positive, true positive plus false positive. And then your recall, how many of the actual positive cases we were able to predict quickly with our model, so test positive is the test positive plus the false negative on there.

Let's look at an example of a common use for linear regression: profit estimation of a company. If I was going to invest in a company, I would like to know how much money I could expect to make. So we'll take a look at a venture capitalist firm and try to understand which companies they should invest in. So we'll take the idea that we need to decide the companies to invest in; we need to predict the profit the company makes, and we're going to do it based on the company's expenses, and even just a specific expense in this case. We have our company; we have the different expenses, so we have our R&D, which is your research and development; we have our marketing; we might have the location; we might have what kind of administration is going through. Based on all this different information, we would like to calculate the profit. Now, in actuality, there's usually about 23 to 27 different markers that they look at if they're a heavy-duty investor; we're only going to take a look at one basic one. We're going to come in, and for simplicity, let's consider a single variable R&D and find out which companies to invest in based on that. So we take R&D, and we're plotting the profit based on the R&D expenditure; how much money they put into the research and development, and then we look at the profit that goes with that. We can predict a line to estimate the profit. So we draw a line right through the data; when you look at that, you can see how much they invest in the R&D as a good marker as to how much profit they're going to have. We can also note that companies spending more on R&D make good profit, so let's invest in the ones that spend a higher rate in their R&D.

What's in it for you? First, we'll have an introduction to machine learning, followed by machine learning algorithms; these will be specific to linear regression and where it fits into the larger model. Then we'll take a look at applications of linear regression, understanding linear regression and multiple linear regression. Finally, we'll roll up our sleeves and do a little programming in use case profit estimation of companies. Let's go ahead and jump in. Let's start with our introduction to machine learning along with some machine learning algorithms and where that fits in with linear regression. Let's look at another example: machine learning based on the amount of rainfall; how much would be the crop yield? So here we have our crops; we have our rainfall, and we want to know how much we're going to get from our crops this year. So we're going to introduce two variables: independent and dependent. The independent variable is a variable whose value does not change by the effect of other variables and is used to manipulate the dependent variable; it is often denoted as x. In our example, rainfall is the independent variable. This is a wonderful example because you can easily see that we can't control the rain, but the rain does control the crop. So we talk about the independent variable controlling the dependent variable. Let's define dependent variable as a variable whose value change when there is any manipulation the values of the independent variables; it is often denoted as y. And you can see here our crop yield is dependent variable, and it is dependent on the amount of rainfall received.

Now that we've taken a look at a real-life example, let's go a little bit into the theory and some definitions on machine learning and see how that fits together with linear regression: numerical and categorical values. Let's take our data coming in; this is kind of random data from any kind of project. We want to divide it up into numerical and categorical. So numerical is numbers: age, salary, height; where categorical would be a description: the color, a dog's breed, gender; is limited to very specific items where numerical is a range of information. Now that you've seen the difference between numerical and categorical data, let's take a look at some different machine learning definitions. We look at our different machine learning algorithms; we can divide them into three areas: supervised, unsupervised, reinforcement. We're only going to look at supervised today. Unsupervised means we don't have the answers; we're just grouping things. Reinforcement is where we give positive and negative feedback to our algorithm to program it, and it doesn't have the information until after the fact. But today we're just looking at supervised because that's where linear regression fits in. In supervised data, we have our data already there and our answers for a group, and then we use that to program our model and come up with an answer. The two most common uses for that is through the regression and classification. Now we're doing linear regression, so we're just going to focus on the regression side. And in the regression, we have simple linear regression; we have multiple linear regression; and we have polynomial linear regression. Now on these three: simple linear regression is the examples we've looked at so far; we have a lot of data, and we draw a straight line through it. Multiple linear regression means we have multiple variables; remember where we had the rainfall and the crops; we might add additional variables in there like how much food do we give our crops, when do we harvest them; those would be additional information add in to our model, and that's why we multiple linear regression. And finally, we have polynomial linear regression; that is, instead of drawing a line, we can draw a curved line through it.

Now that you see where regression model fits into the machine learning algorithms and we're specifically looking at linear regression, let's go ahead and take a look at applications for linear regression. Let's look at a few applications of linear regression: economic growth, used to determine the economic growth of a country or a state in the coming quarter; can also be used to predict the GDP of a country; product price, can be used to predict what would be the price of a product in the future; we can guess whether it's going to go up or down, or should I buy today; housing sales, to estimate the number of houses a builder would sell and what price in the coming months; score predictions, cricket fever, to predict the number of runs a player would score in the coming matches based on the previous performance. I'm sure you can figure out other applications you could use linear regression for. So let's jump in and let's understand linear regression and dig into the theory. Understanding linear regression: linear regression is the statistical model used to predict the relationship between independent and dependent variables by examining two factors. The first important one is which variables in particular are significant predictors of the outcome variable, and the second one that we need to look at closely is how significant is the regression line to make predictions with the highest possible accuracy; if it's inaccurate, we can't use it, so it's very important we find out the most accurate line we can get. Since linear regression is based on drawing a line through data, we're going to jump back and take a look at some Euclidean geometry. The simplest form of a simple linear regression equation with one dependent and one independent variable is represented by y = m * x + c. And if you look at our model here, we plotted two points on here: x1 and y1, x2 and y2; y being the dependent variable, remember that from before, and x being the independent variable. So y depends on whatever x is; m in this case is the slope of the line, where m equals the difference in the y2 - y1 and x2 - x1. And finally, we have c, which is the coefficient of the line or where it happens to cross the zero axes.

Let's go back and look at an example we used earlier of linear regression; we're going to go back to plotting the amount of crop yield based on the amount of rainfall. And here we have our rainfall; remember we cannot change rainfall, and we have our crop yield which is dependent on the rainfall. So we have our independent and our dependent variables; we're going to take this and draw a line through it as best we can through the middle of the data. And then we look at that; we put the red point on the y-axis is the amount of crop yield you can expect for the amount of rainfall represented by the green dot. So if we have an idea what the rainfall is for this year and what's going on, then we can guess how good our crops are going to be, and we've created a nice line right through the middle to give us a nice mathematical formula. Let's take a look and see what the math looks like behind this. Let's look at the intuition behind the regression line. Now, before we dive into the math and the formulas that go behind this and what's going on behind the scenes, I want you to note that when we get into the case study and we actually apply some Python script that this math you're going to see here is already done automatically for you; you don't have to have it memorized. It is, however, good to have an idea what's going on, so if people reference the different terms you'll know what they're talking about. Let's consider a sample data set with five rows and find out how to draw the regression line. We're only going to do five rows because if we did like the rainfall with hundreds of points of data that would be very hard to see what's going on with the mathematics, so we'll go ahead and create our own two sets of data, and we have our independent variable x and our dependent variable y. And when x was 1, we got y = 2; when x was 2, y was 4; and so on and so on. If we go ahead and plot this data on a graph, we can see how it forms a nice line through the middle; you can see where it's kind of grouped going upwards to the right. The next thing we want to know is what the means is of each of the data coming in, the x and the y. The means doesn't mean anything other than the average, so we add up all the numbers and divide by the total, so 1 + 2 + 3 + 4 + 5 / 5 = 3, and the same for y, we get 4. If we go ahead and plot the means on the graph, we'll see we get 3, 4, which draws a nice line down the middle, a good estimate here. We're going to dig deeper into the math behind the regression line. Now remember before I said you don't have to have all these formulas memorized or fully understand them, even though we're going to go into a little more detail of how it works. And if you're not a math wiz and you don't know if you've never seen the sigma character before, which looks a little bit like an e that's opened up, that just means summation; that's all that is. So when you see the sigma character, it just means we're adding everything in that row, and for computers this is great because as a programmer you can easily iterate through each of the x, y points and create all the information you need. So in the top half you can see where we've broken that down into pieces, and as it goes through the first two points it computes the squared value of x, the squared value of y, and x * y, and then it takes all of x and adds them up, all of y adds them up, all of x squared adds them up, and so on and so on, and you can see we have the sum of = 15, the sum is = 20, all the way up to x * y where the sum = 66. This all comes from our formula for calculating a straight line where y = the slope * x + the coefficient c. So we go down below and we're going to compute more like the averages of these, and we'll explain exactly what that is in just a minute and where that information comes from is called the square means error, but we'll go into that in detail in a few minutes. All you need to do is look at the formula and see how we've gone about computing it line by line instead of trying to have a huge set of numbers pushed into it. And down here you'll see where the slope m =, and then the top part if you read through the brackets you have the number of data points times the sum of x * y, which we computed one line at a time there, and that's just the 66, and take all that and you subtract it from the sum of x times the sum of y, and those have both been computed, so you have 15 * 20. And on the bottom we have the number of lines times the sum of x squared, easily computed as 86 for the sum minus; I'll take all that and subtract the sum of x squared, and we end up as we come across with our formula; you can plug in all those numbers, which is very easy to do on the computer; you don't have to do the math on a piece of paper or calculator, and you'll get a slope of 0.6, and you'll get your c coefficient. If you continue to follow through that formula, you'll see it comes out as = 2.2.

Continuing deeper into what's going behind the scenes, let's find out the predicted values of y for corresponding values of x using the linear equation where m = 0.6 and c = 2.2. We're going to take these values and we're going to go ahead and plot them; we're going to predict them, so y = 0.6 * where x = 1 + 2.2 = 2.8; so on and so on. And here the blue points represent the actual y values, and the brown points represent the predicted y values based on the model we created. The distance between the actual and predicted values is known as residuals or errors. The best fit lines should have the least sum of squares of these errors, also known as e-square. If we put these into a nice chart where you can see x and you can see y what we actual values were and you can see y I predict it, you can easily see where we take y - y predicted and we get an answer; what is the difference between those two? And if we square that, y - y prediction squared, we can then sum those squared values; that's where we get the 0.64 + the 0.36 + 1 all the way down until we have a summation = 2.4. So the sum of squared errors for this regression line is 2.4. We check this error for each line and conclude the best fit line having the least e-square value. In a nice graphical representation, we can see here where we keep moving this line through the data points to make sure the best fit line has the least square distance between the data points and the regression line. Now we only looked at the most commonly used formula for minimizing the distance; there are lots of ways to minimize the distance between the line and the data points, like sum of squared errors, sum of absolute errors, root mean square error, etc. What you want to take away from this is whatever formula is being used, you can easily using a computer programming and iterating through the data calculate the different parts of it that way. These complicated formulas you see with the different summations and absolute values are easily computed one piece at a time. Up until this point we've only been looking at two values, x and y; well, in the real world it's very rare that you only have two values when you're figuring out a solution, so let's move on to the next topic: multiple linear regression. Let's take a brief look at what happens when you have multiple inputs. So in multiple linear regression we have—well, we'll start with the simple linear regression where we had y = m + x + c, and we're trying to find the value of y. Now with

Multiple linear regression: We have multiple variables coming in, so instead of having just x, we have x1, x2, x3, and instead of having just one slope, each variable has its own slope attached to it. As you can see here, we have m1, m2, m3, and we still just have the single coefficient. So when you're dealing with multiple linear regression, you basically take your single linear regression and you spread it out, so you have y equals m1 times x1 plus m2 times x2, so on all the way to m to the nth x to the nth, and then you add your coefficient on there.

Implementation of linear regression: Now we get into my favorite part. Let's understand how multiple linear regression works by implementing it in Python. If you remember before, we were looking at a company and just based on its R&D, trying to figure out its profit. We're going to start looking at the expenditure of the company. We're going to go back to that; we're going to predict its profit, but instead of predicting it just on the R&D, we're going to look at other factors like administration costs, marketing costs, and so on. And from there, we're going to see if we can figure out what the profit of that company is going to be.

To start our coding, we're going to begin by importing some basic libraries. And we're going to be looking through the data before we do any kind of linear regression. We're going to take a look at the data to see what we're playing with. Then we'll go ahead and format the data to the format we need to be able to run it in the linear regression model, and then from there we'll go ahead and solve it and just see how valid our solution is. So let's start with importing the basic libraries. Now, I'm going to be doing this in an Anaconda Jupyter Notebook, a very popular IDE. I enjoy it; it's such a visual to look at and so easy to use. Just any IDE for Python will work just fine for this, so break out your favorite Python IDE.

So here we are in our Jupyter Notebook. Let me go ahead and paste our first piece of code in there, and let's walk through what libraries we're importing. First, we're going to import numpy as np, and then I want you to skip one line and look at import pandas as pd. These are very common tools that you need with most of your linear regression. The numpy, which stands for number Python, is usually denoted as np, and you have to almost have that for your sklearn toolbox; you always import that right off the beginning. Pandas, although you don't have to have it for your sklearn libraries, it does such a wonderful job of importing data, setting it up into a data frame so we can manipulate it rather easily, and it has a lot of tools also in addition to that, so we usually like to use the pandas when we can, and I'll show you what that looks like. The other three lines are for us to get a visual of this data and take a look at it, so we're going to import matplotlib.pyplot as plt and then seaborn as sns. Seaborn works with the matplotlib library, so you have to always import matplotlib, and then seaborn sits on top of it, and we'll take a look at what that looks like. You could use any of your own plotting libraries you want; there's all kinds of ways to look at the data. These are just very common ones, and the seaborn is so easy to use; it just looks beautiful. It's a nice representation that you can actually take and show somebody. And the final line is the %matplotlib inline; that is only because I'm doing an inline IDE; my interface in the Anaconda Jupyter Notebook requires I put that in there, or you're not going to see the graph when it comes up. Let's go ahead and run this; it's not going to be that interesting. So we're just setting up variables; in fact, it's not going to do anything that we can see, but it is importing these different libraries and setup.

The next step is load the data set and extract independent and dependent variables. Now here in the slide, you'll see companies = pd.read_csv and it has a long line there with the file at the end 1000 companies.csv. You're going to have to change this to fit whatever setup you have, and the file itself you can request; just go down to the commentary below this video and put a note in there, and Simply Learn will try to get in contact with you and supply you with that file so you can try this coding yourself. So we're going to add this code in here, and we're going to see that I have companies = pd.read_csv, and I've changed this path to match my computer: C:/Simply Learn/1000_companies.csv, and then below there we're going to set the x equals to companies[: , -1], and because this is companies as a pd data set, I can use this nice notation that says take every row—that's what the colon, the first colon is—comma except for the last column—that's what the second part is where we have a colon minus one—and we want the values set into there. So x is no longer a data set, a pandas data set, but we can easily extract the data from our pandas data set with this notation. And then y we're going to set equal to the last row. Well, the question is going to be what are we actually looking at? So let's go ahead and take a look at that, and we're going to look at the companies.head, which lists the first five rows of data, and I'll open up the file in just a second so you can see where that's coming from. But let's look at the data in here as far as the way the pandas sees it. When I hit run, you'll see it breaks it out into a nice setup. This is what pandas—one of the things pandas is really good about—is it looks just like an Excel spreadsheet; you have your rows, and remember when we're programming we always start with zero; we don't start with one. So it shows the first five rows: 0, 1, 2, 3, 4, and then it shows your different columns: R&D Spend, Administration, Marketing Spend, State, Profit. It even notes that the top are column names; it was never told that, but pandas is able to recognize a lot of things that they're not the same as the data rows. Why don't we go ahead and open this file up in a CSV so you can actually see the raw data?

So here I've opened it up as a text editor, and you can see at the top we have R&D Spend, comma, Administration, comma, Marketing Spend, comma, State, comma, Profit. Carrier's return; I don't know about you, but I go crazy trying to read files like this; that's why we use the pandas. You could also open this up in an Excel, and it would separate it since it is a comma-separated variable file, but we don't want to look at this one; we want to look at something we can read rather easily. So let's flip back and take a look at that top part, the first five rows. Now, as nice as this format is where I can see the data, to me it doesn't mean a whole lot. Maybe you're an expert in business and investments, and you understand what $165,349.20 compared to the administration cost of $136,897.80, so on, so on, helps to create the profit of $192,261.83. That makes no sense to me whatsoever—no pun intended. So let's flip back here and take a look at our next set of code where we're going to graph it so we can get a better understanding of our data and what it means. So at this point, we're going to use a single line of code to get a lot of information so we can see where we're going with this. Let's go ahead and paste that into our notebook and see what we got going. And so we have the visualization, and again we're using sns, which is pandas, as you can see we imported the matplotlib.pyplot as plt, which then the seaborn uses, and we imported the seaborn as sns, and then that final line of code helps us show this in our inline coding. Without this, it wouldn't display, and you could display it to a file in other means, and that's the matplotlib inline with the % sign at the beginning. So here we come down to the single line of code; seaborn is great because it actually recognizes the panda data frame, so I can just take the companies.corr for coordinates, and I can put that right into the seaborn, and when we run this we get this beautiful plot. And let's just take a look at what this plot means. If you look at this plot on mine, the colors are probably a little bit more purplish and blue than the original one; we have the columns and the rows; we have R&D spending, we have administration, we have marketing spending, and profit. And if you cross-index any two of these—since we're interested in profit—if you cross-index profit with profit, it's going to show up. If you look at the scale on the right, way up in the dark, why? Because those are the same data; they have an exact correspondence. So R&D spending is going to be the same as R&D spending, and the same thing with administration costs. Right down the middle, you get this dark row or dark diagonal row that shows that this is the highest corresponding data; that's exactly the same. And as it becomes lighter, there's less connection between the data. So we can see with profit, obviously profit is the same as profit, and next it has a very high correlation with R&D spending, which we looked at earlier, and it has a slightly less connection to marketing spending and even less to how much money we put into the administration. So now that we have a nice look at the data, let's go ahead and dig in and create some actual useful linear regression models so that we can predict values and have a better profit.

Now that we've taken a look at the visualization of this data, we're going to move on to the next step. Instead of just having a pretty picture, we need to generate some hard data, some hard values. So let's see what that looks like. We're going to set up our linear regression model in two steps. The first one is we need to prepare some of our data so it fits correctly, and let's go ahead and paste this code into our Jupyter Notebook. And what we're bringing in is we're going to bring in the sklearn.preprocessing where we're going to import the LabelEncoder and the OneHotEncoder. To use the LabelEncoder, we're going to create a variable called label_encoder and set it equal to LabelEncoder. This creates a class that we can reuse for transferring the labels back and forth. Now, about now you should ask, what labels are we talking about? Let's go take a look at the data we processed before and see what I'm talking about here. If you remember when we did the companies.head and we printed the top five rows of data, we have our columns going across; we have column zero, which is R&D spending, column one, which is administration, column two, which is marketing spending, and column three is state, and you'll see under state we have New York, California, Florida. Now, to do a linear regression model, it doesn't know how to process New York; it knows how to process a number. So the first thing we're going to do is we're going to change that New York, California, and Florida, and we're going to change those to numbers; that's what this line of code does here: x = [:, 3]. The first part, the :, means that we're going to look at all the different rows, so we're going to keep them all together, but the only row we're going to edit is the third row, and in there we're going to take the label_coder and we're going to fit and transform the x also the third row. So we're going to take that third row; we're going to set it equal to a transformation, and that transformation basically tells it that instead of having a New York, it has a zero, or a one, or a two. And then finally, we need to do a OneHotEncoder, which equals OneHotEncoder(categorical_features = 3), and then we take the x and we go ahead and do that equal to onehotencoder.fit_transform(x).toarray(). This final transformation preps our data so it's completely set the way we need it; it's just a row of numbers, even though it's not in here. Let's go ahead and print x and just take a look what this data is doing. You'll see you have an array of arrays, and then each array is a row of numbers, and if I go ahead and just do row 0, you'll see I have a nice organized row of numbers that the computer now understands. We'll go ahead and take this out there because it doesn't mean a whole lot to us; it's just a row of numbers.

Next, on setting up our data, we have avoiding dummy variable trap. This is very important, why? Because the computer has automatically transformed our header into the setup, and it's automatically transformed all these different variables, so when we did the encoder, the encoder created two columns, and what we need to do is just have the one because it has both the variable and the name; that's what this piece of code does here. Let's go ahead and paste this in here, and we have x = x[:, 1:]. All this is doing is removing that one extra column we put in there when we did our OneHotEncoder and our LabelEncoder. Let's go ahead and run that, and now we get to create our linear regression model, and let's see what that looks like here. And we're going to do that in two steps. The first step is going to be in splitting the data. Now, whenever we create a predictive model of data, we always want to split it up so we have a training set and we have a testing set; that's very important; otherwise, we'd be very unethical without testing it to see how good our fit is. And then we'll go ahead and create our multiple linear regression model and train it and set it up. Let's go ahead and paste this next piece of code in here, and I'll go ahead and shrink it down a size or two so it all fits on one line. So from the sklearn.model_selection we're going to import train_test_split, and you'll see that we've created four completely different variables: we have X_train, X_test, y_train, y_test. That is the standard way that they usually reference these when we're doing different models; you usually see that a capital X and you see the train and the test and the lowercase y. What this is is x is our data going in; that's our R&D span, our administration, our marketing, and then y, which we're training, is the answer; that's the profit, because we want to know the profit of an unknown entity, so that's what we're going to shoot for in this tutorial. The next part, train_test_split, we take x and we take y; we've already created those; x has the columns with the data in it, and y has a column with profit in it, and then we're going to set the test_size = 0.2; that basically means twenty percent, so twenty percent of the rows are going to be tested; we're going to put them off to the side, so since we're using a thousand lines of data, that means that 200 of those lines we're going to hold off to the side to test for later, and then the random_state = 0; we're going to randomize which ones it picks to hold off to the side. We'll go ahead and run this; it's not overly exciting, so setting up our variables, but the next step is the next step; we actually create our linear regression model.

Now that we got to the linear regression model, we get that next piece of the puzzle. Let's go ahead and put that code in there and walk through it. So here we go; we're going to paste it in there, and let's go ahead and since this is a shorter line of code, let's zoom up there so we can get a good look. And we have from the sklearn.linear_model we're going to import LinearRegression. Now, I don't know if you recall from earlier when we were doing all the math, let's go ahead and flip back there and take a look at that. Do you remember this, or we had this long formula on the bottom, and we were doing all this summarization, and then we also looked at setting it up with the different lines, and then we also looked all the way down to multiple linear regression where we're adding all those formulas together? All of that is wrapped up in this one section. So what's going on here is I'm going to create a variable called regressor, and the regressor = LinearRegression(). That's a linear regression model that has all that math built in, so we don't have to have it all memorized or have to compute it individually. And then we do the regressor.fit; in this case, we do X_train and y_train because we're using the training data; x being the data in and y being profit we're looking at, and this does all that math for us, so within one click and one line we've created the whole linear regression model, and we fit the data to the linear regression model. And you can see that when I run the regressor, it gives an output LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False). It's just giving you some general information on what's going on with that regressor model.

Now that we've created our linear regression model, let's go ahead and use it. And if you remember, we kept a bunch of data aside, so we're going to do a y_predict variable, and we're going to put in the X_test, and let's see what that looks like. Scroll up a little bit; paste that in here; predicting the test set results. So here we have y_predict = regressor.predict(X_test) going in, and this gives us y_predict. Now, because I'm in Jupyter inline, I can just put the variable up there, and when I hit the run button it'll print that array out. I could have just as easily done print(y_predict). So if you're in a different IDE that's not an inline setup like the Jupyter Notebook, you can do it this way: print(y_predict), and you'll see that for the 200 different test variables we kept off to the side is going to produce 200 answers. This is what it says the profit are for those 200 predictions, but let's don't stop there; let's keep going and take a couple look. We're going to take just a short detail here and calculating the coefficients and the intercepts; this gives us a quick flash at what's going on behind the line. We're going to take a short detour here, and we're going to be calculating the coefficient and intercepts so you can see what those look like. What's really nice about our regressor we created is it already has coefficients for us; we can simply just print(regressor.coef_). When I run this, you'll see our coefficients here, and if we can do the regressor.coef_, we can also do the regressor.intercept_, and let's run that and take a look at that. This all came from the multiple regression model, and we'll flip over so you can remember where this is going into where it's coming from. You can see the formula down here where y = m1 times x1 + m2 times x2 and so on and so on + c, the coefficient. So these variables fit right into this formula: y = slope one times column one variable + slope two times column two variable all the way to the m to the n and x to the n + c, the coefficient, or in this case, you have -8.89 to the power of 2, etc., etc., times the first column and the second column and the third column, and then our intercept is the -1030009. Boy, it gets kind of complicated when you look at it; this is why we don't do this by hand anymore; this is why we have the computer to make these calculations easy to understand and calculate. Now, I told you that was a short detour, and we're coming towards the end of our script, as you remember.

From the beginning, I said if we're going to divide this information, we have to make sure it's a valid model, that this model works, and understand how good it works. So calculating the R-squared value, that's what we're going to use to predict how good our prediction is. And let's take a look what that looks like in code.

And so we're going to use this from sklearn.metrics; we're going to import r2_score. That's the R-squared value we're looking at, the error. So in the r2_score, we take our y_test versus our y_predict. y_test is the actual values we're testing; that was the one that was given to us, so we know are true. The y_predict of those 200 values is what we think it was true. And when we go ahead and run this, we see we get a 0.9352. That's the R-squared score. Now it's not exactly a straight percentage, so it's not saying it's 93 percent correct, but you do want that in the upper 90s or higher. Shows that this is a very valid prediction based on the R-squared score. And if R-squared value of 0.91 or 0.92, as we got on our model—remember it does have a random generation involved—this proves the model is a good model, which means success. Yay! We successfully trained our model with certain predictors and estimated the profit of the companies using linear regression.

What is logistic regression? Let's say we have to build a predictive model or a machine learning model to predict whether the passengers of the Titanic ship have survived or not the shipwreck. So how do we do that? So we use logistic regression to build a model for this. How do we use logistic regression? So we have the information about the passengers, their ID, whether they have survived or not, their class and name, and so on and so forth. And we use this information where we already know whether the person has survived or not; that is the labeled information. And we help the system to train based on this information, with based on this labeled data. This is known as labeled data. And during the process of building the model, we probably will remove some of the non-essential parameters or attributes. Here we only take those attributes which are really required to make these predictions. And once we train the model, we run new data through it, whereby the model will predict whether the passenger has survived or not.

So let's see what we will learn in this video. We will talk about what is supervised learning, and we will go into details about classification, which is one of the techniques for supervised learning. And then we will further focus on logistic regression, which is one of the algorithms for performing classification, especially binary classification. Then we will compare linear and logistic regression, and what are some of the logistic regression applications. And finally, we will end with a use case or a demo of actual Python code for doing logistic regression in Jupyter Notebook. All right, so let's start with what is supervised learning.

Supervised learning is one of the two main types of machine learning methods. Here we use what is known as labeled data to help the system learn. This is very similar to how we human beings learn. So let's say you want to teach a child to recognize an apple. How do we do that? We never tell the child, okay, this is an apple; has a certain diameter on the top, certain diameter at the bottom; this has a certain RGB color. No, we just show an apple to the child and tell the child, this is apple. And then next time when we show an apple, the child immediately recognizes, yes, this is an apple. Supervised learning works very similar on the similar lines.

So where does logistic regression fit into the overall machine learning process? Machine learning is divided into two types, mainly two types. There is a third one called reinforcement learning, but we will not talk about that right now. So one is supervised learning, and the other is unsupervised learning. Unsupervised learning uses techniques like clustering and association, and supervised learning uses techniques like classification and regression. Now supervised learning is used when you have labeled data; you have historical data, then you use supervised learning. When you don't have labeled data, then you used unsupervised learning. In supervised learning, there are two types of techniques that are used: classification and regression, based on what is the kind of problem we are solved. Let's say we want to take the data and classify it; it could be binary classification, like a zero or a one. An example of classification we have just seen: whether the passenger has survived or not survived, like a zero or one; that is known as binary classification. Regression, on the other hand, is you need to predict a value, what is known as a continuous value. Classification is for discrete values; regression is for continuous values. Let's say you want to predict a share price, or you want to predict the temperature that will be there; what will be the temperature tomorrow? That is where you use regression, whereas classification are discrete values: is will the customer buy the product or will not buy the product? Will you get a promotion or you will not get a promotion? I hope you're getting the idea. Or it could be multi-class classification as well. Let's say you want to build an image classification model. So the image classification model would take an image as an input and classify into multiple classes: whether this image is of a cat or a dog or an elephant or a tiger. So there are multiple classes, so not necessarily binary classification. So that is known as multi-class classification. So we are going to focus on classification because logistic regression is one of the algorithms used for classification.

Now the name may be a little confusing; in fact, whenever people come across logistic regression, it always causes confusion because the name has regression in it, but we are actually using this for performing classification. Okay, so yes, it is logistic regression, but it is used for classification. And in case you are wondering, is there something similar for regression? Yes, for regression we have linear regression. Keep that in mind. So linear regression is used for regression; logistic regression is used for classification. So in this video, we are going to focus on supervised learning, and within supervised learning, we are going to focus on classification, and then within classification, we are going to focus on the logistic regression algorithm. So first of all, classification. So what are the various algorithms available for performing classification? The first one is decision tree. There are, of course, multiple algorithms, but here we will talk about a few. Decision trees are quite popular and very easy to understand, and therefore they use for classification. Then we have k-nearest neighbors; this is another algorithm for performing classification. And then there is logistic regression, and this is what we are going to focus on in this video, and we are going to go into a little bit of details about logistic regression. All right, what is logistic regression? As I mentioned earlier, positive regression is an algorithm for performing binary classification. So let's take an example and see how this works. Let's say your car has not been serviced for quite a few years, and now you want to find out if it is going to break down in the near future. So this is like a classification problem: find out whether your car will break down or not. So how are we going to perform this classification?

So here's how it looks. If we plot the information along the x and y axis, x is the number of years since the last service was performed, and y is the probability of your car breaking down. And let's say this information was this data, rather, was collected from several car users; it's not just your car, but several car users. So that is our labeled data. So the data has been collected, and for for the number of years and when the car broke down and what was the probability, and that has been plotted along x and y axis. So this provides an idea, or from this graph we can find out whether your car will break down or not. We'll see how. So first of all, the probability can go from 0 to 1, as you all aware; probability can be between zero and one. And as we can imagine, it is intuitive as well. As the number of years are on the lower side, maybe one year, two years, or three years till after the service, the chances of your car breaking down are very limited, right? So, for example, chances of your car breaking down, the probability of your car breaking down within two years of your last service are 0.1 probability; similarly, 3 years is maybe 0.3, and so on. But as the number of years increases, let's say if it was 6 or 7 years, there is almost a certainty that your car is going to break down; that is what this graph shows. So this is an example of an application of the classification algorithm, and we will see in little details how exactly logistic regression is applied here. One more thing needs to be added here is that the dependent variables outcome is discrete. So if we are talking about whether the car is going to break down or not, so that is a discrete value. The y that we are talking about, the dependent variable that we are talking about, what we are looking at is whether the car is going to break down or not: yes or no; that is what we are talking about. So here the outcome is discrete and not a continuous value. So this is how the logistic regression curve looks. Let me explain a little bit what exactly how exactly we are going to uh determine the class at the outcome, rather. So for a logistic regression curve, a threshold has to be set, saying that because this is a probability calculation—remember this is a probability calculation—and the probability itself will not be 0 or 1, but based on the probability we need to decide what the outcome should be. So there has to be a threshold, like, for example, 0.5 can be the threshold, let's say in this case. So any value of the probability below 0.5 is considered to be 0, and any value above 0.5 is considered to be 1. So an output of, let's say, 0.8 will mean that the car will break down; so that is considered as an output of 1. And let's say an output of 0.29 is considered as zero, which means that the car will not break down. So that's the way logistic regression works.

Now let's do a quick comparison between logistic regression and linear regression because they both have the term regression in them, so it can cause confusion. So let's try to remove that confusion. So what is linear regression? Linear regression is a process; is once again an algorithm for supervised learning. However, here you're going to find a continuous value; you're going to determine a continuous value. It could be the price of a real estate property; it could be your hike, how much hike you're going to get; or it could be a stock price. These are all continuous values; these are not discrete compared to a yes or no kind of a response that we are looking for in logistic regression. So this is one example of a linear regression. Let's say the HR team of a company tries to find out what should be the salary hike of an employee. So they collect all the details of their existing employees, their ratings and their salary hikes; what has been given, and that is the labeled information that is available, and the system learns from us; it is trained, and it learns from this labeled information. So that when a new employees information is fed, based on the rating, it will determine what should be the hike. So this is a linear regression problem and a linear regression example. Now salary is a continuous value; you can get five thousand, five thousand five hundred, five thousand six hundred; it is not discrete like a cat or a dog or an apple or a banana; these are discrete, or a yes or no; these are discrete values, right? So this way you are trying to find continuous values is where we use linear regression. So let's say, just to extend on the scenario, we now want to find out whether this employee is going to get a promotion or not. So we want to find out; that is a discrete problem, right, a yes or no kind of a problem. In this case, we actually cannot use linear regression, even though we may have labeled data. So this is the label data. So based on the employee rating, these are the ratings, and then some people got the promotion, and this is the ratings for which people did not get promotion; that is a no, and this is the rating for which people got promotion. We just plotted the data about whether a person has got an employer has got promotion or not: yes, no, right? So there is nothing in between, and what is the employees rating? Okay, and ratings can be continuous; that is not an issue, but the output is discrete in this case: whether employee got promotion: yes, no, okay? So if we try to plot that and we try to find a straight line, this is how it would look, and as you can see it doesn't look very right because looks like there will be lot of errors; this root mean square error, if you remember for linear regression, would be very, very high. And also the the values cannot go beyond zero or beyond one, so the graph should probably look somewhat like this, clipped at zero and one, but still the straight line doesn't look right. Therefore, instead of using a linear equation, we need to come up with something different, and therefore the logistic regression model looks somewhat like this. So we calculate the probability, and if we plot that probability, not in the form of a straight line, but we need to use some other equation, we will see very soon what that equation is, then it is a gradual process, right? So you see here people with some of these ratings are not getting any promotions, and then slowly uh at certain rating they get promotion. So that is a gradual process, and this is how the math behind logistic regression looks. So we are trying to find the odds for a particular event happening, and this is the formula for finding the odds. So the probability of an event happening divided by the probability of the event not happening. So p, if it is the probability of the event happening, probability of the person getting a promotion, and divided by the probability of the person not getting a promotion, that is 1 minus p. So this is how you measure the odds. Now the values of the odds range from 0 to infinity. So when this probability is 0, then the odds will the value of the odds is equal to 0, and when the probability becomes 1, then the value of the odds is 1 by 0, that will be infinity, but the probability itself remains between 0 and 1. Now this is how an equation of a straight line looks: so y is equal to beta0 plus beta1 x, where beta0 is the y-intercept and beta1 is the slope of the line. If we take the odds equation and take a log of both sides, then this would look somewhat like this, and the term logistic is actually derived from the fact that we are doing this; we take a log of dx by 1 minus px; this is an extension of the calculation of odds that we have seen, right, and that is equal to beta0 plus beta1 x, which is the equation of the straight line. And now from here, if you want to find out the value of px, we will see we can take the exponential on both sides, and then if we solve that equation, we will get the equation of px like this: px is equal to 1 by 1 plus e to the power of minus beta0 plus beta1 x, and recall this is nothing but the equation of the line, which is equal to y; y is equal to beta0 plus beta1 x. So that this is the equation also known as the sigmoid function, and this is the equation of the logistic regression. All right, and if this is plotted, this is how the sigmoid curve is obtained. So let's compare linear and logistic regression, how they are different from each other. Let's go back. So linear regression is solved or used to solve regression problems, and logistic regression is used to solve classification problems. So both are called regression, but linear regression is used for solving regression problems where we predict continuous values, whereas logistic regression is used for solving classification problems where we have had to predict discrete values. The response variables in case of linear regression are continuous in nature, whereas here they are categorical or discrete in nature. And linear regression helps to estimate the dependent variable when there is a change in the independent variable, whereas here in case of logistic regression it helps to calculate the probability or the possibility of a particular event happening. And linear regression, as the name suggests, is a straight line; that's why it's called linear regression, whereas logistic regression is a sigmoid function, and the curve is the shape of the curve is s; it's an s-shaped curve. This is another example of application of logistic regression in weather prediction: whether it's going to rain or not rain. Now keep in mind both are used in weather prediction. If we want to find the discrete values, like whether it's going to rain or not rain, that is a classification problem; we use logistic regression. But if we want to determine what is going to be the temperature tomorrow, then we use linear regression. So just keep in mind that in weather prediction we actually use both, but these are some examples of logistic regression. So we want to find out whether it's going to rain or not, is going to be sunny or not, there is going to snow or not; these are all logistic regression examples.

A few more examples: classification of objects; this is a again another example of logistic regression. Now here, of course, one distinction is that these are multi-class classification, so logistic regression is not used in its original form, but it is used in a slightly different form. So we say whether it is a dog or not a dog; I hope you understand. So instead of saying is it a dog or a cat or an elephant, we convert this into saying, so because we need to keep it to binary classification. So we say is it a dog or not a dog? Is it a cat or not a cat? So that's the way logistic regression can be used for classifying objects; otherwise, there are other techniques which can be used for performing multi-class classification. In healthcare, logistic regression is used to find the survival rate of a patient. So they take multiple parameters like trauma score and age and so on and so forth, and they try to predict the rate of survival. All right. Now finally, let's take an example and see how we can apply logistic regression to predict the number that is shown in the image. So this is actually a live demo; I will take you into Jupyter Notebook and show the code, but before that let me take you through a couple of slides to explain what we are trying to do. So let's say you have an eight by eight image, and there the image has a number one, two, three, four, and you need to train your model to predict what this number is. So how do we do this? So the first thing is obviously in any machine learning process, you train your model. So in this case we are using logistic regression, so and then we provide a training set to train the model, and then we test how accurate our model is with the test data, which means that like any machine learning process, we split our initial data into two parts: training set and test set. With the training set, we train our model, and then with the test set we test the model. Then we get good accuracy, and then we use it for for inference, right? So that is typical methodology of uh training, testing, and then deploying of machine learning models. So let's uh take a look at the code and see what we are doing. So I will not go line by line, but just take you through some of the blocks. So first thing we do is import all the libraries, and then we basically take a look at the images and see what is the total number of images; we can display using matplotlib some of the images or a sample of these images. And then we split the data into training and test, as I mentioned earlier, and we can do some exploratory analysis, and then we build our model; we train our model with the training set, and then we test it with our test set and find out how accurate our model is using the confusion matrix; the heat map; use heat map for visualizing this; and I will show you in the code what exactly is the confusion matrix and how it can be used for finding the accuracy. In our example, we got we get an accuracy of about 0.94, which is pretty good, or 94, which is pretty

Good. All right. So what is the confusion matrix? This is an example of a confusion matrix, and this is used for identifying the accuracy of a classification model, or like a logistic regression model. So the most important part in a confusion matrix is that, first of all, this, as you can see, this is a matrix, and the size of the matrix depends on how many outputs we are expecting, right? So the most important part here is that the model will be most accurate when we have the maximum numbers in its diagonal, like in this case. That's why it has almost 93-94 percent, because the diagonals should have the maximum numbers, and the others—other than diagonals, the cells other than the diagonal—should have very few numbers. So here, that's what is happening. So there is a two here; there are—there's a one here, but most of them are along the diagonal. This—what does this mean? This means that the number that has been fed is zero, and the number that has been detected is also zero. So the predicted value and the actual value are the same. So along the diagonals, that is true, which means that let's—let's take this diagonal, right? If the maximum number is here, that means that, like here in this case, it is 34, which means that 34 of the images that have been fed—or rather, actually there are two misclassifications in there—so 36 images have been fed which have number four, and out of which 34 have been predicted correctly as number four, and one has been predicted as number eight, and another one has been predicted as number nine. So these are two misclassifications. Okay. So that is the meaning of saying that the maximum number should be in the diagonal. So if you have all of them—so for an ideal model which has, let's say, 100 percent accuracy, everything will be only in the diagonal; there will be no numbers other than zero in all other cells. So that is like a hundred percent accurate model. Okay. So that's a gist of how to use this matrix—uh, how to use this confusion matrix. I know the name is a little funny sounding, confusion matrix, but actually it is not very confusing; it's very straightforward. So you are just plotting what has been predicted and what is the labeled information, or what is the actual data; that's also known as the ground truth sometimes. Okay. These are some fancy terms that are used: predicted label and the actual name; that's all it is. Okay. Yeah. So we are showing a little bit more information here: 38 have been predicted, and here you will see that all of them have been predicted correctly; there have been 38 zeros, and the predicted value and the actual value is exactly the same. Whereas in this case, right, it has—uh, there are, I think, 37 plus 5, yeah, 42 have been fed, the images; 42 images are of digit 3, and the accuracy is only—37 of them have been accurately predicted; three of them have been predicted as number seven, and two of them have been predicted as number eight, and so on and so forth. Okay. All right. So with that, let's go into Jupyter Notebook and see how the code looks.

So this is the code in—in Jupyter Notebook for logistic regression. In this particular demo, what we are going to do is train our model to recognize digits, which are the images which have digits from, let's say, 0 to 5 or 0 to 9, and—and then we will see how well it is trained and whether it is able to predict these numbers correctly or not. So let's get started. So the first part is, as usual, we are importing some libraries that are required, and then the last line in this block is to load the digits. So let's go ahead and run this code. Then here we will visualize the shape of these digits. So we can see here, if we take a look, this is how the shape is: 1797 by 64. These are like eight by eight images, so that's—that's what is reflected in this shape. Now from here onwards, we are basically—once again importing some of the libraries that are required, like NumPy and MatPlot, and we will take a look at some of the sample images that we have loaded. So this one, for example, creates a figure, and then we go ahead and take a few sample images to see how they look. So let me run this code, and so that it becomes easy to understand. So these are about five images—sample images—that we are looking at: 0, 1, 2, 3, 4. So this is how the image is; this is how the data is. Okay. And based on this, we will actually train our logistic regression model, and then we will test it and see how well it is able to recognize. So the way it works is the pixel information. So as you can see here, this is an eight by eight pixel kind of image, and each pixel—whether it is activated or not activated—that is the information available for each pixel. Now based on the pattern of this activation and non-activation of the various pixels, this will be identified as a zero, for example, right? Similarly, as you can see, so overall each of these numbers actually has a different pattern of the pixel activation, and that's pretty much that our model needs to learn: for which number, what is the pattern of the activation of the pixels, right? So that is what we are going to train our model. Okay. So the first thing we need to do is to split our data into training and test data sets, right? So whenever we perform any training, we split the data into training and test, so that the training data set is used to train the system. So we pass this probably multiple times, and then we test it with the test data set, and the split is usually in the form of—and there are various ways in which you can split this data; it is up to the individual preferences. In our case here, we are splitting in the form of 23 and 77. So when we say test size as 0.23, that means 23 percent of that entire data is used for testing, and the remaining 77 percent is used for training. So there is a readily available function which is called train_test_split, so we don't have to write any special code for the splitting; it will automatically split the data based on the proportion that we give here, which is test size. So we just give the test size; automatically training size will be determined, and we pass the data that we want to split, and the—the results will be stored in x_train and y_train for the training data set, and what is x_train? These are—these are the features, right, which is like the independent variable, and y_train is the label, right? So in this case, what happens is we have the input value which is—or the features value which is in x_train, and since this is the labeled data for each of them—each of the observations—we already have the label information saying whether this digit is a zero or a one or a two, so that this is what will be used for comparison to find out whether the—the system is able to recognize it correctly or there is an error. For each observation, it will compare with this, right? So this is the label. So the same way, x_train, y_train is for the training data set; x_test, y_test is for the test data set. Okay. So let me go ahead and execute this code as well, and then we can go and check quickly what is the—how many entries are there and in each of this. So x_train, the shape is 1383 by 64, and y_train has 1383 because there is—uh, nothing like the second part is not required here, and then x_test shape we see is 414. So actually there are 414 observations in test and 1383 observations in train. So that's basically what these four lines of code are—are saying. Okay. Then we import the logistic regression library, and which is a part of scikit-learn, so we—we don't have to implement the logistic regression process itself; we just call these the function, and let me go ahead and execute that so that we have the logistic regression library imported. Now we create an instance of logistic regression, right? So logistic_regr is—is an instance of logistic regression, and then we use that for training our model. So let me first execute this code—so these two lines—the first line basically creates an instance of logistic regression model, and then the second line is where we are passing our data—the training data set, right? This is our—the—the predictors, and—this is our target; we are passing this data set to train our model. All right. So once we do this—in this case the data is not large, but by and large the training is what takes usually a lot of time. So we spend in machine learning activities—in machine learning projects—we spend a lot of time for the training part of it. Okay. So here the data set is relatively small, so it was pretty quick. So all right. So now our model has been trained using the training data set, and we want to see how accurate this is. So what we'll do is we will test it out in probably phases. So let me first try out how well this is working for one image. Okay, I will just try it out with one image—my—the first entry in my test data set—and see whether it is accurately predicting or not. So—and in order to test it—so for training purpose we use the fit method; there is a method called fit which is for training the model, and once the training is done, if you want to test for a particular value—new input—you use the predict method. Okay. So let's run the predict method, and we pass this particular image, and we see that the shape is—or the prediction is four. So let's try a few more. Let me see for the next 10; seems to be fine. So let me just go ahead and test the entire data set. Okay. That's basically what we will do. So now we want to find out how accurately this has performed, so we use the score method to find what is the percentages of accuracy, and we see here that it has performed up to 94 percent accurate. Okay. So that's on this part. Now what we can also do is we can also see this accuracy using what is known as a confusion matrix. So let us go ahead and try that as well, so that we can also visualize how well this model has done. So let me execute this piece of code, which will basically import some of the libraries that are required, and we—we basically create a confusion matrix—an instance of confusion matrix—by running confusion_matrix and passing these values. So we have—so this confusion_matrix method takes two parameters: one is the y_test, and the other is the prediction. So what is the y_test? These are the labeled values which we already know for the test data set, and predictions are what the system has predicted for the test data set. Okay. So this is known to us, and this is what the system has—the model has generated. So we kind of create the confusion matrix, and we will print it, and this is how the confusion matrix looks. As the name suggests, it is a matrix, and the key point out here is that the accuracy of the model is determined by how many numbers are there in the diagonal; the more the numbers in the diagonal, the better the accuracy is. Okay. And first of all, the total sum of all the numbers in this whole matrix is equal to the number of observations in the test data set; that is the first thing, right? So if you add up all these numbers, that will be equal to the number of observations in the test data set, and then out of that, the maximum number of them should be in the diagonal; that means the accuracy is pretty good. If the—the numbers in the diagonal are less, and in all other places there are a lot of numbers, which means the accuracy is very low. The diagonal indicates a correct prediction; that means that the actual value is the same as the predicted value. Here again, actual values same as the predicted value, and so on, right? So the moment you see a number here, that means the actual value is something, and the predicted value is something else, right? Similarly, here the actual value is something, and the predicted value is something else. So that is basically how we read the confusion matrix. Now how do we find the accuracy? You can actually add up the total values in the diagonal, so it's like 38 plus 44 plus 43 and so on, and divide that by the total number of test observations; that will give you the percentage accuracy using a confusion matrix. Now let us visualize this confusion matrix in a slightly more sophisticated way using a heat map. So we will create a heat map with some—we'll add some colors as well; it's—it's like more visually—visually more appealing. So that's the whole idea. So if we—let me run this piece of code, and this is how the heat map looks, and as you can see here, the diagonals—again, all the values are here—most of the values—so which means reasonably this seems to be reasonably accurate, and yeah, basically the accuracy score is 94 percent. This is calculated as I mentioned by adding all these numbers divided by the total test values or the total number of observations in the test data set. Okay. So this is the confusion matrix for logistic regression. All right. So now that we have seen the confusion matrix, let's take a quick sample and see how well the system has classified, and we will take a few examples of the data. So if we see here, we picked up randomly a few of them, so this is number four, which is the actual value, and also the predicted value; both are four. This is an image of zero, so the predicted value is also zero; actual value is, of course, zero. Then this is the image of nine, so this has also been predicted correctly—nine—and actual value is nine, and this is an image of one, and again this has been predicted correctly as—like the actual value. Okay.

So this was a quick demo of logistic regression—how to use logistic regression to identify images. What is a decision tree? Let's go through a very simple example before we dig in deep. A decision tree is a tree-shape diagram used to determine a course of action. Each branch of the tree represents a possible decision or occurrence or reaction. Let's start with a simple question: how to identify a random vegetable from a shopping bag? So we have this group of vegetables in here, and we can start off by asking a simple question: is it red? And if it's not, then it's going to be the purple fruit to the left—probably an eggplant. If it's true, it's going to be one of the red fruits. Is the diameter greater than 2? If false, it's going to be a—what looks to be a red chili, and if it's true, it's going to be a bell pepper from the capsicum family. So it's a capsicum. Problems that decision trees can solve. So let's look at the two different categories the decision tree can be used on: it can be used on classification—the true/false, yes/no—and it can be used on regression, where we figure out what the next value is in a series of numbers or a group of data. In classification, the classification tree will determine a set of logical if-then conditions to classify problems; for example, discriminating between three types of flowers based on certain features. In regression, a regression tree is used when the target variable is numerical or continuous in nature. We fit the regression model to the target variable using each of the independent variables. Each split is made based on the sum of squared error. Before we dig deeper into the mechanics of the decision tree, let's take a look at the advantages of using a decision tree, and we'll also take a glimpse at the disadvantages. The first thing you'll notice is that it's simple to understand, interpret, and visualize. It really shines here because you can see exactly what's going on in a decision tree. Little effort is required for data preparation, so you don't have to do special scaling; there's a lot of things you don't have to worry about when using a decision tree. It can handle both numerical and categorical data, as we discovered earlier, and non-linear parameters don't affect its performance. So even if the data doesn't fit an easy curved graph, you can still use it to create an effective decision or prediction. If we're going to look at the advantages of a decision tree, we also need to understand the disadvantages of a decision tree. The first disadvantage is overfitting. Overfitting occurs when the algorithm captures noise in the data; that means you're solving for one specific instance instead of a general solution for all the data. High variance: the model can get unstable due to small variation in data. Low bias: a highly complicated decision tree tends to have a low bias, which makes it difficult for the model to work with new data.

Decision tree important terms. Before we dive in further, we need to look at some basic terms; we need to have some definitions to go with our decision tree and the different parts we're going to be using. We'll start with entropy. Entropy is a measure of randomness or unpredictability in the data set. For example, we have a—in this case we do animals—in different kinds of animals, and this data set is considered to have a high entropy; you really can't pick out what kind of animal it is based on looking at just the four animals as a big clump of entities. So as we start splitting it into subgroups, we come up with our second definition, which is information gain. Information gain: it is a measure of decrease in entropy after the data set is split. So in this case, based on the color yellow, we've split one group of animals on one side as true and those who aren't yellow as false. As we continue down the yellow side, we split based on the height—true or false equals ten—and on the other side, height is less than ten—true or false—and as you see, as we split it, the entropy continues to be less and less and less, and so our information gain is simply the entropy e1 from the top and how it's changed to e2 in the bottom. And we'll look at the deeper math, although you really don't need to know a huge amount of math when you actually do the programming in Python because they'll do it for you, but we'll look on the actual math of how they compute entropy. Finally, we went on the different parts of our tree, and they call the leaf node. Leaf node carries the classification or the decision, so it's the final end at the bottom. The decision node has two or more branches; this is where we're breaking the group up into different parts, and finally you have the root node; the topmost decision node is known as the root node. How does a decision tree work? Wonder what kind of animals all get in the jungle today? Maybe you're the hunter with a gun, or if you're more into photography, you're a photographer with a camera. So let's look at this group of animals, and let's try to classify different types of animals based on their features using a decision tree. So the problem statement is to classify the different types of animals based on their features using a decision tree. The data set is looking quite messy, and the entropy is high in this case. So let's look at a training set—or a training data set—and we're looking at color; we're looking at height, and then we have our different animals; we have our elephants, our giraffes, our monkeys, and our tigers, and they're of different colors and shapes. Let's see what that looks like, and how do we split the data? We have to frame the conditions that split the data in such a way that the information gain is the highest. Note gain is the measure of decrease in entropy after splitting. So the formula for entropy is the sum—that's what this symbol looks like—that looks like kind of like a funky e—of k, where i equals 1 to k; k would represent the number of animals—the different animals in there—where value or p value of i would be the percentage of that animal times the log base 2 of the same—the percentage of that animal. Let's try to calculate the entropy for the current data set and take a look at what that looks like, and don't be afraid of the math.

You don't really have to memorize this math, just be aware that it's there and this is what's going on in the background. And so we have three giraffes, two tigers, one monkey, two elephants; a total of eight animals gathered. If we plug that into the formula, we get an entropy that equals three over eight. So we have three giraffes, a total of eight, times the log—usually they use base two on the log—so log base two of three over eight, plus, in this case, this hits yellow fence, two over eight (two elephants over a total of eight) times log base two of two over eight, plus one monkey over a total of eight, log base two of one over eight, and plus two over eight of the tigers, log base two of two over eight. If we plug that into our computer, our calculator—I obviously can't do logs in my head—we get an entropy equal to 0.571. The program will actually calculate the entropy of the data set similarly after every split to calculate the gain.

Now we're not going to go through each set one at a time to see what those numbers are; we just want you to be aware that this is a formula, or the mathematics behind it. Gain can be calculated by finding the difference of the subsequent entropy values after a split. Now we'll try to choose a condition that gives us the highest gain. We will do that by splitting the data using each condition and checking that the gain we get out of them; the condition that gives us the highest gain will be used to make the first split. Can you guess what that first split will be just by looking at this image? As a human, it's probably pretty easy to split it. Let's see if you're right. If you guessed the color yellow, you're correct. Let's say the condition that gives us the maximum gain is yellow. So we will split the data based on the color yellow. If it's true, that group of animals goes to the left; if it's false, it goes to the right. The entropy after the splitting has to decrease considerably; however, we still need some splitting of both the branches to attain an entropy value equal to zero. So we decide to split both the nodes using height as the condition. Since every branch now contains a single label type, we can say that entropy in this case has reached the least value. And here you see we have the giraffes, the tigers, the monkey, and the elephants all separated into their own groups. This tree can now predict all the classes of animals present in the data set with 100 percent accuracy. That was an easy use case: loan repayment prediction.

Let's get into my favorite part and open up some Python and see what the programming code in the scripting looks like. In here, we're going to want to do a prediction, and we start with this individual here who's requesting to find out how good his customers are going to be, whether they're going to repay their loan or not for his bank. And from that, we want to generate a problem statement: to predict if a customer will repay loan amount or not. And then we're going to be using the decision tree algorithm in Python. Let's see what that looks like, and let's dive into the code. In our first few steps of implementation, we're going to start by importing the necessary packages that we need from Python, and we're going to load up our data and take a look at what the data looks like. So the first thing I need is I need something to edit my Python and run it in. So let's flip on over, and here I'm using the Anaconda Jupyter Notebook. Now you can use any Python IDE you like to run it in, but I find the Jupyter Notebook's really nice for doing things on the fly. And let's go ahead and just paste that code in the beginning. And before we start, let's talk a little bit about what we're bringing in, and then we're going to do a couple things in here; we have to make a couple changes as we go through this first part of the import. The first thing we bring in is numpy as np; that's very standard when we're dealing with mathematics, especially with very complicated machine learning tools. You almost always see the numpy come in for your num—your number—it's called number Python; it has your mathematics in there. In this case, we actually could take it out, but generally you'll need it for most of your different things you work with. And then we're going to use pandas as pd; that's also a standard. The pandas is a data frame setup, and you can liken this to taking your basic data and storing it in a way that looks like an Excel spreadsheet. So as we come back to this, when you see np or pd, those are very standard uses; you'll know that that's the pandas, and I'll show you a little bit more; we explore the data in just a minute. Then we're going to need to split the data, so I'm going to bring in our train_test_split, and this is coming from the sklearn package; cross_validation. In just a minute, we're going to change that, and we'll go over that too. And then there's also the sk.tree import decision_tree_classifier; that's the actual tool we're using. Remember I told you don't be afraid of the mathematics; it's going to be done for you. Well, the decision tree classifier has all that mathematics in there for you, so you don't have to figure it back out again. And then we have sklearn.metrics for accuracy_score; we need to score our setup; that's the whole reason we're splitting it between the training and testing data. And finally, we still need the sklearn import tree, and that's just the basic tree function is needed for the decision tree classifier. And finally, we're going to load our data down here, and I'm going to run this, and we're going to get two things on here: one, we're going to get an error, and two, we're going to get a warning. Let's see what that looks like.

So the first thing we had is we have an error. Why is this error here? Well, it's looking at this; it says I need to read a file, and when this was written, the person who wrote it, this is their path where they stored the file. So let's go ahead and fix that, and I'm going to put in here my file path; I'm just going to call it full_file_name, and you'll see it's on my C drive, and this is a very lengthy setup on here where I stored the data2.csv file. Don't worry too much about the full path because on your computer it will be different. The data_2.csv file was generated by simplylearn; if you want a copy of that, you can comment down below and request it here in the YouTube. And then if I'm going to give it a name, full_file_name, I'm going to go ahead and change it here to full_file_name. So let's go ahead and run it now and see what happens. And we get a warning. When you're coding, understanding these different warnings and these different errors that come up is probably the hardest lesson to learn. So let's just go ahead and take a look at this and use this as an opportunity to understand what's going on here. If you read the warning, it says the cross_validation is depreciated, so it's a warning; it's being removed, and it's going to be moved in favor of the model_selection. So if we go up here, we have sklearn.cross_validation, and if you research this and go to the sklearn site, you'll find out that you can actually just swap it right in there with model_selection. And so when I come in here and I run it again, that removes a warning. What they've done is they've had two different developers develop it in two different branches, and then they decided to keep one of those and eventually get rid of the other one; that's all that is, and very easy and quick to fix.

Before we go any further, I went ahead and opened up the data from this file—remember the data file we just loaded on here, the data_2.csv—let's talk a little bit more about that and see what that looks like, both as a text file because it's a comma-separated variable file and in a spreadsheet. This is what it looks like as a basic text file. You can see at the top they've created a header, and it's got one, two, three, four, five columns, and each column has data in it. And let me flip this over because we're also going to look at this in an actual spreadsheet, so you can see what that looks like. And here I've opened it up in the OpenOffice Calc, which is pretty much the same as Excel, and zoomed in, and you can see we've got our columns and our rows of data—a little easier to read in here. We have a result: yes, yes, no; we have initial payment, last payment, credit score, house number. If we scroll way down, we'll see that this occupies 1001 lines of code, or lines of data, with the first one being a column and then 1000 lines of data. Now, as a programmer, if you're looking at a small amount of data, I usually start by pulling it up in different sources so I can see what I'm working with, but in larger data you won't have that option; it'll just be too, too large. So you need to either bring in a small amount that you can look at it like we're doing right now, or we can start looking at it through the Python code. So let's go ahead and move on and take the next couple steps to explore the data using Python. Let's go ahead and see what it looks like in Python to print the length and the shape of the data. So let's start by printing the length of the database. We can use a simple len function from Python, and when I run this, you'll see that it's a thousand long, and that's what we expected; there's a thousand lines of data in there. If you subtract the column head, this is one of the nice things: when we did the balance_data from the pandas.read_csv, you'll see that the header is row 0, so it automatically removes a row and then shows the data separate; it does a good job sorting that data out for us. And then we can use a different function, and let's take a look at that. And again, we're going to utilize the tools in pandas, and since the balance_data was loaded as a pandas data frame, we can do a shape on it. And let's go ahead and run the shape and see what that looks like. What's nice about this shape is not only does it give me the length of the data—we have a thousand lines—it also tells me there's five columns. So we were looking at the data; we had five columns of data. And then let's take one more step to explore the data using Python.

Now that we've taken a look at the length and the shape, let's go ahead and use the pandas module for head. Another beautiful thing in the data set that we can utilize. So let's put that on our sheet here, and we have print(dataset) and balance_data.head(). This is a pandas print statement of its own, so it has its own print feature in there, and then we went ahead and gave a label for a print job here of dataset, just a simple print statement. And when we run that, let's just take a closer look at that. Let me zoom in here. There we go. Pandas does such a wonderful job of making this a very clean, readable data set, so you can look at the data; you can look at the column headers; you can have it—when you put it as the head, it prints the first five lines of the data, and we always start with zero, so we have five lines: we have 0, 1, 2, 3, 4 instead of 1, 2, 3, 4, 5. That's a standard scripting and programming set, as you want to start with the zero position, and that is what the data.head() does; it pulls the first five rows of data, puts in a nice format that you can look at and view; very powerful tool to view the data. So instead of having to flip and open up an Excel spreadsheet or OpenOffice Calc or trying to look at a Word doc where it's all scrunched together and hard to read, you can now get a nice open view of what you're working with. We're working with a shape of a thousand long, five wide, so we have five columns, and we do the full data.head(); you can actually see what this data looks like: the initial payment, last payment, credit scores, house number. So let's take this—now that we've explored the data—and let's start digging into the decision tree.

In our next step, we're going to train and build our data tree, and to do that we need to first separate the data out; we're going to separate into two groups so that we have something to actually train the data with, and then we have some data on the side to test it to see how good our model is. Remember, with any of the machine learning, you always want to have some kind of test set to weigh it against so you know how good your model is when you distribute it. Let's go ahead and break this code down and look at it in pieces. So first we have our x and y; where did x and y come from? Well, x is going to be our data, and y is going to be the answer or the target; you can look at it as source and target. In this case, we're using x and y to denote the data in and the data that we're actually trying to guess what the answer is going to be. And so to separate it, we can simply put in x = balance_data.values[:]; the first brackets means that we're going to select all the lines in the database, so it's all the data, and the second one says we're only going to look at columns one through five; remember we always start with zero; zero is a yes or no, and that's whether the loan went default or not, so we want to start with one. If we go back up here, that's the initial payment, and it goes all the way through the house number. Well, if we want to look at one through five, we can do the same thing for y, which is the answers, and we're going to set that just equal to the zero row, so it's just the zero row, and then it's all rows going in there. So now we've divided this into two different data sets: one of them with the data going in and one with the answers. Next we need to split the data, and here you'll see that we have it split into four different parts: the first one is your x_train, your x_test, your y_train, your y_test. Simply put, we have x going in where we're going to train it, and we have to know the answer to train it with, and then we have x_test where we're going to test that data, and we have to know in the end what the y was supposed to be, and that's where this train_test_split comes in that we loaded earlier in the modules; this does it all for us. And you can see they set the test_size = 0.3, so that's roughly 30 percent will be used in the test, and then we use a random_state, so it's completely random which rows it takes out of there. And then finally we get to actually build our decision tree, and they've called it here clf_entropy; that's the actual decision tree or decision tree classifier, and in here they've added a couple variables which we'll explore in just a minute. And then finally we need to fit the data to that, so we take our clf_entropy that we created and we fit the x_train, and since we know the answers for x_train or the y_train, we go ahead and put those in. And let's go ahead and run this. And what most of these sklearn modules do is when you set up the variable—in this case we set the clf_entropy decision tree classifier—it automatically prints out what's in that decision tree. There's a lot of variables you can play within here, and it's quite beyond the scope of this tutorial to go through all of these and how they work, but we're working on entropy; that's one of the options we've added; that it's completely a random state of 100, so 100 percent, and we have a max_depth of three. Now the max_depth, if you remember above when we were doing the different graphs of animals, means it's only going to go down three layers before it stops. And then we have minimal samples of leaves is five, so it's going to have at least five leaves at the end, so I'll have at least three splits; I'll have no more than three layers and at least five end leaves with the final result at the bottom.

Now that we've created our decision tree classifier—not only created it but trained it—let's go ahead and apply it and see what that looks like. So let's go ahead and make a prediction and see what that looks like. We're going to paste our predict code in here, and before we run it, let's just take a quick look at what it's doing here. We have a variable y_predict that we're going to do, and we're going to use our variable clf_entropy that we created, and then you'll see .predict(), and it's very common in the sklearn modules that their different tools have the predict() when you're actually running a prediction. In this case, we're going to put our x_test data in here. Now, if you delivered this for use and actual commercial use and distributed it, this would be the new loans you're putting in here to guess whether the person is going to be uh pay them back or not. In this case though, we need to test out the data and just see how good our sample is; how good our tree does at predicting the loan payments. And finally, since Anaconda Jupyter Notebook is—it works as a command line for Python—we can simply put the y_predict in to print it; I could just as easily put the print and put brackets around y_predict in to print it out. We'll go ahead and do that; it doesn't matter which way you do it, and you'll see right here that runs a prediction; this is roughly 300 in here; remember it's 30 percent of a thousand, so you should have about 300 answers in here, and this tells you which each one of those lines of our test went in there, and this is what our y_predict came out. So let's move on to the next step; we're going to take this data and try to figure out just how good a model we have.

So here we go. Since sklearn does all the heavy lifting for you and all the math, we have a simple line of code to let us know what the accuracy is. And let's go ahead and go through that and see what that means and what that looks like. Let's go ahead and paste this in, and let me zoom in a little bit. There we go, so you have a nice full picture. And we'll see here we're just going to do a print("Accuracy is: ") and then we do the accuracy_score(), and this was something we imported um earlier; if you remember at the very beginning, let me just scroll up there real quick so you can see where that's coming from; that's coming from here, down here: from sklearn.metrics import accuracy_score. And you could probably run a script, make your own script to do this very easily; how accurate is it? How many out of 300 do we get right? And so we put in our y_test, that's the one we ran the predict() on, and then we put in our y_predict, that's the answers we got, and we're just going to multiply that by 100 because this is just going to give us an answer as a decimal, and we want to see it as a percentage. And let's run that and see what it looks like. And if you see here, we got an accuracy of 93.66667. So when we look at the number of loans and we look at how good our model fit, we can tell people it has about a 93.6 fitting to it. So just a quick recap on that: we now have accuracy set up on here, and so we have created a model that uses the decision tree algorithm to predict whether a customer will repay the loan or not. The accuracy of the model is about 94.6 percent. The bank can now use this model to decide whether it should approve the loan request from a particular customer or not. And so this information is really powerful; we may...

Not be able to, as individuals, understand all these numbers, because they have thousands of numbers that come in. But you can see that this is a smart decision for the bank to use a tool like this to help them predict how good their profit is going to be off of the loan balances and how many are going to default or not.

Today, we're going to be looking at random forest, one of the many powerful tools in the machine learning library. Before we dive into the topic, let's start by looking at a few of the uses for random forest. Currently, today, it's used in remote sensing, for example. They're used in the ETM devices—if you're a space buff, that's the Enhanced Thematic Mapper—they use on satellites, which see, uh, far outside the human spectrum for looking at land masses and they acquire images of the earth's surface. The accuracy is higher, and training time is less than many other machine learning tools out there. Also, object detection—multi-class object detection—is done using random forest algorithms. A good example is a traffic where you try to sort out the different cars, buses, and things, and it provides better detection in complicated environments. They're very complicated up there.

And then we have, uh, another example: Connect. Let's take a little closer look at Connect. Connect uses a random forest as part of the game console, and what it does is it tracks the body movements and it recreates it in the game. Let's see what that looks like. We have a user who performs a step—in this case, it looks like Elvis Presley—going there. That is then recorded, so the Connect registers the movement, and then it marks the user based on accuracy. It looks like we have, uh, prints going on this one from Elvis Presley—leader prints—that's great. So, Mark's user based on the accuracy. If we look at that a little closer, we have a training set to identify body parts: where are the hands, where are the feet, what's going on with the body. That then goes into a random forest classifier that learns from it. Once we trained the classifier, and then identifies the body parts while the person's dancing, it's able to represent that in a computer format. And then, based on that, it scores the game and how accurate you are as being Elvis Presley or Prince and you're dancing.

Let's take an overview of what we're going to cover today. What's in it for you? We're going to start with: what is machine learning? We're not going to go into detail on that. We're going to specifically look at how the random forest fits in the machine learning hierarchy. Then we're going to look at some applications of random forest; what is classification, which is its primary use; why use random forest; what's the benefits of it; and how does it actually come together; what is random forest; and then we'll get into random forests and the decision tree—how all that's like the final step—and how it works. And finally, we'll get some Python code in there, and we'll use the case: the iris flower analysis. Now, if you don't know what any of these terms mean or where we're going with this, don't worry. We're going to cover all the basics and have you up and running and even having doing some basic script in Python by the end.

Let's take a closer look at types of machine learning. Specifically, we're going to look at where the decision tree fits in with the different machine learning packages out there. We'll start with the basic types of machine learning. There's supervised learning, where you have lots of data and you're able to train your models. There's unsupervised learning, where it has to look at the data and then divide it based on its own algorithms without having any training. And then there's reinforcement learning, where you get a plus or negative if you have the answer correct. This particular tool belongs to the supervised learning. Let's take a closer look at that; what that means in supervised learning. Supervised learning falls into two groups: classification and regression. We'll talk about regression a little later and how that differs. This particular format goes underneath classification, so we're looking at supervised learning and classification in the machine learning tools. Classification is a kind of problem when the outputs are categorical in nature, like yes or no, true or false, or zero or one. In that particular framework, there's the k-NN, where the NN stands for nearest neighbor; Naïve Bayes; the decision tree, which is part of the random forest that we're studying today.

So why random forest? It's always important to understand why we use this tool over the other ones. What are the benefits here? And so with the random forest, the first one is: there's no overfitting. If you use multiple trees, reduce the risk of overfitting. Training time is less. Overfitting means that we have fit the data so close to what we have as our sample that we pick up on all the weird parts, and instead of predicting the overall data, you're predicting the weird stuff, which you don't want. High accuracy; runs efficiently in large database; for large data, it produces highly accurate predictions. In today's world of big data, this is really important, and this is probably where it really shines. This is where why random forest really comes in. It estimates missing data. Data in today's world is very messy, so when you have a random forest, it can maintain the accuracy when a large proportion of the data is missing. What that means is if you have data that comes in from five or six different areas, and maybe they took one set of statistics in one area and they took a slightly different set of statistics in the other, so they have some of the same, same shared data, but one is missing, like the number of children in the house if you're doing something over demographics, and the other one is missing the size of the house, it will look at both of those separately and build two different trees, and then it can do a very good job of guessing which one fits better, even though it's missing that data.

Let us dig deep into the theory of exactly how it works, and let's look at what is a random forest. Random forest, or random decision forest, is a method that operates by constructing multiple decision trees. The decision of the majority of the trees is chosen by the random forest as the final decision. And this, uh, we have some nice graphics here. We have a decision tree, and they actually use a real tree to denote the decision tree, which I love. And given a random, some kind of picture of a fruit, this decision tree decides that the output is: it's an apple. And we have a decision tree two, where we have that picture of the fruit goes in, and this one decides that it's a lemon. And the decision three tree gets another image, and it decides it's an apple. And then this all comes together in what they call the random forest, and this random forest then looks at it and says, okay, I got two votes for apple, one vote for lemon; the majority is apples, so the final decision is apples. To understand how the random forest works, we first need to dig a little deeper and take a look at the random forest and the actual decision tree and how it builds that decision tree.

In looking closer at how the individual decision trees work, we'll go ahead and continue to use the fruit example since we're talking about trees and forests. A decision tree is a tree-shaped diagram used to determine a course of action. Each branch of the tree represents a possible decision, occurrence, or reaction. So in here, we have a bowl of fruit, and if you look at that, it looks like they switched from lemons to oranges. We have oranges, cherries, and apples. And the first decision of the decision tree might be: is a diameter greater than or equal to three? And if it says false, it knows that they're cherries because everything else is bigger than that, so all the cherries fall into that decision. So we have all that data we're training; we can look at that; we know that that's what's going to come up. Is the color orange? Well, goes hmm, orange or red. Well, if it's true, then it comes out as the orange, and if it's false, that leaves apples. So in this example, it sorts out the fruit in the bowl, or the images of the fruit. A decision tree—these are very important terms to know because these are very central to understanding the decision tree when working with them—the first is entropy. Everything on the decision tree and how it makes this decision is based on entropy. Entropy is a measure of randomness or unpredictability in the data set. Then they also have information gain, the leaf node, the decision node, and the root node. We'll cover these other four terms as we go down the tree, but let's start with entropy.

So starting with entropy, we have here a high amount of randomness. What that means is that whatever is coming out of this decision, if it was going to guess based on this data, it wouldn't be able to tell you whether it's a lemon or an apple; it would just say it's a fruit. So the first thing we want to do is we want to split this apart, and we take the initial data set; we're going to create a data set 1 and a data set 2. We just split it in 2. And if you look at these new data sets after splitting them, the entropy of each of those sets is much less. So for the first one, whatever comes in there, it's going to sort that data, and it's going to say, okay, if this data goes this direction, it's probably an apple, and if it goes into the other direction, it's probably a lemon. So that brings us up to information gain. It is the measure of decrease in the entropy after the data set is split. What that means in here is that we've gone from one set which has a very high entropy to two lower sets of entropy, and we've added in the values of E1 for the first one and E2 for the second two, which are much lower. And so that information gain is increased greatly in this example, and so you can find that the information gain simply equals decision E1 minus E2.

As we're going down our list of, uh, definitions, we'll look at the leaf node, and the leaf node carries the classification or the decision. So we look down here to the leaf node; we finally get to our set one or our set two when it comes down there, and it says, okay, this object's gone into set one; if it's gone into set one, it's going to be split by some means, and we'll either end up with apples on the leaf node or a lemon on the leaf node. And on the right, it'll either be an apple or lemons. Those leaf nodes, or those final decisions or classifications, that's the definition of leaf node in here. If we're going to have a final leaf where we make the decision, we should have a name for the nodes above it, and they call those decision nodes. A decision node has two or more branches, and you can see here where we have the, uh, five apples and one lemon, and in the other case, the five lemons and one apple; they have to make a choice of which tree it goes down based on some kind of measurement or information given to the tree. And that brings us to our last definition: the root node. The topmost decision node is known as the root node, and this is where you have all of your data and you have your first decision it has to make or the first split in information.

So far, we've looked at a very general image with the fruit being split. Let's look and see exactly what that means to split the data, and how do we make those decisions on there. Let's go in there and find out how does a decision tree work. So let's try to understand this, and let's use a simple example, and we'll stay with the fruit. We have a bowl of fruit, and so let's create a problem statement, and the problem is we want to classify the different types of fruits in the bowl based on different features. The data set in the bowl is looking quite messy, and the entropy is high in this case. So if this ball was our decision maker, it wouldn't know what choice to make; it has so many choices—which one do you pick: apple, grapes, or lemons? And so we look in here; we're going to start with the—a training set, so this is our data that we're training our data with, and we have a number of options here: we have the color, and under the color we have red, yellow, purple; we have a diameter: 3, 3, 1, 3, 3, 1; and we have a label: apple, lemon, grapes, apple, lemon, grapes. And how do we split the data? We have to frame the conditions to split the data in such a way that the information gain is the highest. It's very key to note that we're looking for the best gain; we don't want to just start sorting out the smallest piece in there; we want to split it the biggest way we can. And so we measure this decrease in entropy—that's what they call it—entropy. There's our entropy after splitting, and now we'll try to choose a condition that gives us the highest gain. We will do that by splitting the data using each condition and checking the gain that we get out of them. The conditions that give us the highest gain will be used to make the first split. So let's take a look at these different conditions: we have color, we have diameter, and if we look underneath that, we have a couple different values: we have diameter equals three, color equals yellow, red, diameter equals one. And when we look at that, you'll see over here we have one, two, three, four threes—that's a pretty hardy selection. So let's say the condition gives us a maximum gain of three, so we have the most pieces fall into that range. So our first split from our decision node is: we split the data based on the diameter—is it greater than or equal to three? If it's not—that's false—it goes into the grape bowl, and if it's true, it goes into a bowl full of lemon and apples. The entropy after splitting has decreased considerably, so now we can make two decisions. If you look at, they're very much less chaos going on there. This node has already attained an entropy value of 0, as you can see there's only one kind of label left for this branch, so no further splitting is required for this node. However, this node on the right is still requires a split to decrease the entropy further, so we split the right node further based on color. If you look at this, if I split it on color, that pretty much cuts it right down the middle; it's the only thing we have left in our choices of color and diameter too. And if the color is yellow, it's going to go to the right bowl, and if it's false, it's going to go to the left bowl. So the entropy in this case is now zero, so now we have three bowls with zero entropy—there's only one type of data in each one of those bowls—so we can predict a lemon with a hundred percent accuracy, and we can predict the apple also with 100 accuracy along with our grapes up there. So we've looked at kind of a basic tree in our forest, but what we really want to know is how does a random forest work as a whole.

So to begin our random forest classifier, let's say we already have built three trees, and we're going to start with the first tree that looks like this, just like we did in the example. This tree looks at the diameter—if it's greater than or equal to three, it's true, otherwise it's false. So one side goes to the smaller diameter, one side goes to larger diameter. And if the color is orange, it's going to go to the right—true; we're using oranges now instead of lemons—and if it's red, it's going to go to the left—false. We build a second tree, very similar but split differently. Instead of the first one being split by a diameter, this one, when they created it, if you look at that first bowl, it has a lot of red objects, so it says: is the color red? Because that's going to bring our entropy down the fastest. And so, of course, if it's true, it goes to the left; if it's false, it goes to the right, and then it looks at the shape—false or true—and so on and so on. And tree three is: the diameter equal to one, and it came up with this because there's a lot of cherries in this bowl, so that would be the biggest split on there is: is the diameter equal to one? That's going to drop the entropy the quickest. And as you can see, it splits it into true if it goes false, and they've added another category: does it grow in the summer? And if it's false, it goes off to the left; if it's true, it goes off to the right. Let's go ahead and bring these three trees—you can see them all in one image—so this would be three completely different trees categorizing a fruit. And let's take a fruit now. Let's try this, and this fruit, if you look at it, we've blackened it out; you can't see the color on it, so it's missing data. Remember one of the things we talked about earlier is that a random forest works really good if you're missing data, if you're missing pieces. So this fruit has an image, but maybe the person had a black and white camera when they took the picture, and we're going to take a look at this, and it's going to have—they put the color in there—so ignore the color down there, but the diameter equals 3; we find out it grows in the summer equals yes, and the shape is a circle. And if you go to the right, you can look at what one of the decision trees did. This is the third one: is the diameter greater than equal to three? Is the color orange? Well, it doesn't really know on this one, but if you look at the value, it's a true, and it goes to the right. Tree two classifies it as cherries: is the color equal red? Is the shape of circle true? It is a circle, so this would look at it and say, oh, that's a cherry. And then we go to the other classifier, and it says: is the diameter equal one? Well, that's false; does it grow in the summer? True. So it goes down and looks at as oranges. So how does this random forest work? The first one says it's an orange; the second one said it was a cherry; and the third one says it's an orange. And you can guess: if you have two oranges and one says it's a cherry, when you add that all together, the majority of the vote says orange, so the answer is: it's classified as an orange, even though we didn't know the color and we're missing data on it.

I don't know about you, but I'm getting tired of fruit, so let's switch. And I did promise you we'd start looking at a case example and get into some Python coding today. We're going to use the case: the iris flower analysis. This is the exciting part as we roll up our sleeves and actually look at some Python coding. Before we start the Python coding, we need to go ahead and create a problem statement: wonder what species of iris do these flowers belong to? Let's try to predict the species of the flowers using machine learning in Python. Let's see how it can be done.

So here we begin to go ahead and implement our Python code, and you'll find that the first half of our implementation is all about organizing and exploring the data coming in. Let's go ahead and take this first step, which is loading the different modules into Python, and let's go ahead and put that in our favorite editor—whatever your favorite editor is. In this case, I'm going to be using the Anaconda Jupyter Notebook, which is one of my favorites. Certainly, there's Notepad++, and Eclipse, and dozens of others, or just even using the Python terminal window; any of those will work just fine to go ahead and explore this Python coding. So here we go. Let's go ahead and flip over to our Jupyter Notebook, and I've already opened up a new page for Python 3 code, and I'm just going to paste this right in there, and let's take a look and see what we're bringing into our Python. The first thing we're going to do is: from the sklearn.datasets import load_iris. Now, this isn't the actual data; this is just the module that allows us to bring in the data—the load_iris—and the iris is so popular; it's been around since 1936 when Ronald Fisher published a paper on it, and they're measuring the different parts of the flower, and based on those measurements

Predicting what kind of flower it is, and then if we're going to do a random forest classifier, we need to go ahead and import a random forest classifier from the sklearn module. So, sklearn.ensemble import RandomForestClassifier. And then we want to bring in two more modules, and these are probably the most commonly used modules in Python and data science, with any of the other modules that we bring in. One is going to be pandas; we're going to import pandas as pd. pd is a common term used for pandas, and pandas is basically creates a data format for us where, when you create a pandas data frame, it looks like an excel spreadsheet, and you'll see that in a minute when we start digging deeper into the code. Panda is just wonderful because it plays nice with all the other modules in there. And then we have numpy, which is our numbers Python, and the numbers Python allows us to do different mathematical sets on here. We'll see right off the bat we're going to take our np, and we're going to go ahead and seed the randomness with it was zero, so np.random.seed is seating that is zero. This code doesn't actually show anything; we're gonna go ahead and run it because I need to make sure I have all those loaded. And then let's take a look at the next module on here.

The next six slides, including this one, are all about exploring the data. Remember, I told you half of this is about looking at the data and getting it all set. So let's go ahead and take this code right here, the script, and let's get that over into our Jupyter notebook, and here we go. We've gone ahead and run the imports, and I'm going to paste the code down here, and let's take a look and see what's going on. The first thing we're doing is we're actually loading the iris data, and if you remember up here we loaded the module that tells it how to get the iris data. Now we're actually assigning that data to the variable iris, and then we're going to go ahead and use the df to define data frame, and that's going to equal pd—and if remember that's pandas as pd—so that's our pandas and panda data frame, and then we're looking at iris.data and columns equals iris.feature_names, and we're going to do the df.head. Let's run this. You can understand what's going on here. The first thing you want to notice is that our df has created what looks like an excel spreadsheet, and in this excel spreadsheet we have set the columns, so up on the top you can see the four different columns, and then we have the data iris.data down below. It's a little confusing without knowing where this data is coming from, so let's look at the bigger picture, and I'm going to go print—I'm just going to change this for a moment—and we're going to print all of iris and see what that looks like.

So, when I print all of iris, I get this long list of information, and you can scroll through here and see all the different titles on there. What's important to notice is that first off there's a bracket at the beginning, so this is a Python dictionary, and in a Python dictionary you'll have a key or a label, and this label pulls up whatever information comes after it. So, feature_names, which we actually used over here under columns, is equal to an array of sepal length, sepal width, petal length, petal width; these are the different names they have for the four different columns. And if you scroll down far enough, you'll also see data down here—oh goodness, it came up right towards the top—and data is equal to the different data we're looking at. Now there's a lot of other things in here like target; we're going to be pulling that up in a minute, and there's also the names, the target_names, which is further down, and we'll show you that also in a minute. Let's go ahead and set that back to the head. This is one of the neat features of pandas and panda data frames: when you do df.head or the pandadataframe.head, it'll print the first five lines of the dataset in there, along with the headers if you have it. In this case, we have the column header set to iris.feature_names, and in here you'll see that we have one, two, three, four. In Python, most arrays always start at zero, so when you look at the first five, it's going to be zero, one, two, three, four, not one, two, three, four, five. So now we've got our iris data imported into a data frame. Let's take a look at the next piece of code in here.

So, in this section here of the code, we're going to take a look at the target, and let's go ahead and get this into our notebook, this piece of code, so we can discuss it a little bit more in detail. So here we are in our Jupyter notebook; I'm going to put the code in here, and before I run it, I want to look at a couple things going on. So we have df['species'], and this is interesting because right here you'll see where I have df['species'] in brackets, which is uh the key code for creating another column, and here we have iris.target. Now these are both in the pandas setup on here, so in pandas we can do either one; I could have just as easily done iris['target'], depending on what I'm working on; both are acceptable. Let's go ahead and run this code and see how this changes. And what we've done is we've added the target from the iris data set as another column on the end. Now, what species is this? This is what we're trying to predict. So we have our data which tells us the answer for all these different pieces, and then we've added a column with the answer. That way, when we do our final setup, we'll have the ability to program our neural network to look for these—this different data—and know what a setosa is, or a versicolor, which we'll see in just a minute, or virginica. Those are the three that are in there. And now we're going to add one more column. I know we're organizing all this data over and over again; it's kind of fun. There's a lot of ways to organize it. What's nice about putting everything onto one data frame is I can then do a printout, and it shows me exactly what I'm looking at, and I'll show you that where you—where that's different, where you can alter that and do it slightly differently, but let's go ahead and put this into our script up to that now.

And here we go; we're going to put that down here, and we're going to run that, and let's talk a little bit about what we're doing. Now we're exploring data, and one of the challenges is knowing how good your model is; did your model work? And to do this, we need to split the data, and we split it into two different parts; they usually call it the training and the testing. And so in here we're going to go ahead and put that in our database so you can see it clearly, and we've set it df—remember you can put brackets; this is creating another column—is_train, so we're going to use part of it for training, and this equals np—remember that stands for numpy—dot random dot uniform, so we're generating a random number between 0 and 1, and we're going to do it for each of the rows; that's where the len(df) comes from. So each row gets a generated number, and if it's less than 0.75, it's true, and if it's greater than 0.75, it's false. This means we're going to take 75 percent of the data, roughly, because there's a randomness involved, and we're going to use that to train it, and then the other 25 we're going to hold off to the side and use that to test it later on. So let's flip back on over and see what the next step is.

So now that we've labeled our database for which is training and which is testing, let's go ahead and sort that into two different variables, train and test. And let's take this code and let's bring it into our project, and here we go. Let's paste it on down here, and before I run this, let's just take a quick look at what's going on here. Is—we have up above we created—remember there's our df.head, which prints the first five rows, and we've added a column is_train at the end. And so we're going to take that; we're going to create two variables; we're going to create two new data frames; one's called train, one's called test; 75 percent in train, 25 percent in test. And then to sort that out, we're going to do that by doing df—our main original data frame with the iris data in it—and if df['is_train'] equals true, that's going to go in the train, and if df['is_train'] equals false, it goes in the test. And so when I run this, we're going to print out the number in each one. Let's see what that looks like, and you'll see that it puts 118 in the training module, and it puts 32 in the testing module, which lets us know that there was 150 lines of data in here. So if you went and looked at the original data, you could see that there's 150 lines, and that's roughly 75 and 25 for us to test our model on afterward. So let's jump back to our code and see where this goes in the next two steps.

We want to do one more thing with our data, and that's make it readable to humans. I don't know about you, but I hate looking at zeros and ones. So let's start with the features, and let's go ahead and take those and make those readable to humans, and let's put that in our code. Let's see here; we go, paste it in, and you'll see here we've done a couple very basic things. We know that the columns in our data frame—again, this is a panda thing—the df.columns, and we know the first four of them—zero, one, two, three—that'd be the first four are going to be the features or the titles of those columns. And so when I run this, you'll see down here that it creates an index: sepal length, sepal width, petal length, and petal width. And this should be familiar because if you look up here, here's our column titles going across, and here's the first four. One thing I want you to notice here is that when you're in a command line, whether it's Jupyter notebook or you're running command line in the terminal window, if you just put the name of it, it'll print it out. This is the same as doing print(features), and the shorthand is you just put features in here. If you're actually writing a code and saving the script and running it by remote, you really need to put the print in there, but for this, when I run it, you'll see it gives me the same thing. But for this, we want to go ahead and we'll just leave it as features because it doesn't really matter. And this is one of the fun things about Jupyter notebooks is I'm just building the code as we go. And then we need to go ahead and create the labels for the other part. So let's take a look and see what that—for our final step in prepping our data before we actually start running the training and the testing—is we're going to go ahead and convert the species on here into something the computer understands. So let's put this code into our script and see where that takes us. All right, here we go. We've set y = pd.factorize(train['species'].iloc[0])[0]. Let's break this down just a little bit. We have our pandas right here, pd.factorize. What's factorize doing? I'm going to come back to that in just a second. Let's look at what train['species'] is and why we're looking at the group zero on there. And let's go up here, and here is our species—remember this—on that we created this whole column here for species, and then it has setosa, setosa, setosa, setosa, and if you scroll down enough you'd also see virginica and versicolor. We need to convert that into something the computer understands: zeros and ones. So the train['species'].iloc[0] because this is in the format of a—of an array of arrays, so you have to have the zero on the end, and then species is just that column. Factorize goes in there, looks at the fact that there's only three of them, so when I run this, you'll see that y generates an array that's equal to—in this case it's the training set—and it's zeros, ones, and twos representing the three different kinds of flowers we have. So now we have something the computer understands, and we have a nice table that we can read and understand. And now finally we get to actually start doing the predicting.

So here we go; we have two lines of code. Oh my goodness, that was a lot of work to get to two lines of code, but there is a lot in these two lines of code. So let's take a look and see what's going on here and put this into our full script that we're running. And let's paste this in here, and let's take a look and see what this is. We have—we're creating a variable clf, and we're going to set this equal to the RandomForestClassifier, and we're passing two variables in here, and there's a lot of variables you can play with as far as these two are concerned; they're very standard. n_jobs just does is to prioritize it; not something to really worry about. Usually when you're doing this on your own computer, you do n_jobs=2. If you're working in a larger or big data and you need to prioritize it differently, this is what that number does; is it changes your priorities and how it's going to run across the system and things like that. And then the random_state is just how it starts; zero is fine for here, but uh let's go ahead and run this. We also have clf.fit(train_features, y), and before we run it, let's talk about this a little bit more. clf.fit: so we're fitting; we're training it; we are actually creating our random forest classifier right here. This is a code that does everything, and we're going to take our training set—remember we kept our test off to the side—and we're going to take our training set with the features, and then we're going to go ahead and put that in, and here's our target, the y. So the y is 0, 1, and 2 that we just created, and the features is the actual data going in that we put into the training set. Let's go ahead and run that. And this is kind of an interesting thing because it printed out the RandomForestClassifier and everything around it. And so when you're running this in your terminal window or in a script like this, this automatically treats this like—just like when we were up here and I typed in y and I printed out y instead of print(y)—this does the same thing; it treats this as a variable and prints it out. But if you're actually running your code, that wouldn't be the case, and what is printed out is it shows us all the different variables we can change. And if we go down here, you can actually see n_jobs=2; you can see the random_state=0; those are the two that we sent in there. You would really have to dig deep to find out all these different meanings of all these different settings on here; some of them are self-explanatory if you kind of think about it a little bit, like max_features='auto', so all the features that we're putting in there is just going to automatically take all four of them. Whatever we send it, it'll take some of them. Might have so many features because you're processing words; there might be like 1.4 million features in there because you're doing legal documents, and that's how many different words are in there. At that point, you probably want to limit the maximum features that you're going to process, and leaf nodes—that's the end nodes; remember we had the fruit, and we're talking about the leaf nodes. Like I said, there's a lot in this; we're looking at a lot of stuff here, so you might have—in this case there's probably only think three leaf nodes, maybe four—you might have thousands of leaf nodes, at which point you do need to put a cap on that and say, okay, can only go so far. And then we're going to use all of our resources on processing this, and that really is what most of these are about: is limiting the process and making sure we don't overwhelm a system. And there's some other settings in here; again, we're not going to go over all of them. warm_start=False; warm_start is if you're programming it one piece at a time externally. Since we're not, we're not going to have—like we're not going to continually train this particular learning tree. And again, like I said, there's a lot of things in here that you'll want to look up more detail from the sklearn, and if you're digging in deep and running a major project on here. For today, though, all we need to do is fit our train our features and our target y. So now we have our training model; what's next? If we're going to create a model, we now need to test it. Remember, we set aside the test feature, test group, 25 percent of the data. So let's go ahead and take this code and let's put it into our script and see what that looks like.

Okay, here we go, and we're going to run this, and it's going to come out with a bunch of zeros, ones, and twos, which represents the three types of flowers: the setosa, the virginica, and the versicolor. And what we're putting into our predict is the test_features, and I always kind of like to know what it is I am looking at. So real quick, we're gonna do test_features, and remember features is an array of sepal length, sepal width, petal length, petal width. So when we put it in this way, it actually loads all these different columns that we loaded into features. So if we did just features—let me just do features in here so you can see what features looks like—this is just playing with the—with pandas data frames—you'll see that it's an index. So when you put an index in like this into test_features into test, it then takes those columns and creates a pandas data frame from those columns, and in this case we're going to go ahead and put those into our predict. So we're going to put each one of these lines of data—the 5.0, 3.4, 1.5, .2—and we're going to put those in, and we're going to predict what our new forest classifier is going to come up with. And this is what it predicts: it predicts 0, 0, 0, 1, 2, 1, 1, 2, 2, 2, and—and again this is the flower type: setosa, virginica, and versicolor. So now that we've taken our test features, let's explore that; let's see exactly what that data means to us.

So the first thing we can do with our predicts is we can actually generate a different prediction model. When I say different, we're going to view it differently; it's not that the data itself is different. So let's take this next piece of code and put it into our script. So we're pasting it in here, and you'll see that we're doing predict, and we've added _proba for probability. So there's our clf.predict_proba; so we're running it just like we ran it up here, but this time with this we're going to get a slightly different result, and we're only going to look at the first 10. So you'll see down here, instead of looking at all of them—which was uh what, 27—you'll see right down here that this generates a much larger field on the probability, and let's take a look and see what that looks like and what that means. So when we do the predict_proba for probability, it generates three numbers. So we had three leaf nodes at the end, and if you remember from all the theory we did, this is the predictors; the first one is predicting a 1 for setosa; it predicts a 0 for virginica, and it predicts a 0 for versicolor, and so on and so on and so on. And let's—you know what, I'm going to change this just a little bit; let's look at 10 to 20 just because we can, and we start to get a little different of data, and you'll see right down here it gets to this one, this line right here, and this line has 0, 0.5, 0.5, 0.5. And so if we're going to vote and we have two equal votes, it's going to go with

The first one, so it says setosa gets zero votes, virginica gets point five votes versus color gets point five votes. But let's just go with the virginica since these two are equal, and so on and so on down the list. You can see how they vary on here.

So now we've looked at both how to do a basic predict of the features, and we've looked at the predict probability. Let's see what's next on here. So now we want to go ahead and start mapping names for the plants. We want to attach names so that it makes a little more sense for us, and this we're going to do in these next two steps. We're going to start by setting up our predictions and mapping them to the name. So let's see what that looks like, and let's go ahead and paste that code in here and run it.

This goes along with the next piece of code, so we'll skip through this quickly and then come back to a little bit. So here's iris.target_names, and uh, if you remember correctly, this was the the names that we've been talking about this whole time: the setosa, virginica, versus color. And then we're gonna go ahead and do the prediction again. We've run that; we could have just set a variable equal to this instead of re-running it each time, but we'll go ahead and run it again: clf.predict(test_features). Remember that returns the zeros, the ones, and the twos. And then we're going to set that equal to predictions. So this time we're actually putting it in a variable, and when I run this, it distributes it; it comes out as an array, and the array is setosa, setosa, setosa, setosa, setosa. We're only looking at the first five; we could actually do—let's do the first 25, just so we can see a little bit more on there—and you'll see that it starts mapping it to all the different flower types, the versa color and the virginica in there.

Let's see how this goes with the next one. So let's take a look at the top part of our species in here, and we'll take this code and put it in our script. Let's put that down here and paste it. There we go, and we'll go ahead and run it. Let's talk about both these sections of code here and how they go together. The first one is our predictions, and I went ahead and did predictions through 25. Let's just do five, and so we have setosa, setosa, setosa, setosa; that's what we're predicting from our test model. And then we come down here; we look at test species. I remember I could have just done test.species.head, and you'll see it says setosa, setosa, setosa, setosa, and they match. So the first one is what our forest is doing, and the second one is what the actual data is. Now, is we need to combine these so that we can understand what that means. We need to know how good our forest is, how good it is at predicting the features. So that's where we come up to the next step, which is lots of fun. We're going to use a single line of code to combine our predictions and our actuals so we have a nice chart to look at. And let's go ahead and put that in our script, in our Jupyter notebook here. Let's see; let's go ahead and paste that in, and then I'm going to—because I'm on the Jupyter notebook—I can do a control minus, so you can see the whole line there. There we go, resize it, and let's take a look and see what's going on here. We're going to create, in pandas—remember pd stands for pandas—and we're doing a cross-tab. This function takes two sets of data and creates a chart out of them. So when I run it, you'll get a nice chart down here, and we have the predicted species. So across the top you'll see the setosa, versus color, virginica, and the actual species: setosa, versacolor, virginica. And so the way to read this chart—and let's go ahead and take a look on how to read this chart here—when you read this chart, you have setosa where they meet, you have versus color where they meet, and you have virginica where they meet, and they're meeting where the actual and the predicted agree. So this is the number of accurate predictions. So in this case, it equals 30. If you had 13 plus 5 plus 12, you get 30. And then we notice here where it says virginica but it was supposed to be versacolor; this is inaccurate. So now we have two, two inaccurate predictions and 30 accurate predictions. So we'll say that the model accuracy is 93%; that's just 30 divided by 32, and if we multiply it by a hundred, we can say that it is 93 percent accurate. So we have a 93% accuracy with our model.

I did want to add one more quick thing in here on our scripting before we wrap it up. So let's flip back on over to my script in here. We're going to take this line of code from up above; I don't know if you remember it, but preds equals the iris.target_names. So we're going to map it to the names, and we're going to run the prediction, and we read it on test features, but you know we're not just testing it; we want to actually deploy it. So at this point, I would go ahead and change this, and this is an array of arrays; this is really important when you're running these to know that, so you need the double brackets. And I could actually create data; maybe let's just do two flowers, so maybe I'm processing more data coming in, and we'll put two flowers in here, and then I actually want to see what the answer is. So let's go ahead and type in preds and print that out, and when I run this, you'll see that I've now predicted two flowers that maybe I measured in my front yard as versacolor and versacolor. Not surprising since I put the same data in for each one. This would be the actual end product going out to be used on data that you don't know the answer for.

So that's going to conclude our scripting part of this today. We're going to cover the k-nearest neighbors, like referred to as k-NN. K-NN is really a fundamental place to start in the machine learning; it's the basis of a lot of other things, and just the logic behind it is easy to understand and incorporated in other forms of machine learning. So today, what's in it for you? Why do we need k-NN? What is k-NN? How do we choose the factor k? When do we use k-NN? How does a k-NN algorithm work? And then we'll dive in to my favorite part: the use case—predict whether a person will have diabetes or not. That is a very common and popular used data set as far as testing out models and learning how to use the different models in machine learning.

By now we all know machine learning models make predictions by learning from the past data available. So we have our input values; our machine learning model builds on those inputs of what we already know, and then we use that to create a predicted output. Is that a dog? Little kid looking over there watching the black cat cross their path? No, dear, you can differentiate between a cat and a dog based on their characteristics. Cats—cats have sharp claws, uses to climb, smaller length of ears, meows and purrs, doesn't love to play around. Dogs—they have dull claws, bigger length of ears, barks, loves to run around. You usually don't see a cat running around people, although I do have a cat that does that, where dogs do. And we can look at these; we can say we can evaluate the sharpness of the claws—how sharp are their claws—and we can evaluate the length of the ears, and we can usually sort out cats from dogs based on even those two characteristics. Now, tell me if it is a cat or a dog—not a question; usually little kids know cats and dogs by now, unless they live a place where there's not many cats or dogs. So if we look at the sharpness of the claws, the length of the ears, and we can see that the cat has a smaller ears and sharper claws than the other animals; its features are more like cats; it must be a cat. Sharp claws, length of ears, and it goes in the cat group. Because k-NN is based on feature similarity, we can do classification using a k-NN classifier. So we have our input value—the picture of the black cat—it goes into our trained model, and it predicts that this is a cat coming out.

So what is k-NN? What is the k-NN algorithm? K-nearest neighbors is what that stands for, is one of the simplest supervised machine learning algorithms, mostly used for classification. So we want to know: is this a dog or is not a dog? Is it a cat or not a cat? It classifies a data point based on how its neighbors are classified. K-NN stores all available cases and classifies new cases based on a similarity measure. And here we gone from cats and dogs right into wine, another favorite of mine. K-NN stores all available cases and classifies new cases based on a similarity measure. And here you see we have a measurement of sulfur dioxide versus the chloride level, and then the different wines they've tested and where they fall on that graph based on how much sulfur dioxide and how much chloride. K, and k-NN is a perimeter that refers to the number of nearest neighbors to include in the majority of the voting process. And so if we add a new glass of wine there—red or white—we want to know what the neighbors are. In this case, we're going to put k equals 5. We'll talk about k in just a minute. A data point is classified by the majority of votes from its five nearest neighbors. Here the unknown point would be classified as red since four out of five neighbors are red.

So how do we choose k? How do we know k equals five? I mean, that's what's the value we put in there? So we're going to talk about it. How do we choose a factor k? K-NN algorithm is based on feature similarity; choosing the right value of k is a process called parameter tuning and is important for better accuracy. So at k equals three, we can classify—we have a question mark in the middle—as either a as a square or not; is it a square or is it, in this case, a triangle? And so if we set k equals to 3, we're going to look at the three nearest neighbors; we're going to say this is a square. And if we put k equals a 7, we classify as a triangle, depending on what the other data is around. And you can see as the k changes, depending on where that point is, that drastically changes your answer. And we jump—here we go—how do we choose the factor of k? You'll find this in all machine learning—choosing these factors; that's the face you get; he's like, "Oh my gosh, did I choose the right k? Did I set it right?" My values in whatever machine learning tool you're looking at so that you don't have a huge bias in one direction or the other. And in terms of k-NN, the number of k, if you choose it too low, the bias is based on—it's just too noisy; it's right next to a couple things, and it's going to pick those things, and you might get a skewed answer. And if your k is too big, then it's going to take forever to process, so you're going to run into processing issues and resource issues. So what we do—the most common use, and there's other options for choosing k—is to use the square root of n, so it is a total number of values you have; you take the square root of it. In most cases, you also—if it's an even number—so if you're using uh, like this case, squares and triangles, if it's even, you want to make your k value odd; that helps it select better. So in other words, you're not going to have a balance between two different factors that are equal. So usually take the square root of n, and if it's even, you add one to it or subtract one from it, and that's where you get the k value from; that is the most common use, and it's pretty solid; it works very well.

When do we use k-NN? We can use k-NN when data is labeled, so you need a label on it; we know we have a group of pictures with dogs, dogs, cats, cats. Data is noise free, and so you can see here when we have a class and we have like underweight, 140, 23, hello kitty, normal; that's pretty confusing; we have a high variety of data coming in, so it's very noisy, and that would cause an issue. Data set is small, so we're usually working with smaller data sets where you might get into a gig of data if it's really clean, doesn't have a lot of noise, because k-NN is a lazy learner, i.e., it doesn't learn a discriminative function from the training set, so it's very lazy. So if you have very complicated data and you have a large amount of it, you're not going to use the k-NN, but it's really great to get a place to start, even with large data; you can sort out a small sample and get an idea of what that looks like using the k-NN, and also just using for smaller data sets; k-NN works really good.

How does a k-NN algorithm work? Consider a data set having two variables: height in centimeters and weight in kilograms, and each point is classified as normal or underweight. So we see right here we have two variables; you know, true, false; they're either normal or they're not; they're underweight. On the basis of the given data, we have to classify the below set as normal or underweight using k-NN. So if we have new data coming in that says 57 kilograms and 177 centimeters, is that going to be normal or underweight? To find the nearest neighbors, we'll calculate the Euclidean distance. According to the Euclidean distance formula, the distance between two points in the plane with the coordinates x, y and a, b is given by distance d equals the square root of x minus a squared plus y minus b squared. And you can remember that from the two edges of a triangle; we're computing the third edge since we know the x side and the y side. Let's calculate it to understand clearly. So we have our unknown point, and we placed it there in red, and we have our other points where the data is scattered around. The distance d1 is the square root of 170 minus 167 squared plus 57 minus 51 squared, which is about 6.7. And distance 2 is about 13. And distance 3 is about 13.4. Similarly, we will calculate the Euclidean distance of the unknown data point from all the points in the data set, and because we're dealing with a small amount of data, that's not that hard to do; it's actually pretty quick for a computer, and it's not a really complicated mass; you can just see how close is the data based on the Euclidean distance. Hence, we have calculated the Euclidean distance of the unknown data point from all the points as shown, where x1 and y1 equal 57 and 170, whose class we have to classify. So now we're looking at that; we're saying, well, here's the Euclidean distance; who's going to be their closest neighbors? Now let's calculate the nearest neighbor at k equals three, and we can see the three closest neighbors puts them at normal, and that's pretty self-evident when you look at this graph; it's pretty easy to say, okay, what we're just voting normal, normal, normal; three votes for normal; this is going to be a normal weight. So the majority of neighbors are pointing towards normal; hence, as per the k-NN algorithm, the class of 57, 170 should be normal.

So a recap of k-NN: a positive integer k is specified along with a new sample; we select the k entries in our database which are closest to the new sample; we find the most common classification of these entries; this is the classification we give to the new sample. So as you can see, it's pretty straightforward; we're just looking for the closest things that match what we got. So let's take a look and see what that looks like in a use case in Python.

So let's dive into the predict diabetes use case. So use case: predict diabetes. The objective: predict whether a person will be diagnosed with diabetes or not. We have a data set of 768 people who were or were not diagnosed with diabetes. And let's go ahead and open that file and just take a look at that data. And this is in a simple spreadsheet format; the data itself is comma separated—very common set of data—and it's also a very common way to get the data. And you can see here we have columns A through I—that's what, one, two, three, four, five, six, seven, eight—um, eight columns with a particular attribute, and then the ninth column, which is the outcome, is whether they have diabetes. As a data scientist, the first thing you should be looking at is insulin—well, you know, if someone has insulin, they have diabetes because that's why they're taking it—and that could cause issue in some of the machine learning packages, but for a very basic setup, this works fine for doing the k-NN. And the next thing you notice is it didn't take very much to open it up; I can scroll down to the bottom of the data; there's 768. Pretty much a small data set, you know, at 769; I can easily fit this into my RAM on my computer; I can look at it; I can manipulate it, and it's not going to really tax just a regular desktop computer; you don't even need an enterprise version to run a lot of this.

So let's start with importing all the tools we need, and before that, of course, we need to discuss what IDE I'm using. Certainly can use any particular editor for Python, but I like to use, for doing a very basic visual stuff, the Anaconda, which is great for doing demos with the Jupyter notebook. And just a quick view of the Anaconda Navigator, which is the new release out there, which is really nice. You can see under home, I can choose my application; we're going to be using Python 3.6; I have a couple different uh, versions on this particular machine. If I go under environments, I can create a unique environment for each one, which is nice, and there's even a little button there where I can install different packages. So if I click on that button and open the terminal, I can use a simple pip install to install different packages I'm working with. Let's go ahead and go back under home, and we're going to launch our notebook, and I've already—you know, kind of like the old cooking shows—I've already prepared a lot of my stuff, so we don't have to wait for it to launch because it takes a few minutes for it to open up a browser window. In this case, I'm going to—it's going to open up Chrome because that's my default that I use, and since the script is pre-done, you'll see I have a number of windows open up at the top, the one we're working in. And since we're working on the k-NN—predict whether a person will have diabetes or not—let's go and put that title in there. And I'm also going to go up here and click on cell—actually, we want to go ahead and first insert a cell below—and then I'm going to go back up to the top cell, and I'm going to change the cell type to markdown. That means this is not going to run as Python; it's a markdown language. So if I run this first one, it comes up in nice big letters, which is kind of nice, remind us what we're working on. And by now you should be familiar with doing all of our imports; we're going to import the pandas as pd, import numpy as np. Pandas is the pandas data frame, and numpy is a number array—very powerful tools to use in here. So we have our imports; so we've brought in our pandas, our numpy, our two general Python tools. And then you can see over here we have our train_test_split; by now you should be familiar with splitting the data; we want to split part of it for training our thing and then training our particular model, and then we want to go ahead and test the remaining data to see how good it is. Pre-processing: a standard scalar preprocessor, so we don't have a bias of really large numbers. Remember, in the data we had like number of pregnancies isn't going to get very large, where the amount of insulin they take can get up to 256, so 256 versus 6; that will skew results, so we want to go ahead and change that so that they're all uniform between

-1 and 1. And then the actual tool; this is the k-neighbors classifier we're going to use. And finally, the last three are three tools to test all about testing our model: how good is it? We just put down "test" on there, and we have our confusion matrix, our F1 score, and our accuracy. So we have our two general Python modules we're importing, and then we have our six modules specific from the scikit-learn setup. And then we do need to go ahead and run this; so these are actually imported. There we go, and then move on to the next step.

And so in this set, we're going to go ahead and load the database. We're going to use pandas; remember pandas is pd, and we'll take a look at the data in Python. We looked at it in a simple spreadsheet, but usually I like to also pull it up so we can see what we're doing. So here's our dataset = pd.read_csv; that's a pandas command, and the diabetes folder I just put in the same folder where my iPython script is. If you put it in a different folder, you need the full length on there. We can also do a quick length of the dataset; that is a simple Python command, len for length. We might even—let's go ahead and print that. We'll go print, and if you do it on its own line like that dataset in the Jupyter Notebook, it'll automatically print it, but when you're in most of your different setups, you want to do the print in front of there. And then we want to take a look at the actual dataset, and since we're in pandas, we can simply do dataset.head. And again, let's go ahead and add the print in there. If you put a bunch of these in a row, you know, the dataset one head, dataset two head, it only prints out the last one, so I usually always like to keep the print statement in there. But because most projects only use one data frame, pandas data frame, doing it this way doesn't really matter; the other way works just fine. And you can see when we hit the run button, we have the 768 lines, which we knew, and we have our pregnancies; it's automatically given a label on the left. Remember the head only shows the first five lines, so we have zero through four, and just a quick look at the data; you can see it matches what we looked at before. We have pregnancy, glucose, blood pressure, all the way to age, and then the outcome on the end.

And we're going to do a couple of things in this next step. We're going to create a list of columns where we can't have zero; there's no such thing as zero skin thickness or zero blood pressure, zero glucose—any of those, you'd be dead. So not a really good factor if they don't—if they have a zero in there because they didn't have the data. And we'll take a look at that because we're going to start replacing that information with a couple of different things. Let's see what that looks like. So first, we create a nice list, as you can see, we have the values talked about: glucose, blood pressure, skin thickness; and this is a nice way when you're working with columns is to list the columns you need to do some kind of transformation on, a very common thing to do. And then for this particular setup, we certainly could use the—there's some pandas tools that will do a lot of this where we can replace the NA, but we're going to go ahead and do it as a dataset[column] = dataset[column].replace; this is—this is still pandas; you can do a direct—there's also one that's—that you look for your NaN; a lot of different options in here. But the NaN, NumPy NaN is what that stands for, is none, doesn't exist. So the first thing we're doing here is we're replacing the zero with a NumPy none; there's no data there; that's what that says; that's what this is saying right here. So put the zero in, and we're going to place zeros with no data. So if it's a zero, that means the person's—well, hopefully not dead; hope they just didn't get the data. The next thing we want to do is we're going to create the mean, which is the integer from the dataset from the column.mean where we skip NAs; we can do that; that is a pandas command there, the skipna. So we're going to figure out the mean of that dataset, and then we're going to take that dataset[column], and we're going to replace all the np.nan with the means. Why did we do that? We could have actually just taken this step and gone right down here and just replaced zero and skip anything where—except you could actually—there's a way to skip zeros and then just replace all the zeros. But in this case, we want to go ahead and do it this way so you could see that we're switching this to a non-existent value; then we're going to create the mean. Well, this is the average person, so if we don't know what it is, if they did not get the data, and the data is missing, one of the tricks is you replace it with the average; what is the most common data for that? This way you can still use the rest of those values to do your computation, and it kind of just brings that particular value of those missing values out of the equation. Let's go ahead and take this, and we'll go ahead and run it; doesn't actually do anything, so we're still preparing our data. If you want to see what that looks like, we don't have anything in the first few lines; just not going to show up. But we certainly could look at a row; let's do that. Let's go into our dataset with print(dataset), and let's pick—in this case, let's just do glucose. And if I run this, this is going to print all the different glucose levels going down, and we thankfully don't see anything in here that looks like missing data, at least on the ones it shows you. You can see you skipped a bunch in the middle because that's what it does if you have too many lines in Jupyter Notebook; it'll skip a few and go on to the next in a dataset. Let me go and remove this, and we'll just zero out that.

And of course, before we do any processing, before proceeding any further, we need to split the dataset into our train and testing data, that way we have something to train it with and something to test it on. And you're going to notice we did a little something here with the pandas database code. There we go, my drawing tool; we've added in this right here of the dataset, and what this says is that the first one in pandas—this is from the pd pandas—it's going to say within the dataset we want to look at the i location, and it is all rows; that's what that says. So we're going to keep all the rows, but we're only looking at column 0 to 8. Remember column nine here, it is right up here; we put it in here is outcome; well, that's not part of the training data; that's part of the answer. Yes, column nine, but it's listed as eight, number eight. So zero to eight is nine columns, so uh, eight is the value, and when you see it in here, 0—this is actually 0 to 7; it doesn't include the last one. And then we go down here to y, which is our answer, and we want just the last one, just column 8, and you can do it this way with this particular notation. And then if you remember, we imported the train_test_split; that's part of the sklearn right there, and we simply put in our x and our y. We're going to do random_state = 0; you don't have to necessarily seed it; that's a seed number. I think the default is one when you seed it; I have to look that up. And then the test_size, test_size is 0.2; that simply means we're going to take 20% of the data and put it aside so that we can test it later; that's all that is. And again, we're going to run it; not very exciting so far; we haven't had any printout other than to look at the data, but that is a lot of—this is prepping this data. Once you prep it, the actual lines of code are quick and easy, and we're almost there with the actual writing of our knn.

We need to go ahead and do a scale—the data. If you remember correctly, we're fitting the data in a standard scalar, which means instead of the data being from, you know, 5 to 303 in one column and the next column is 1 to 6, we're going to set that all so that all the data is between -1 and 1. That's what that standard scalar does; keeps it standardized. And we only want to fit the scalar with the training set, but we want to make sure the testing set—the x_test going in—is also transformed so it's processing it the same. So here we go with our standard scalar; we're going to call it sc_x for the scalar, and we're going to import the StandardScaler into this variable. And then our x_train = sc_x.fit_transform; so we're creating the scalar on the x_train variable, and then our x_test, we're also going to transform it. So we've trained and transformed the x_train, and then the x_test isn't part of that training; it isn't part of that—of training the transformer; it just gets transformed; that's all it does. And again, we're gonna go and run this. If you look at this, we've now gone through these steps, all three of them. We've taken care of replacing our zeros for key columns that shouldn't be zero, and we replace that with the means of those columns that way that they fit right in with our data models. We've come down here; we split the data, so now we have our test data and our—and then we've taken and we've scaled the data, so all of our data going in—no, no, we don't tr—we don't train the y part, the y_train and y_test; that never has to be trained; it's only the data going in that's what we want to train in there.

Then define the model using k-neighbors classifier and fit the train data in the model. So we do all that data prep, and you can see down here we're only going to have a couple of lines of code where we're actually building our model and training it. That's one of the cool things about Python and how far we've come; it's such an exciting time to be in machine learning because there's so many automated tools. Let's see; before we do this, let's do a quick length of—and let's do y; we want—let's just do len(y), and we get 768. And if we import math, we do math.sqrt—let's do len(y_train). There we go; it's actually supposed to be len(x_train). Before we do this, let's go ahead and do import math and do math.sqrt(len(y_test)), and when I run that, we get 12.409. I want to show you where this number comes from; we're about to use 12; it's an even number. So if you know—if you're ever voting on things, remember the neighbors all vote; don't want to have an even number of neighbors voting, so we want to do something odd, and let's just take one away; we'll make it 11. Let me delete this out of here, this one. The reasons I love Jupyter Notebook is you can flip around and do all kinds of things on the fly. So we'll go ahead and put in our classifier; we're creating our classifier now, and it's going to be the KNeighborsClassifier(n_neighbors=11). Remember we did 12 - 1 for 11, so we have an odd number of neighbors, p=2 because we're looking for—is it—are they diabetic or not? And we're using the Euclidean metric; there are other means of measuring the distance; you could do like square—square means values, all kinds of measures, but the Euclidean is the most common one, and it works quite well.

It's important to evaluate the model; let's use the confusion matrix to do that, and we're going to use the confusion_matrix; wonderful tool. And then we'll jump into the f1_score and finally accuracy_score, which is probably the most commonly used quoted number when you go into a meeting or something like that. So let's go ahead and paste that in there, and we'll set the cm = confusion_matrix(y_test, y_predict); so those are the two values we're going to put in there. And let me go ahead and run that and print it out. And the way you interpret this is you have the y_predicted, which would be your title up here; you could do—let's just do pred—predicted across the top and actual going down. Actual—it's always hard to write in here—actual. That means that this column here down the middle—that's the important column—and it means that our prediction said 94, and prediction and actual agreed on 94 and 32. This number here, the 13 and the 15, those are what was wrong. So you could have like three different—if you're looking at this across three different variables instead of just two, you'd end up with the third row down here and that column going down the middle. So in the first case, we have the—and I believe the zero is in 94 people who don't have diabetes; the prediction said that 13 of those people did have diabetes and were at high risk, and the 32 that had diabetes, it had correct, but our prediction said another 15—out of that 15, it classified as incorrect. So you can see where that classification comes in and how that works on the confusion matrix. Then we're going to go ahead and print the f1_score. Let me just run that, and you see we get a 0.69 in our F1 score. The F1 takes into account both sides of the balance of false positives, where if we go ahead and just do the accuracy account, and that's what most people think of, is it looks at just how many we got right out of how many we got wrong. So a lot of people, when you're a data scientist and you're talking to other data scientists, they're going to ask you what the F1 score, the F-score is. If you're talking to the general public or the decision-makers in the business, they're going to ask what the accuracy is, and the accuracy is always better than the F1 score, but the F1 score is more telling; it lets us know that there's more false positives than we would like on here, but 82%; not too bad for a quick flash look at people's different statistics in running an scikit-learn and running the knn, the k-nearest neighbor on it. So we have created a model using knn, which can predict whether a person will have diabetes or not, or at the very least whether they should go get a checkup and have their glucose checked regularly or not. The print(accuracy_score)—we got the 0.818—was pretty close to what we got, and we can pretty much round that off and just say we have an accuracy of 80%; tells us it is a pretty fair fit in the model.

What is k-means clustering? K-means clustering is an unsupervised learning algorithm. In this case, you don't have labeled data, unlike in supervised learning. So you have a set of data, and you want to group them, and as the name suggests, you want to put them into clusters, which means objects that are similar in nature, similar in characteristics, need to be put together. So that's what k-means clustering is all about. The term k is basically—is a number; so we need to tell the system how many clusters we need to perform. So if k=2, there will be two clusters; if k=3, three clusters, and so on and so forth. That's what the k stands for. And of course, there is a way of finding out what is the best or optimum value of k for a given data; we will look at that. So that is k-means clustering.

So let's take an example. K-means clustering is used in many, many scenarios, but let's take an example of cricket, the game of cricket. Let's say you received data of a lot of players from maybe all over the country or all over the world, and this data has information about the runs scored by the people, ordered by the player, and the wickets taken by the player, and based on this information, we need to cluster this data into two clusters: batsmen and bowlers. So this is an interesting example; let's see how we can perform this. So we have the data which consists of primarily two characteristics, which is the runs and the wickets. So the bowlers basically take wickets, and the batsmen score runs. There will be, of course, a few bowlers who can score some runs, and similarly, there will be some batsmen who will—who would have taken a few wickets. But with this information, we want to cluster those players into batsmen and bowlers. So how does this work? Let's say this is how the data is; so there are information—there is information on the y-axis about the runs scored and on the x-axis about the wickets taken by the players. So if we do a quick plot, this is how it would look. And when we do the clustering, we need to have the clusters like shown in the third diagram. We need to have a cluster which consists of people who have scored high runs, which is basically the batsman, and then we need a cluster with people who have taken a lot of wickets, which is typically the bowlers. There may be a certain amount of overlap, but we will not talk about it right now. So with k-means clustering, we will have here—that means k=2, and we will have two clusters, which is batsmen and bowlers.

So how does this work? The way it works is the first step in k-means clustering is the allocation of two centroids randomly. So two points are assigned as so-called centroids. So in this case, we want two clusters, which means k=2, so two points have been randomly assigned as centroids. Keep in mind these points can be anywhere; they are random points; they are not initially—they are not really the centroids. Centroid means it's a central point of a given data set, but in this case, when it starts off, it's not really the centroid, okay? So these points, though—in our presentation here, we have shown them one point closer to these data points and another closer to these data points—they can be assigned randomly anywhere, okay? So that's the first step. The next step is to determine the distance of each of the data points from each of the randomly assigned centroids. So, for example, we take this point and find the distance from this centroid and the distance from this centroid. This point is taken, and the distance is found from this centroid and the center, and so on and so forth. So for every point, the distance is measured from both the centroids, and then whichever distance is less, that point is assigned to that centroid. So, for example, in this case, visually it is very obvious that all these data points are assigned to this centroid, and all these data points are assigned to this centroid, and that's what is represented here in blue color and in this yellow color. The next step is to actually determine the central point or the actual centroid for these two clusters. So we have this one initial cluster, this one initial cluster, but as you can see, these points are not really the centroid. Centroid means it should be the central position of this dataset, central position of this dataset. So that is what needs to be determined as the next step. So the central point of the actual centroid is determined, and the original randomly allocated centroid is repositioned to the actual centroid of these new clusters. And this process is actually repeated. Now what might happen is some of these points may get reallocated. In our example, that is not happening probably, but it may so happen that the distance is found between each of these data points once again with these centroids, and if there is—if it is required, some points may be reallocated. We will see that in a later example, but for now, we will keep it simple. So this process is continued till the centroid repositioning stops, and that is our final cluster. So this is our—so after iteration, we come to this position, this situation where the centroid doesn't need any more repositioning, and that means our algorithm has converged; convergence has occurred, and we have the cluster—two clusters; we have the clusters with a centroid.

So this process is repeated: the process of calculating the distance and repositioning the centroid is repeated till the repositioning stops, which means that the algorithm has converged and we have the final cluster with the data points and the centroids. So this is what you're going to learn from this session. We will talk about the types of clustering; what is k-means clustering; application of k-means clustering. K-means clustering is done using distance measure, so we will talk about the common distance measures, and then we will talk about how k-means clustering works and go into the details of the k-means clustering algorithm. And then we will end with a demo and a use case for k-means clustering. So let's begin.

First of all, what are the types of clustering? There are primarily two categories of clustering: hierarchical clustering and then partitional clustering. And each of these categories are further subdivided into agglomerative and divisive clustering, and k-means and fuzzy c-means clustering. Let's take a quick look at what each of these types of clustering are. In hierarchical clustering, the clusters have a tree-like structure, and hierarchical clustering is further divided into agglomerative and divisive. Agglomerative clustering is a bottom-up approach; we begin with each element as a separate cluster and merge them into successively larger clusters. So, for example, we have a, b, c, d, e, f. We start by combining b and c from one cluster; d and e form one more; then we combine d, e, and f—one more bigger cluster—and then add b, c to that, and then finally a to it. Compared to that, divisive clustering, or divisive clustering, is a top-down approach; we begin with the whole set and proceed to divide it into successively smaller clusters. So we have a, b, c, d, e, f. We first take that as a single cluster and then break it down into a, b, c, d, e, and f. Then we have partitional clustering, split into two subtypes: k-means clustering and fuzzy c-means. In k-means clustering, the objects are divided into the number of clusters mentioned by the number k—that's where the k comes from. So if we say k is equal to 2, the objects are divided into two clusters, c1 and c2. And the way it is done is the features or characteristics are compared, and all objects having similar characteristics are clubbed together. So that's how k-means clustering is done; we will see it in more detail as we move forward. And fuzzy c-means is very similar to k-means in the sense that it clubs objects that have similar characteristics together, but while in k-means clustering two objects cannot belong to, or any object, a single object cannot belong to two different clusters; in c-means, objects can belong to more than one cluster. So that is the primary difference between k-means and fuzzy c-means.

So what are some of the applications of k-means clustering? K-means clustering is used in a variety of examples, or variety of business cases, in real life, starting from academic performance diagnostic systems, search engines, and wireless sensor networks, and many more. So let us take a little deeper look at each of these examples. Academic performance: so based on the scores of the students, students are categorized into a, b, c, and so on. Clustering forms a backbone of search engines; when a search is performed, the search results need to be grouped together. The search engines very often use clustering to do this. And similarly, in case of wireless sensor networks, the clustering algorithm plays the role of finding the cluster heads which collects all the data in its respective cluster. So clustering, especially k-means clustering, uses distance measure, so let's take a look at what is distance measure. So while these are the different types of clustering, in this video we will focus on k-means clustering.

Distance measure tells how similar some objects are. So the similarity is measured using what is known as distance measure. And what are the various types of distance measures? There is Euclidean distance; there is Manhattan distance; then we have squared Euclidean distance measure and cosine distance measure. These are some of the distance measures supported by k-means clustering. Let's take a look at each of these. What is Euclidean distance measure? This is nothing but the distance between two points. So we have learnt in high school how to find the distance between two points; this is a little sophisticated formula for that, but we know a simpler one is square root of y2 minus y1 whole square plus x2 minus x1 whole square. So this is an extension of that formula. So that is the Euclidean distance between two points. What is the squared Euclidean distance measure? It's nothing but the square of the Euclidean distance—the name says so. Instead of taking the square root, we leave the square as it is. And then we have Manhattan distance measure. In case of Manhattan distance, it is the sum of the distances across the x-axis and the y-axis, and note that we are taking the absolute value so that the negative values don't come into play. So that is the Manhattan distance measure. Then we have cosine distance measure; in this case, we take the angle between the two vectors formed by joining the points from the origin. So that is the cosine distance measure. Okay, so that was a quick overview about the various distance measures that are supported by k-means. Now let's go and check how exactly k-means clustering works.

Okay, so this is how k-means clustering works. This is like a flowchart of the whole process. There is a starting point, and then we specify the number of clusters that we want. Now there are a couple of ways of doing this; we can do by trial and error. So we specify a certain number, maybe k is equal to 3 or 4 or 5 to start with, and then as we progress we keep changing until we get the best clusters, or there is a technique called elbow technique whereby we can determine the value of k; what should be the best value of k; how many clusters should be formed. So once we have the value of k, we specify that, and then the system will assign that many centroids. So it picks randomly, that to start with, randomly that many points that are considered to be the centroids of these clusters. And then it measures the distance of each of the data points from these centroids and assigns those points to the corresponding centroid from which the distance is minimum. So each data point will be assigned to the centroid which is closest to it, and thereby we have k number of initial clusters. However, this is not the final clusters. The next step it does is, for the new groups, for the clusters that have been formed, it calculates the mean position, thereby calculates the new centroid position. The position of the centroid moves compared to the randomly allocated one. So it's an iterative process. Once again, the distance of each point is measured from this new centroid point, and if required the data points are reallocated to the new centroids, and the mean position or the new centroid is calculated once again. If the centroid moves, then the iteration continues, which means the convergence has not happened; the clustering has not converged. So as long as there is a movement of the centroid, this iteration keeps happening, but once the centroid stops moving, which means that the cluster has converged or the clustering process has converged, that will be the end result. So now we have the final position of the centroid, and the data points are allocated accordingly to the closest centroid. I know it's a little difficult to understand from this simple flowchart, so let's do a little bit of visualization and see if we can explain it better.

Let's take an example: if we have a data set for a grocery shop. So let's say we have a data set for a grocery shop, and now we want to find out how many clusters this has to be spread across. So how do we find the optimum number of clusters? There is a technique called the elbow method. So when these clusters are formed, there is a parameter called within sum of squares, and the lower this value is, the better the cluster is; that means all these points are very close to each other. So we use this within sum of squares as a measure to find the optimum number of clusters that can be formed for a given data set. So we create clusters, or we let the system create clusters of a variety of numbers, maybe of 10—10 clusters—and for each value of k the within SS is measured, and the value of k which has the least amount of within SS, or WSS, that is taken as the optimum value of k. So this is the diagrammatic representation. So we have on the y-axis the within sum of squares, or WSS, and on the x-axis we have the number of clusters. So as you can imagine, if you have k is equal to one, which means all the data points are in a single cluster, the within SS value will be very high because they are probably scattered all over. The moment you split it into two, there will be a drastic fall in the within SS value—that's what is represented here—but then as the value of k increases, the decrease, the rate of decrease will not be so high; it will continue to decrease, but probably the rate of decrease will not be high. So that gives us an idea. So from here we get an idea; for example, the optimum value of k should be either 2 or 3 or at the most four, but beyond that increasing the number of clusters is not dramatically changing the value in WSS because that pretty much gets stabilized.

Okay, now that we have got the value of k, and let's assume that these are our delivery points, the next step is basically to assign two centroids randomly. So let's say c1 and c2 are the centroids assigned randomly. Now the distance of each location from the centroid is measured, and each point is assigned to the centroid which is closest to it. So for example, these points are very obvious that these are closest to c1, whereas this point is far away from c2. So these points will be assigned—which are close to c1—will be assigned to c1, and these points or locations which are close to c2 will be assigned to c2. And then, so this is the how the initial grouping is done; this is part of c1, and this is part of c2. Then the next step is to calculate the actual centroid of this data because remember c1 and c2 are not the centroids; they have been randomly assigned points, and only thing that has been done was the data points which are closest to them have been assigned. But now in this step the actual centroid will be calculated, which may be for each of these data sets somewhere in the middle. So that's like the main point that will be calculated, and the centroid will actually be positioned or repositioned there; same with c2. So the new centroid for this group is c2 in this new position, and c1 is in this new position. Once again, the distance of each of the data points is calculated from these centroids. Now remember it's not necessary that the distance still remains the, or each of these data points still remain in the same group. By recalculating the distance, it may be possible that some points get reallocated, like so you see this. So this point earlier was closer to c2 because c2 was here, but after recalculating, repositioning, it is observed that this is closer to c1 than c2. So this is the new grouping; some points will be reassigned, and again the centroid will be calculated. And if the centroid doesn't change, so that is a repetitive process, iterative process, and if the centroid doesn't change—once the centroid stops changing—that means the algorithm has converged, and this is our final cluster with this as the centroid, c1 and c2 as the centroids, these data points as a part of each cluster. So I hope this helps in understanding the whole process, iterative process of k-means clustering. So let's take a look at the k-means clustering algorithm.

Let's say we have x1, x2, x3, n number of points as our inputs, and we want to split this into k clusters, or we want to create k clusters. So the first step is to randomly pick k points and call them centroids; they are not real centroids because centroid is supposed to be a center point, but they are just called centroids. And we calculate the distance of each and every input point from each of the centroids. So the distance of x1 from c1, from c2, c3—each of the distances we calculate—and then find out which distance is the lowest and assign x1 to that particular random centroid. Repeat that process for x2; calculate its distance from each of the centroids, c1, c2, c3 to ck, and find which is the lowest distance and assign x2 to that particular center; same with x3 and so on. So that is the first round of assignment that is done. Now we have k groups because there are—we have assigned the value of k—so there are k centroids, and so there are k groups; all these inputs have been split into k groups. However, remember we pick the centroids randomly, so they are not real centroids. So now what we have to do, we have to calculate the actual centroids for each of these groups, which is like the mean position, which means that the position of the randomly selected centroids will now change, and they will be the main positions of these newly formed k groups. And once that is done, we once again repeat this process of calculating the distance, right. So this is what we are doing as a part of step four; we repeat step two and three. So we again calculate the distance of x one from the centroid c one, c two, c three and then c, which is the lowest value, and assign x1 to that; calculate the distance of x2 from c1, c2, c3 or whatever up to ck, and find whichever is the lowest distance and assign x2 to that centroid, and so on. In this process, there may be some reassignment; x1 pro was probably assigned to cluster c2, and after doing this calculation maybe now x1 is assigned to c1; so that kind of reallocation may happen. So we repeat the steps two and three till the position of the centroids don't change or stop changing, and that's when we have convergence.

So let's take a detailed look at it at each of these steps. So we randomly pick k cluster centers; we call them centroids because they are not initially—they are not really the centroids. So we let us name them c1, c2 up to ck. And then step two, we assign each data point to the closest center. So what we do, we calculate the distance of each x value from each c value. So the distance between x1, c1; distance between x1, c2; x1, c3; and then we find which is the lowest value, right—that's the minimum value we find—and assign x1 to that particular centroid. Then we go next to x2; find the distance of x2 from c1, x2 from c2, x2 from c3 and so on up to ck, and then assign it to the point or to the centroid which has the lowest value, and so on. So that is step number two. In step number three, we now find the actual centroid for each group. So what has happened as a part of step number two, we now have all the points, all the data points grouped into k groups because we—we wanted to create k clusters, right? So we have k groups; each one may be having a certain number of input values; they need not be equally distributed, by the way. Based on the distance, we will have k groups, but remember the initial values of the c1, c2 were not really the centroids of these groups, right? We assigned them randomly. So now in step 3, we actually calculate the centroid of each group, which means the original point which we thought was the centroid will shift to the new position which is the actual centroid for each of these groups. Okay, and we again calculate the distance. So we go back to step two, which is what we calculate again the distance of each of these points from the newly positioned centroids, and if required we reassign these points to the new centroids. So as I said earlier, there may be a reallocation. So we now have a new set or a new group; we still have k groups, but the number of items and the actual assignment may be different from what was in step two here. Okay, so that might change. Then we perform step three once again to find the new centroid of this new group. So we have again a new set of clusters, new centroids, and new assignments. We repeat this step two again; once again we find, and then it is possible that after iterating through three or four or five times the centroid will stop moving, in the sense that when you calculate the new value of the centroid that will be same as the original value, or there will be very marginal change. So that is when we say convergence has occurred, and that is our final cluster; that's the formation of the final cluster. All right.

So let's see a couple of demos of k-means clustering. We will actually see some live demos in python notebook using pattern notebook, but before that let's find out what's the problem that we are trying to solve. The problem statement is: let's say Walmart wants to open a chain of stores across the state of Florida, and it wants to find the optimal store locations. Now the issue here is, if they open too many stores close to each other, obviously the—they will not make profit, but if they—if the stores are too far apart, then they will not have enough sales. So how do they optimize this? Now for an organization like Walmart, which is an e-commerce giant, they already have the addresses of their customers in their database, so they can actually use this information or this data and use k-means clustering to find the optimal location. Now before we go into the python notebook and show you the live code, I wanted to take you through very quickly a summary of the code in the slides, and then we will go into the python notebook.

So in this block we are basically importing all the required libraries like numpy, matplotlib and so on, and we are loading the data that is available in the form of, let's say, the addresses; for simplicity sake we will just take them as some data points. Then the next thing we do is quickly do a scatter plot to see how they are related to each other with respect to each other. So in the scatter plot we see that there are a few distinct groups already being formed, so you can actually get an idea about how the cluster would look and how many clusters—what is the optimal number of clusters—and then starts the actual k-means clustering process. So we will assign each of these points to the centroids and then check whether they are the optimal distance, which is the shortest distance, and assign each of the points, data points, to the centroids, and then go through this iterative process till the whole process converges, and finally we get an output like this. So we have four distinct clusters, and which is—if we can say that this is how the population is probably distributed across Florida state, and the centroids are like the location where the store should be—the optimum location where the store should be. So that's the way we determine the best locations for the store, and that's how we can help Walmart find the best locations for the stores in Florida. So now let's take this into python notebook; let's see how this looks when we are running—running the code live.

All right, so this is the code for k-means clustering in Jupyter notebook. We have a few examples here which we will demonstrate how k-means clustering is used, and even there is a small implementation of k-means clustering as well. Okay, so let's get started. Okay, so this block is basically importing the various libraries that are required like matplotlib and numpy and so on and so forth, which would be used as a part of the code. Then we are going and creating blobs which are similar to clusters. Now this is a very neat feature which is available in scikit-learn; make_blobs is a nice feature which creates clusters of data sets. So that's a wonderful functionality that is readily available for us to create some test data kind of thing. Okay, so that's exactly what we are doing here; we are using make_blobs, and we can specify how many clusters we want; centers we are mentioning here, so it will go ahead and—so we just mentioned four—so it will go ahead and create some test data for us, and this is how it looks. As you can see visually also, we can figure out that there are four distinct classes or clusters in this data set, and that is what make_blobs actually provides. Now from here onwards we will basically run the standard k-means functionality that is readily available. So we really don't have to implement k-means itself; the k-means functionality or the function is readily available; you just need to feed the data, and we'll

Create the clusters. So this is the code for that. We import k-means, and then we create an instance of k-means, and we specify the value of k. This n_clusters is the value of k. Remember, k-means; in k-means, k is basically the number of clusters that you want to create, and it is an integer value. So this is where we are specifying that. So we have k is equal to 4, and so that instance is created. We take that instance and, as with any other machine learning functionality, fit is what we use, the function or the method rather; fit is what we use to train the model. Here, there is no real training kind of thing, but that's the call, okay? So we are calling fit, and what we are doing here, we are just passing the data. So x has these values, the data that has been created, right? So that is what we are passing here, and this will go ahead and create the clusters.

And then we are using, after doing uh fit, we run the predict, which basically assigns, for each of these observations, which cluster where it belongs to. All right? So it will name the clusters—maybe this is cluster 1, this is 2, 3, and so on—or I will actually start from 0: cluster 0, 1, 2, and 3 maybe. And then, for each of the observations, it will assign, based on which cluster it belongs to, it will assign a value. So that is stored in y_kmeans when we call predict. That is what it does, and we can take a quick look at these uh y_kmeans, or with the cluster numbers that have been assigned for each observation. So this is the cluster number assigned for observation 1, maybe this is for observation 2, observation 3, and so on. So we have how many—about I think 300 samples, right? So all the 300 samples, there are 300 values here; each of them, the cluster number is given, and the cluster number goes from 0 to 3. So there are four clusters; so the numbers go from zero, one, two, three. So that's what is seen here. Okay. Now, so this was a quick example of generating some dummy data and then clustering that. Okay, and this can be applied; if you have proper data, you can just load it up into x, for example here, and then run the game. So this is the central part of the k-means clustering program example. So you basically create an instance, and you mention how many clusters you want by specifying this parameter n_clusters, and that is also the value of k, and then pass the data to get the values.

Now, the next section of this code is the implementation of a k-means. Now, this is kind of a rough implementation of the k-means algorithm, so we will just walk you through—I will walk you through the code—at each step what it is doing, and then we will see a couple of more examples of how k-means clustering can be used in maybe some real-life examples, real-life use cases. All right. So in this case here, what we are doing is basically implementing k-means clustering, and there is a function or a library calculates, for a given two pairs of points, it will calculate the the distance between them and see which one is the closest and so on. So this is like—this is pretty much like what k-means does, right? So it calculates the distance of each point or each data set from predefined centroid, and then based on whichever is the lowest, this particular data point is assigned to that centroid. So that is basically available as a standard function, and we will be using that here. So as explained in the slides, the first step that is done in case of k-means clustering is to randomly assign some centroids. So as a first step, we randomly allocate a couple of centroids, which we call here—we are calling as centers—and then we put this in a loop, and we take it through an iterative process. For each of the data points, we first find out, using this function pairwise distance argument, for each of the points we find out which one—which center or which randomly selected centroid is the closest—and accordingly we assign that data or the data point to that particular centroid or cluster. And once that is done for all the data points, we calculate the new centroid by finding out the mean position, which is the center position, right? So we calculate the new centroid, and then we check if the new centroid is the coordinates or the position is the same as the previous centroid—the positions we will compare—and if it is the same, that means the process has converged. So remember, we do this process till the centroids or the centroid doesn't move anymore, right? So the centroid gets relocated each time this reallocation is done. So the moment it doesn't change anymore—the position of the centroid doesn't change anymore—we know that convergence has occurred. So till then—so you see here, this is like an infinite loop; while True is an infinite loop; it only breaks when the centers are the same—the new center and the old center positions are the same—and once that is done, we return the centers and the labels. Now, of course, as explained, this is not a very sophisticated and advanced implementation—very basic implementation—because one of the flaws in this is that sometimes what happens is the centroid—the position will keep moving, but in the change will be very minor. So in that case also—with that is actually convergence, right? So for example, the change is 0.0001; we can consider that as convergence; otherwise, what will happen is this will either take forever or it will be never-ending. So that's a small flaw here. So that is something additional checks may have to be added here, but again, as mentioned, this is not the most sophisticated implementation; this is like a kind of a rough implementation of the k-means clustering. Okay. So if we execute this code, this is what we get as the output. So this is the definition of this particular function, and then we call that find_clusters, and we pass our data x and the number of clusters, which is four, and if we run that and plot it, this is the output that we get. So this is, of course, each cluster is represented by a different color. So we have a cluster in green color, yellow color, and so on and so forth, and these big points here, these are the centroids—this is the final position of the centroids—and as you can see visually also, this appears like a kind of a center of all these points here, right? Similarly, this is like the center of all these points here, and so on. So this is the example, or this is an example of an implementation of k-means clustering, and next we will move on to see a couple of examples of how k-means clustering is used in maybe some real-life scenarios or use cases.

In the next example or demo, we are going to see how we can use k-means clustering to perform color compression. We will take a couple of images, so there will be two examples, and we will try to use k-means clustering to compress the colors. This is a common situation in image processing when you have an image with millions of colors, but then you cannot render it on some devices which may not have enough memory; uh, so that is the scenario where something like this can be used. So before again we go into the python notebook, let's take a look at quickly the the code. As usual, we import the libraries, and then we import the image, and then we will flatten it. So the reshaping is basically—we have the image information is stored in the form of pixels, and if the image is like, for example, 427 by 640, and it has three colors, so that's the overall dimension of the of the initial image—we just reshape it, and then feed this to our algorithm, and this will then create clusters of only 16 clusters. So this—this colors—there are millions of colors, and now we need to bring it down to 16 colors. So we use k is equal to 16, and this is how—when we visualize—this is how it looks. There are—these are all about 16 million possible colors; the input color space has 16 million possible colors, and we just sub-compress it to 16 colors. So this is how it would look when we compress it to 16 colors, and this is how the original image looks, and after compression to 16 colors, this is how the new image looks. As you can see, there is not a lot of information that has been lost, though the image quality is definitely reduced a little bit. So this is an example which we are going to now see in python notebook. Let's go into the python node, and once again, as always, we will import some libraries and load this image called flower.jpg. Okay, so let me load that, and this is how it looks. This is the original image which has, I think, 16 million colors, and this is the shape of this image, which is basically—what is the shape is nothing but the overall size, right? So this is 427 pixel by 640 pixel, and then there are three layers, which is this three basically is for RGB, which is red, green, blue. So color image will have that, right? So that is the shape of this. Now what we need to do is data—let's take a look at how data is looking. So let me just create a new cell and show you what is in data. Basically, we have captured this information. So data is what—let me just show you here—all right. So let's take a look at china—what are the values in china—and if we see here, this is how the data is stored. This is nothing but the pixel values. Okay, so this is like a matrix, and each one has about four for this 427 by 640 pixels. All right, so this is how it looks. Now the issue here is these values are large—the numbers are a large—so we need to normalize them to between zero and one, right? So that's why we will basically create one more variable, which is data, which will contain the values between 0 and 1, and the way to do that is divide by 255. So we divide china by 255, and we get the new values in data. So let's just run this piece of code, and this is the shape. So we now have also—yeah, what we have done is we changed—using reshape—we converted into the three-dimensional into a two-dimensional data set, and let us also take a look at how—let me just insert uh probably a cell here and take a look at how data is looking. All right, so this is how data is looking, and now you see this is the values are between zero and one, right? So if you earlier noticed in case of china, the values were large numbers; now everything is between zero and one. This is one of the things we need to do. All right. So after that, the next thing that we need to do is to visualize this, and we can take random set of maybe 10,000 points and plot it and check and see how this looks. Let us just plot this, and so this is how the original—the color—the pixel distribution is. These are two plots: one is red against green, and another is red against blue, and this is the original distribution of the color. So then what we will do is we will use k-means clustering to create just 16 clusters for the various colors and then apply that to the image. Now what will happen is since the data is large because there are millions of colors, using regular k-means may be a little time-consuming, so there is another version of k-means which is called mini batch k-means, so we will use that, which is—which processes—in the overall concept remains the same, but this basically processes it in smaller batches; that's the only thing. Okay, so the results will pretty much be the same. So let's go ahead and execute this piece of code and also visualize this so that we can see that there are—this is how the 16 colors uh would look. So this is red against green, and this is red against blue; there is uh quite a bit of similarity between this original color schema and the new one, right? So it doesn't look very very completely different or anything like that. Now we apply this—the newly created colors to the image—and we can take a look uh how this is looking. Now we can compare both the images. So this is our original image, and this is our new image. So as you can see, there is not a lot of information that has been lost; it pretty much looks like the original image. Yes, we can see that, for example, here there is a little bit—it appears a little dullish compared to this one, right? Because we kind of took off some of the finer details of the color, but overall the high-level information has been maintained; at the same time, the main advantage is that now this can be—this is an image which can be rendered on a device which may not be that very sophisticated.

Now let's take one more example with a different image. In the second example, we will take an image of the Summer Palace in China, and we repeat the same process. This is a high-definition color image with millions of colors and also three-dimensional. Now we will reduce that to 16 colors using k-means clustering, and we do the same process like before: we reshape it, and then we cluster the colors to 16, and then we render the image once again, and we will see that the color—the quality of the image—slightly deteriorates. As you can see here, this has much finer details in this which are probably missing here, but then that's the compromise because there are some devices which may not be able to handle this kind of high-density images. So let's run this code in python notebook. All right, so let's apply the same technique for another picture which is uh even more intricate and has probably much more complicated color schema. So this is the image. Now once again, we can take a look at the shape, which is 427 by 640 by 3, and this is the new data would look somewhat like this compared to the flower image. So we have some new values here, and we will also bring this—as you can see, the numbers are much big—so we will—much bigger—so we will now have to scale them down to values between 0 and 1, and that is done by dividing by 255. So let's go ahead and do that and reshape it. Okay, so we get a two-dimensional matrix, and we will then—as a next step—we will go ahead and visualize this—how it looks—does the 16 colors, and this is basically how it would look: 16 million colors, and now we can create the clusters out of this—the 16 k-means clusters we will create. So this is how the distribution of the pixels would look with 16 colors, and then we go ahead and apply this and visualize how it is looking for with the new—just the 16 color. So once again, as you can see, this looks much richer in color, but at the same time—and this probably doesn't have—as we can see, it doesn't look as rich as this one, but nevertheless, the information is not lost—the shape and all that stuff—and this can be also rendered on a slightly—a device which is probably not that sophisticated. Okay, so that's pretty much it. So we have seen two examples of how color compression can be done uh using k-means clustering, and we have also seen in the previous examples of how to implement k-means—the code to—roughly how to implement k-means clustering—and we use some sample data using blob to just execute the k-means clustering.

Now you'll notice here we've got—we're using pandas—whoops—drew it through the pandas, and we're going to go ahead and read it as a pandas data frame, and of course pandas sits on numpy, so a lot of stuff you do with pandas can go right into numpy very easily. And the reason I bring this up is because pandas has a lot of tools in there for just looking at the data real quick and easy. And so the first one we want to go ahead and do is do credit.df or credit_df—our data—we read in info and just see what that looks like. And when we go ahead and print this out, you can see here we have 13 columns. Remember, we were looking at our data, and it had those—that row—you couldn't read across the top; um, then it has non-null count—how many values don't have null—what kind of objects are they? They're an object, an integer, different things like that, and we look at object—is it a checking account? This is probably text on here; uh, duration—how long credit history—amount—savings account—present employee—employed sense—so a lot of different numbers coming in here, and these are just the columns we're looking at. And if you're going to look at the columns, it's kind of nice to actually look at the data, and so we'll use the pandas to go ahead and print it out just like you would on a spreadsheet. Hard to see them all on here, but you can see we have—here's our row across the top—because pandas does such a good job sorting out columns and the actual data—checking account—a11, a12, a14—not sure what that means—duration—how long they've had it—credit history—not sure what the a33 or a34 means—I'm sure it has a special meaning on there—the amount—here's our savings account—and all the information on that—and this is actually a very—a much more easy-to-read view than what we had before as far as like that text file—when you're just looking at data through the database, sometimes those can be really hard to read—this you can really see—and if I take it down a notch, I can get all the columns in here. We're going to keep it zoomed in just so we can see that what's going on a little easier, and then we can also do credit_df.iloc—the iloc feature in the pandas is going to go ahead and do rows 0 through 5, and we're going to look at columns 1 through 7. And let's go ahead and just run that, and you can see right here how we just split it up, so it's really easy to manipulate our data and get a a smaller view of different parts of the data as we're going through it. And just like we can look at the first seven columns, we can also look at the last seven columns; uh, same kind of format—we're looking at rows zero through five, column seven to the end—if you leave the number out there—and you can see we can easily now look at this on a smaller screen—much better for displaying—those are just tricks to help you display your information so that when you present this in front of a group of people, it makes it very easy to read, or you just copy this out and put it into a document, that kind of thing. And then we also like to go ahead and do like credit_status.value_counts and things like that. Again, we're just looking at the different uh sets of information on here, and in our pandas data frame, value_counts is—we're looking at unique values. So there's a lot of unique values in here—that's 700 and 300—that's a significantly diverse set of data. And if we look at the data, one of the things we're looking at is the 0 status 1, and I'm going to guess that that means that it is either default or not default, so that's probably what we're going to want to look for. Hopefully, you've had your specialty and domain—remember, we talked about the different parts of data science—domain is what field it is in—this would be in banks and loans—and so if we want to figure out what the status of a new loan is, we're going to separate the status out. And so if we take our x_features equals a list of the columns—that's all this is—credit_df.columns—we remove status—and then we go ahead—and this will print our features out—and you can see it's everything but the status on the end, because that's what we want to predict—we want to predict the new data whether that status is 0 or 1 or not. And next we want to do a little bit more data preparation; we're going to go ahead and encode—um—and we're going to take an encoded credit_df; we're going to create that off of our um setup on here—and you're going to ask, well, what are we encoding on here? Well, when we go up to here—let me just go…

Back all the way when we're looking at all this information up here, you can see there's object object object. Well, if I'm doing a prediction and something's just a random object, it doesn't really mean anything; there's no numbers to process. And so when we come down a little further, we can see down here that we are looking at A65. I'm not sure what A65 means. We're looking at a 93 personal status; we're looking at A143. These aren't actual numbers; there's some kind of code that the bank put in, probably. And so for our model, they really don't mean anything unless we convert them to numbers. And you convert them to numbers by literally creating a bunch of yes/no's. So instead of thinking it as A173 and A172 as being 0, 1, and if you had an A174, maybe 0, 1, 2, you would think of this as a label: A173 is true or false, and then you'd have another label A172 as true or false. So for every unique job in here, that it becomes another feature added in, and it's on or off; it's true or false. That's what this encoding is doing. And we'll go ahead and list our encoding as it comes out. And if you come up here, you can see it did exactly what I was describing. We've now encoded it. So here's checking account A13, checking account A12, one here's all of our uh checking account numbers, so there was three of them. Uh, here's our credit history: A31, A32, A33, A34. Again, the computer has no idea of knowing what those are; all it knows is those are each unique features. And the same thing with the savings account A62 and so on going down. And so once we have our nice list of there, we can go ahead and do—let me pull this in here—we go. Let's look at our new data set. And if we go ahead and run the new data set, we have our income credit df uh checking, and we're just going to look at these and look at the head. This is just the first few of these so we can see what it actually looks like in the data. And just like I said, it's now uh 0 0 0 0 1 1 0 0. This first column is just the count 0 1 2 3 4; that's not part of the data, but you can see here that it just puts a check mark of one because it's an AC A12, and we have a checking account AC A14 where it's also a check mark of one going across.

So we've done a little data preparation; we've come in here and done some exploratory data analysis, and now we want to start building our data model. And we'll kind of go back and forth between exploratory data analysis and data model and then interpreting the results on this. And so when we go ahead and import our statsmodels as api as sm, now a lot of times we use the sk learner, the scikit models. In this one, we're going to go ahead and use the sm model from statsmodels, which is pretty common for doing a very basic logistics regression model. And then from the scikit, we're going to have our train_test_split. With any data you want to go ahead and train your model, and then you need something to test it on some unknown data so you can see how well you did. Very common in your day in your setting up your models is to split it up into your training set and your testing set. And what I want you to notice on here is we have x_train. Well, what is x_train? That's all the columns except for the last one, which was the status: was it default or was it in good standing? And then, of course, our x_test is part of the data. We're going to split that up by 0.3, so 30 percent of the data is going to be—30 of our rows are going to be put in the x_test. And the y_train and y_test is just the status. And then we create our—once we've created our data and split it up, we go and create our model, and it's a logit. We're just going to call it logit, but you—a lot of times you end up seeing logic model with logit—it equals the sm.logit x_train um y_train. x_train in sk learn it's always x_train y_train, uh, but in the stats model it's reversed on there. And then they go ahead and run the fit. So we load the data up, and then we fit it, and then we can go ahead and summarize that model and see what it looks like. Let's go ahead and run that; it takes it just a moment, and we get a nice summary of our data. Again, we're still looking at the data, and we have—takes just a moment to to kind of feed through—and you can certainly see where we have our coefficient, which has to do with the weights in the model, standard error. That's very important as we look at this coefficient and how it affects the model. P is greater than the absolute value of z 0.025. These are all your statistical notes on here. So as you look at these, you might look at like how big is a standard error; that's one of the big ones that I usually look at. As we go up and we start looking at the different things, the higher the error, the less—the more we want to be a little careful of it; the lower the error, the better, in more general terms. They also have dealing with the model and it's iterations. You can see some general status on here, how many iterations that I have to go through before I was able to come up with the numbers, because it kind of guesses; it's playing a high-low game, and it takes it six iterations until it finds what it considers the best fit. And you can see there's more information up here as far as when it converts, how it converts, and so on.

Now, when we have our summary up there, one of the questions starts coming in. Remember, I told you we're looking at like our standard error, and you know how these numbers all are telling us how important each one of these features is in running our prediction. And if you have a ton of features, and some of them are just kind of all over the place, all they do is create noise, so we really want to get rid of them. They also cause issues in that if you're running a huge amount of data, and you have to rerun this model over and over again, this can really text your time, and so you're losing both time, money, and your results might not be as good because they might be weighted based on some of these very fuzzy numbers. So we're going to look at significant variables and put together a little routine for that. Again, we're—we've come in—if you remember from back here, we're still in kind of an exploratory data analysis and data modeling; they kind of go hand in hand sometimes. We haven't quite got to interpreting the results in running predictions. And so we'll go ahead and build this thing on here. It looks at the lm values, so that's coming up here where this was generated. I have to actually look that up for the this particular model because I usually use the sk learn version of this. So as we look at this, we're gonna look at the p-values of our variables, and basically we're looking to see how good the p-values are. If they're less than or equal to 0.05, we probably—those are probably solid values, and if they're greater than that, then probably we're not interested in them. And so let's go ahead and run this, and we see that when we pull out all the values that are less than or equal to 0.05 for the p-values, and if we go up here—here we go—here's our p—I knew it was up here; I just wasn't seeing it correctly. So you can probably see duration, that's going to be good amount, 0.01, that looks good; instance, instant inst rate, which I'm guessing is the percentage rate or the interest it's trying to say in inst and interest at the same time. So we have our interest rate and all that. If we come down here, we see duration, amount, our instant rate, age, checking account A13, checking account A14, credit history A34, savings account A65. These are our primary features; these ones are related. When we look at the data, these are the ones that are going to predict whether it's a good or bad loan. And so now that we know which ones work, we want to go ahead and create our—recreate our sm model or our final logit model. And you'll see here we now have the x_train, the y_train, and then instead of the x_train, it's x_train significant variables right there. So all we're doing is is running this just on the significant variables in the setup, and we'll go ahead and fit it. And you can see here you can actually double bracket and do it all in one line. Up above we had to do it in the two lines; this is how the setup is for this particular module. And then we'll go ahead and do a final logit summary and just see what that looks like. And we can see here—here's our p scores; remember they're all supposed to be under 0.0—underneath the 0.05; these are all under 0.05. You can see up here different statuses as far as how it ran; it still took six iterations to go through the data, the date and so forth. And now that we have a solid model—and that's if you remember correctly—let me flip back on over here—we've done our data modeling, we've created a solid model, and so now we need to actually run a prediction so that we can interpret the results and weigh in on that and see how good they are. And so for that, I'm going to go ahead and—let's find it over here—y_predict. So we have our model that we created; we want to go ahead and create our y_predict, and it's based on our final object. And here's the magic word: predict or .predict. And so this is where we're going to come in and actually get our predictions on here that our model comes up. And remember we're testing—x_test is going in; there's our x_test, and it's only the significant variables. So we've created our model; now we want to push in our test version that's never seen this data before, so this is all new. And then we're just going to take a random sample to see what it looks like; what do those predictions look like? In this random sample, we're just looking at 10 of these, and you can see here is the actual number; it says 0.08 gives us a 1, 0.07. This is our probability number, which is—I would have to actually look to see whether um higher or lower is better; I'm guessing the closer to zero the better it is, but I'm not completely sure on that; I have to actually look up on and see with this particular framework. But you can see the actual prediction: 1 0 0 0. So I guess this is a good loan, a bunch of defaults and some good loans; that's what it predicts it's going to come out as. Now, just looking at the data, you know, it's confusing. This is—you have different models; they have different setup; they have the predicted probability comes out on this, and other models it doesn't. So you really need a tool to dig deeper into the setup to understand what this means. And so what we're going to do is we're going to take our predicted values, which you can see right here has an actual predicted probability, and then we're going to go ahead and insert on there from a sample. Let me just run this because it'll make a lot more sense. We have our actual value and our predicted value. So this is what we predicted, and this is the actual setup. If we just look at this right off the bat, we can see that we predicted zero, and this was a one. These all came out correct. Number 332 came out correct. 917 was an actual—they were good on their loan; they predicted it's going to be a default. You start to see that there's some variance. So how do we measure that? How do we know whether our model was any good or not? Well, for that, we're going to go ahead and load up a nice graph, and we'll use—for our graph, we'll import matplotlib.pyplot and seaborn. Seaborn sits on matplotlib, and this matplotlib inline is just because I'm using Jupyter Notebook; with the new version I'm in, it doesn't even need it, but I tend to leave it in for older versions. And then we will import from our sklearn.metrics; this is your scikit—scikit is the main package, and sklearn is how it looks on here—and we'll go ahead and run that so it's nicely loaded. And then we'll go ahead and create a little routine in just a moment here to print this out so we can see what that looks like. And this metrics—it's kind of a fun matrix because it—it tells you like how many you have correct on the correct one, how many you had wrong on the wrong one. And this top part is just all a draw routine, so it comes in here and has our actual and are predicted, and it's our cm metrics. It breaks it up, puts it into the heat map; the sns.heatmap gives it some colors, add some labels, um, and then we go ahead and show it, and we're just going to dump that in there and run that. And you can spend forever building graphics; it's so much fun. We come in here; we have bad credit, good credit; we have bad credit, good credit; we have predicted label. And so when you look at this, you can see here if it had on the good credit, it tells you how well it predicted it and how many false positive it has. And on our bad credit here, we have bad credit, bad credit; it said 30 of those were bad credit; it predicted them bad credit. Well, 21 were good credit that it predicted would be bad credit. And then we have here—we have 188 that were good credit, uh, but the actual val, and then they had 61 there were bad credit that it said were predicted good credit. Uh, so at this point you look at this and you'd say, well, if I'm predicting—these are—this is our good credit here; let me flip this on over here, and this is our bad credit. There we go. So we have bad credit, good credit. You start looking at and you say, okay, can we afford that many defaults after five years or whatever it is, um, and so those are the questions you start asking: Is this model work good enough to predict whether we should give these people loans or not? You're always going to have somebody default on them. And again, that starts going into domain—banking domain—something I don't know very much about, clearly. I know whether I default on a loan or not, which I don't, and I try very hard not to. And then if we're going to draw the chart, which is really nice to have, just a clear visual chart. Metrics also has our classification report, and you can see these are just the numbers we're looking at; they let us know the precision, our f-score and recall and all that stuff on here. You can just print a straight up metrics classification report. Now we looked at a regression model; let's go ahead and look at a clustering model to see what that looks like. And we'll go ahead and create a new troop Python on there, and this is going to be our clustering setup. Let me flip this off, and swear—and done some exploratory data analysis, and now we want to start building our data model. And we'll kind of go back and forth between exploratory data analysis and data model and then interpreting the results on this. And so when we go ahead and import our statsmodels as api as sm, now a lot of times we use the sk learner, the scikit models. In this one, we're going to go ahead and use the sm model from statsmodels, which is pretty common for doing a very basic logistics regression model. And then from the scikit, we're going to have our train_test_split. With any data you want to go ahead and train your model, and then you need something to test it on some unknown data so you can see how well you did. Very common in your day in your setting up your models is to split it up into your training set and your testing set. And what I want you to notice on here is we have x_train. Well, what is x_train? That's all the columns except for the last one, which was the status: was it default or was it in good standing? And then, of course, our x_test is part of the data. We're going to split that up by 0.3, so 30 percent of the data is going to be—30 of our rows are going to be put in the x_test. And the y_train and y_test is just the status. And then we create our—once we've created our data and split it up, we go and create our model, and it's a logit. We're just going to call it logit, but you—a lot of times you end up seeing logic model with logit—it equals the sm.logit x_train um y_train. x_train in sk learn it's always x_train y_train, uh, but in the um stats model it's reversed on there. Uh, and then they go ahead and run the fit. So we load the data up, and then we fit it, and then we can go ahead and summarize that model and see what it looks like. Let's go ahead and run that; it takes it just a moment, and we get a nice summary of our data. Again, we're still looking at the data, and we have—takes just a moment to to kind of feed through—and you can certainly see where we have our coefficient, which has to do with the weights in the model, standard error. That's very important as we look at this coefficient and how it affects the model. P is greater than the absolute value of z 0.025. These are all your statistical notes on here. So as you look at these, you might look at like how big is a standard error; that's one of the big ones that I usually look at. As we go up and we start looking at the different things, the higher the error, the less—the more we want to be a little careful of it; the lower the error, the better, in more general terms. They also have dealing with the model and it's iterations. You can see some general status on here, how many iterations that I have to go through before I was able to come up with the numbers, because it kind of guesses; it's playing a high-low game, and it takes it six iterations until it finds what it considers the best fit. And you can see there's more information up here as far as when it converts, how it converts, and so on.

Probably, we're not interested in them, and so let's go ahead and run this. And we see that when we pull out all the values that are less than or equal to 0.05 for the p-values, and if we go up here, here we go. Here's our p; I knew it was up here; I just wasn't seeing it correctly. So you can probably see duration, that's going to be a good amount, 0.01; that looks good. Instance, instant, inst rate, which I'm guessing is the percentage rate or the interest it's trying to say in inst and interest at the same time. So we have our interest rate and all that. If we come down here, we see duration, amount, our instant rate, age, checking account a13, checking account a14, credit history a34, savings account 65. These are our primary features; these ones are related when we look at the data. These are the ones that are going to predict whether it's a good or bad loan.

And so now that we know which ones work, we want to go ahead and create our recreate our sm model or our final logit model. And you'll see here we now have the x-train, the y-train, and then instead of the x-train, it's x-train significant variables right there. So all we're doing is is running this just on the significant variables in the setup. And we'll go ahead and fit it, and you can see here you can actually double bracket and do it all in one line. Up above we had to do it in the two lines; this is how the setup is for this particular module. And then we'll go ahead and do a final legit summary and just see what that looks like. And we can see here, here's our p scores; remember they're all supposed to be under 0.0, um, underneath the 0.05. These are all under 0.05. You can see up here are different statuses as far as how it ran; it still took six iterations to go through the data, the date and so forth.

And now that we have a solid model, uh, and that's if you remember correctly, let me flip back on over here, we've done our data modeling; we've created a solid model. And so now we need to actually run a prediction so that we can interpret the results and weigh in on that and see how good they are. And so for that, I'm going to go ahead and let's find it over here, y predict. So we have our model that we created; we want to go ahead and create our y predict, and it's based on our final object, and here's the magic word, predict or dot predict. And so this is where we're going to come in and actually get our predictions on here that our model comes up. And remember we're testing; x test is going in; there's our x test, and it's only the significant variables. So we've created our model; now we want to push in our test version that's never seen this data before, so this is all new. And then we're just going to take a random sample to see what it looks like; what do those predictions look like? In this random sample, we're just looking at 10 of these, and you can see here is the actual number; it says 0.08 gives us a 1, 0.07; this is our probability number, which is I would have to actually look to see whether higher or lower is better. I'm guessing the closer to zero the better it is, but I'm not completely sure on that; I have to actually look up on and see with this particular framework. But you can see the actual prediction, 1 0 0 0. So I guess this is a good loan, a bunch of defaults and some good loans; that's what it predicts it's going to come out as.

Now, just looking at the data, you know, it's confusing; this is you have different models; they have different setups; they have the predicted probability comes out on this and other models it doesn't. So you really need a tool to dig deeper into the setup to understand what this means. And so what we're going to do is we're going to take our predicted values, which you can see right here has an actual predicted probability, and then we're going to go ahead and insert on there from a sample. Let me just run this because it'll make a lot more sense. We have our actual value and our predicted value. So this is what we predicted, and this is the actual setup. If we just look at this right off the bat, we can see that we predicted zero and this was a one; these all came out correct; uh, number 332 came out correct; 917 was an actual; they were good on their loan; they predicted it's going to be a default. You start to see that there's some variance. So how do we measure that? How do we know whether our model was any good or not? Well, for that, we're going to go ahead and load up a nice graph, and we'll use for our graph, we'll import matplot library pie plot and seaborn. Seaborn sits on matplot library, and this uh matplot library inline is just because I'm using Jupyter notebook; uh, with the new version I'm in it doesn't even need it, but I tend to leave it in for older versions. And then we roll import from our sklearn metrics; this is your scikit; scikit is the main package, and sk learn is how it looks on here. And we'll go ahead and run that so it's nicely loaded. And then we'll go ahead and create a little routine in just a moment here to print this out so we can see what that looks like.

And this metrics, it's kind of a fun metrics because it it tells you like how many you have correct on the correct one, how many you had wrong on the wrong one. And this top part is just all a draw routine, so it comes in here and has our actual and are predicted as our cm metrics; it breaks it up, puts it into the heat map; the sn heat map gives it some colors, add some labels, um, and then we go ahead and show it, and we're just going to dump that in there and run that. And you can spend forever building graphics; it's so much fun. We come in here; we have bad credit, good credit; uh, we have bad credit, good credit; we have predicted label. And so when you look at this, you can see here it had um on the good credit; it tells you how well it predicted it and how many false positive it has. And on our bad credit, um, here we have bad credit, bad credit; it said 30 of those were bad credit; it predicted them bad credit, while 21 were good credit that it predicted would be bad credit. And then we have here we have 188 that were good credit, uh, but the actual val, and then it had 61 there were bad credit that it said were good, were predicted good credit. Uh, so at this point, you'd look at this and you'd say, well, if I'm predicting uh these are this is our good credit here; let me flip this on over here, and this is our bad credit; there we go. So we have bad credit, good credit; you start looking at and you say, okay, can we afford that many defaults after five years or whatever it is? And so those are the questions you start asking; is does this model work good enough to predict whether we should give these people loans or not? You're always going to have somebody default on them, and again that starts going into domain banking domain, something I don't know very much about, clearly. I know whether I default that alone or not, which I don't, and I try very hard not to.

And then if we're going to draw the chart, which is really nice to have, just a clear visual chart; metrics also has our classification report, and you can see these are just the numbers we are looking at; they let us know the precision, uh, our f score and recall and all that stuff on here; you can just print a straight out metrics classification report. Now we looked at a regression model; let's go ahead and look at a clustering model to see what that looks like. And we'll go ahead and create a new group Python on there, and this is going to be our clustering setup. Let me flip this off and switch on over to my clustering menu. In this case, we're going to be looking at k-means clustering, so let's just put that up here: k-means clustering. Now the last one was a regression model, so it figured out coefficients for lines; you can go back to your old Euclidean geometry or the algebra expression for a slope on a line, y equals mx plus b or whatever, and it goes on from there as far as finding distances and figuring out the lines on there. This is a little bit different; we talk about clustering; it uses similar ideas, but it has to do with edges, two points, and how do you figure out a distance between two points, and what does that mean? Now you'll see here we have our pandas pd, our numpy as np, our matplot library; seaborn sits on matplot libraries; we talked about; they've added some parameters here as far as colors, and then we did an inline for the matplot library again; that's not really necessary even in the new version of the Jupyter notebook; in the old version Jupyter notebook, if you didn't do that, it would just kind of hang or disappear. And then we're going to import some data; it's important to have our data set up in here. And instead of looking at this and guessing what the data is, usually open up the csv file at this time; let's just go ahead and print it under the df head because really those those text files are so messy, and you look at the df head, and it's easy to look at. Whoops, had a path issue there. So college data, it's a csv, so automatically knows there's a dot csv on there, and we can see here that they downloaded from a number of colleges, um, Abilene Christian University, Adelphi University, Adrian College, and then it has whether it's private, I'm guessing yes or no; apps, I'm not sure what that is; acceptance rate, applications I'm betting, and then the acceptance rate, enrollment, top ten percent, top two five percent, undergrad, so on. So these are just statistics based on the college, their college, getting into a college.

And just like we did before, let's explore the data. If you can remember from the chart we were looking at, we're going through our data discovery, data preparation, uh, and exploratory data analysis. So we're really kind of looking at the data, and what do we have here? What came in at this point? The data has already been prepared; we are not doing a whole lot as far as getting rid of null values and finding out whether it's bad data or good data and that kind of thing. And data discovery would be figuring out which data to send so that we've already done. So we're going to do the exploratory data, jump into data modeling, and interpret the results just like we did with the other one. And we'll go ahead and use the panda tool to get the info df. You'll see df very commonly; it just stands for data frame; a lot of people use it as a generic setup when they import from pandas. And so we have our info, and you can see right here numbers, here's our count, so each one of these has 777 non-null values. And again, here's our type of data we're dealing with. And if you remember from last time, here's an object, um, this was our yes or no; everything else looks like it's integer; we do have a float value here, uh, integer, integer, floats and integers; usually they're when you're running them on a prediction; they're pretty interchangeable in a lot of cases, but whether it's private, true or false, that's pretty important on here; the fact that they could probably be switched to a zero or one is pretty helpful also. And if we're going to look at all that data, we can go ahead and describe it. So there's our data frame describe, and you have the same thing; here's our count; all these should be 777; we have the mean value, so you can actually look at what our average is; standard deviation, minimum, different quartiles, 25, 50, the third quartile, that so on, and of course our max value on there, min and max.

And as we explore the data, we can do a lot of fun things on here. Um, we're going to use our seaborn, and we're going to look at room and board versus graduation date; we're going to add; this is going to be a heat map, so we're going to add; whether it's a private college or not, how it fits the size, the aspect ratio and all that; that's what this line is here; seaborn; you can almost do your own class on matplot library in seaborn because there's so many features in there. But when you look at the very basics, you can throw up some data and have a nice colorful map set up. And here we have our graduation rate compared to room and board costs and in private or public. So you can see that the graduation rate compared to room and board, hit play; interesting thought; how much did I spend on room and board; am I going to graduate? The more I spend on room and board, and you can see numbers go up there on graduation rate, and you can see that the graduation rate tends to go way up for your private schools, and the room and board also goes continues to grow on private schools, kind of thing; it's kind of just a fun heat map. And we can explore other things in our doing our heat maps; we can look at out state, whether it's out of state, undergrad, private again is one of our setups on here. And you can see here that for undergrad there certainly is a lot more public schools, and whether it's out of state, interestingly enough, the out-of-state number goes way up; how many people coming from out of state into the school for the private schools; just kind of fun things. If you're trying to figure out where you're going to go, maybe you'd want to do this for picking a bachelor's degree or a master's degree or something, or certainly you could also plot this with fun locations and download information on tourism sites if you're traveling; that kind of thing; you do the same kind of thing with, and we'll go ahead and throw out another map on here. Let me go and run this; this is a slightly different setup on here with the outer state, and so we're running; let's see; pull this up; uh, bins 20; the bins is each how many bins are you going to split it into? And this is kind of a fun one where you can see private versus public; we have how many out of state, and I'd have to look this up to see what they picked on here; oh, there we go. Okay, so you can see we have another chart that has to do with the out-of-state and the number of students coming in and private and public, uh, so a lot of different setups on here as we dig into the different graphing.

So at this point, we've just been kind of looking at the graphs on here; let's go ahead and import our cluster, our k-means cluster, and you can see here we have; we're going to call it km, I guess for k means, or maybe it's k means model; I always label mine with model equals k means in clusters equals 2, random state equals 90. Uh, and so we're when we cluster it, we're only going to cluster into two groups; that's what this this is meaning here, um, across the different settings on there. And then we're going to go ahead and fit, and we're going to drop private, so we're not going to be looking at whether it's a private or public school; we're just going to go ahead and run the cluster on there. And then once we run the cluster on there, we can actually look at cluster centers; let's go ahead and do that on this; we talk about k means; let me run this; boom; there we go. So we have all these different columns, and each of those columns has a center, and we start looking how the data comes together on those centers; that's what all these numbers, this is the hidden stuff behind the clusters on here. And so we can now do; maybe we'll create a little converter here; if private equals yes, return one; else returns zero; we're going to run our cluster private and apply the converter, and then we'll go ahead and print that out; let me just show you what that looks like. So you can see what we're doing here. And then we go ahead and do uh df cluster; we'll create a new column on here, private apply converter, and we can see here it creates some private apps; we've just added on to our data frame on here, and we did that so that we can run the confusion matrix; make sure we import our confusion matrix; we're going to do a classification report, confusion matrix, and we'll go ahead and run this on our clustering and use; you can see here as we're predictions; here's our confusion matrix up here, just like we had on the other setup where it has how many false, true, false, false, uh, true, false and false, false as your accuracy and are set up on there. This is really uh if you're looking at which where you want to go; maybe you're looking at where you want to go in a, I think more of a tourism thing, but um for colleges, this is a great way for looking for a bachelor's or a master's coming in here, and you can start finding out which one you want to go to based on this accuracy, whether it's private or public or how many graduations are on there.

So we looked at a couple of examples, went through some of the parts of the cycle, whether using a k-means or linear regression model, and certainly this cycle and these models you start to see that there are some similarities; we're doing the same thing, and each model has a better setup depending on what you're trying to accomplish on there. Reinforcement learning by Simply Learn; my name is Richard Kirschner with the Simply Learn team; that's www.simplylearn.com; get certified, get ahead. We're going to cover reinforcement learning today, and what's in it for you? We'll start with why reinforcement learning; we'll look at what is reinforcement learning; we'll see what the different kinds of learning strategies are that are being used today in computer models under supervised versus unsupervised versus reinforcement; we'll cover important terms specific to reinforcement learning; we'll talk about Markov's decision process, and we'll take a look at a reinforcement learning example, but we'll teach a tic-tac-toe how to play.

Why reinforcement learning? Training a machine learning model requires a lot of data, which might not always be available to us. Further, the data provided might not be reliable; learning from a small subset of actions will not help expand the vast realm of solutions that may work for a particular problem. You can see here we have the robot learning to walk; very complicated setup when you're learning how to walk, and you'll start asking questions like if I'm taking one step forward and left, what happens if I pick up a 50-pound object? How does that change how a robot would walk? These things are very difficult to program because there's no actual information on it until it's actually tried out. Learning from a small subset of actions will not help expand the vast realm of solutions that may work for a particular problem. And we'll see here, learned how to walk; this is going to slow the growth that technology is capable of; machines need to learn to perform actions by themselves and not just learn off humans. And you see the objective, climb a mountain; a real interesting point here is that as human beings we can go into a very unknown environment, and we can adjust for it and kind of explore and play with it; most of the models, the non-reinforcement models in computer machine learning aren't able to do that very well; there's a couple of them that can be used or integrated; see how it goes is what we're talking about with reinforcement learning.

So what is reinforcement learning? Reinforcement learning is a sub-branch of machine learning that trains a model to return an optimum solution for a problem by taking a sequence of decisions by itself. Consider a robot learning to go from one place to another; the robot is given a scenario, must arrive at a solution by itself; the robot can take different paths to reach the destination; it will know the best path by the time taken on each path; it might even come up with a unique solution all by itself. And that's really important is we're looking for unique solutions; we want the best solution, but you can't find it unless you try it. So we're looking at our different systems, our different models; we have supervised versus unsupervised versus reinforcement learning. And with the supervised learning, that is probably the most controlled environment; we have a

Lot of different supervised learning models, whether it's linear regression, neural networks; there's all kinds of things in between, decision trees. The data provided is labeled data with output values specified, and this is important because we talk about supervised learning. You already know the answer for all this information; you already know the picture has a motorcycle in it, so you're supervised learning. You already know that um the outcome for tomorrow, for you know, going back a week, you're looking at stock; you can already have like the graph of what the next day looks like, so you have an answer for it, and you have labeled data which is used; you have an external supervision and solves problems by mapping labeled input to known output—so very controlled.

Unsupervised learning, and unsupervised learning is really interesting because it's now taking part in many other models they start with, and you can actually insert an unsupervised learning model in almost either supervised or reinforcement learning as part of the system, which is really cool. Data provided is unlabeled data; the outputs are not specified. The machine makes its own predictions, used to solve association with clustering problems. Unlabeled data is used; no supervision. Solves problems by understanding patterns and discovering output, so you can look at this and you can think some of these things go with each other; they belong together. So it's looking for what connects in different ways, and there's a lot of different algorithms that look at this. When you start getting into those, are some really cool images that come up of what unsupervised learning is; how we can pick out, say, the area of a donut. One model will see the area of the donut, and the other one will divide it into three sections based on its location versus what's next to it, so there's a lot of stuff that goes in with unsupervised learning.

And then we're looking at reinforcement learning—probably the biggest industry in today's market, uh, in machine learning, or growing market; it's very, it's very infant stage as far as how it works and what it's going to be capable of. The machine learns from its environment using rewards and errors, used to solve reward-based problems. No predefined data is used; no supervision. Follows trail and error problem-solving approach. So again, we have a random; first you start with a random, I try this; it works, and this is my reward; doesn't work very well maybe, or maybe doesn't even get you where you're trying to get it to do, and you get your reward back, and then it looks at that and says, well, let's try something else, and it starts to play with these different things, finding the best route. So let's take a look at important terms in today's reinforcement model, and this has become pretty standardized over the last few years, so these are really good to know. We have the agent; agent is the model that is being trained via reinforcement learning, so this is your actual entity that has however you're doing it, whether using a neural network or uh cue table or whatever combination thereof; this is the actual agent that you're using; this is the model. And you have your environment; the training situation that the model must optimize to is called its environment. And you can see here, I guess we have a robot who's trying to get a chest full of gyms or whatever, and that's the output. And then you have your action; this is all possible steps that can be taken by the model, and it picks one action, and you can see here it's picked three different uh routes to get to the chest of diamonds and gyms. We have a state; the current position, condition returned by the model. And you could look at this; if you're playing like a video game, this is the screen you're looking at. So when you go back here, the environment is a whole game board; so if you're playing one of those mobius games, you might have the whole game board going on, but then you have your current position; where are you on that game board? What's around that? What's around you? If you were talking about a robot, the environment might be moving around the yard; where it is in the yard and what it can see; what input it has in that location; that would be the current position, condition returned by the model. And then the reward; uh, to help the model move in the right direction, it is rewarded; points are given to it to appraise some kind of action. So yeah, you did good or didn't do as good; trying to maximize the reward and have the best reward possible. And then policy; policy determines how an agent will behave at any time; it acts as a mapping between action and present state; this is part of the model; what, what is your action that you're, you're going to take? What's the policy you're using to have an output from your agent? One of the reasons they separate uh policy as its own entity is that you usually have a prediction of different options, and then the policy—well, how am I going to pick the best based on those predictions? I'm going to guess at different options, and we'll actually weigh those options in and find the best option we think will work, so it's a little tricky, but the policy thing is actually pretty cool how it works.

Let's go and take a look at a reinforcement learning example, and just in looking at this, we're going to take a look; consider what a dog that we want to train. So the dog would be like the agent; so you have your, your puppy or whatever, and then your environment is going to be the whole house or whatever it is or where you're training them, and then you have an action; we want to teach the dog to fetch; so action equals fetching, and then we have a little biscuit. So we can get the dog to perform various actions by offering incentives such as a dog biscuit as a reward; the dog will follow a policy to maximize this reward and hence will follow every command and might even learn new actions like begging by itself. So you have, you know, so we start off with fetching; it goes, oh, I get a biscuit for that; it tries something else; you get a handshake or begging or something like that, and it goes, oh, this is also reward based, and so it kind of explores things to find out what will bring is biscuit, and that's very much like how reinforced model goes; is it looks for different rewards; how do I find; can I try different things and find a reward that works? The dog also will want to run around and play in explorers environment; this quality of model is called exploration, so there's a little randomness going on in exploration, and explores new parts of the house; climbing on the sofa doesn't get a reward; in fact, it usually gets kicked off the sofa.

So let's talk a little bit about Markov's decision process. Markov's decision process is a reinforcement learning policy used to map a current state to an action where the agent continuously interacts with the environment to produce new solutions and receive rewards, and you'll see here's all of our different uh vocabulary we just went over; we have a reward, our state, our agent, our environment interaction, and so even though the environment kind of contains everything that you really when you're actually writing the program, your environment is going to put out a reward in state that goes into the agent; the agent then looks at this state or it looks at the reward usually um first, and it says, okay, I got rewarded for whatever I just did, or it didn't get rewarded, and it looks at the state, and then it comes back, and if you remember from policy, the policy comes in um, and then we have a reward; the policy is that part that's connected at the bottom, and so it looks at that policy and it says, hey, what's a good action that will probably be similar to what I did, or sometimes are completely random, but what's a good action that's going to bring me a different reward? So taking the time to just understand these different pieces as they go is pretty important in most of the models today, and so a lot of them actually have templates based on this; you can pull in and start using pretty straightforward as far as once you start seeing how it works; you can see your environment sends it says, hey, this is the agent did this; if you're a character in the game, this happened, and it shoots out a reward in a state; the agent looks at the reward, looks at the new state and then takes a little guess and says, I'm going to try this action, and then that action goes back into the environment; it affects the environment; the environment then changes depending on what the action was, and then it has a new state and a new reward that goes back to the agent. So in the diagram shown, we need to find the shortest path between node A and D; each path has a reward associated with it, and the path with a maximum reward is what we want to choose; the nodes A, B, C, D denote the nodes to travel; from node A to B is an action; reward is the cost of each path, and policy is each path taken, and you can see here A can go to B or A can go to C right off the bat, or can go right to D, and if explored all three of these, you would find that A going to D was a zero reward; A going to C and D would generate a different reward, or you could go A, C, B, D; there's a lot of options here, and so when we start looking at this diagram, you start to realize that even though today's reinforced learning models do really good at finding an answer, they end up trying almost all the different directions you see, and so they take up a lot of work uh, or a lot of processing time for reinforcement learning; they're right now in their infant stage, and they're really good at solving simple problems, and we'll take a look at one of those in just a minute in the tic-tac-toe game, but you can see here once it's gone through these and it's explored, it's going to find that A, C, D is the best reward; he gets a full 30 points for it.

So let's go ahead and take a look at a reinforcement learning demo. In this demo, we're going to use reinforcement learning to make a tic-tac-toe game; you'll be playing this game against the machine learning model, and we'll go ahead; we're doing it in Python, so let's go ahead and go through my; always not always; I actually have a lot of Python tools; let's go through Anaconda, which will open up a Jupyter Notebook. It seems like a lot of steps, but it's worth it to keep all my stuff separate, and it's also has a nice display when you're in the Jupyter Notebook for doing Python. So here's our Anaconda Navigator; I open up the notebook, which is going to take me to a webpage, and I've gone in here and created a new Python folder; in this case, I've already done it and enabled it to change the name to tic-tac-toe, and then for this example, we're going to go ahead and import a couple things; we're going to import numpy as np; we'll go ahead and import pickle; numpy, of course, is our number array, and then pickle is just a nice way sometimes for storing different information uh, different states that we're going to go through on here, and so we're going to create a class called state; we're going to start with that, and there's a lot of lines of code to this class that we're going to put in here; don't let that scare you too much; there's not as much here; it looks like there's going to be a lie here, but there really is just a lot of setup going on in the in our class state, and so we have up here we're going to initialize it um, we have our board; it's a tic-tac-toe board, so we're only dealing with nine spots on the board uh, we have player one, player two uh, is end; we're gonna create a board hash uh, we'll look at that in just a minute; we're just gonna store some information in there; symbol player equals one; so there's a few things going on as far as the initialization; then something simple; we're just going to get the hash of the board; we get the information from the board on there which is columns and rows; we want to know when a winner occurs; so if you get three in a row, that's what this whole section here is for uh, let me go ahead and scroll up a little bit, and you can get a copy of this code if you send a note over to simply learn; we'll send you over this particular file, and you can play with it yourself and see how it's put together; I don't want to spend a huge amount of time on this because this is just some real general Python coding uh, but you can see here we're just going through all the rows, and you add them together, and if it equals three; three in a row; same thing with columns uh, diagonal; so you gotta check the diagonal; that's what all this stuff does here is it just goes through the different areas; actually let me go ahead and put there we go um, and then it comes down here we do our sum, and it says true minus three; just says did somebody win or is it a tie? So you got to add up all the numbers on there anyway just in case they're all filled up, and next we also need to know available positions; these are ones that don't no one's ever used before; this way when you try something or the computer tries something, it's not going to give it an illegal move; that's what the available positions is doing; then we want to update our state, and so you have your position going in; we're just sending in the position that you just chose, and you'll see there's a little user interface we put in there; we pick the row and column in there, and again I mean this is a lot of code uh, so really it's kind of a thing you'd want to go through and play with a little bit and just read through it; get a copy of it; a great way to understand how this works, and here is a given reward; so we're going to give a reward; result equals self.winner; this is one of the hearts of what's going on here uh, is we have a result self.winner; so if there's a winner, then we have a result that the result equals one; here's our feedback uh, if it doesn't equal one, then it gets a zero; so it only gets a reward in this particular case if it wins, and that's important to know because different systems of reinforced learning do rewarding a lot differently depending on what you're trying to do; this is a very simple example with a three by three board; imagine if you're playing a video game uh, certainly you only have so many actions, but your environment is huge; you have a lot going on in the environment, and suddenly a reward system like this is going to be just; it's going to have to change a little bit; it's going to have to have different rewards and different setup, and there's all kinds of advanced ways to do that as far as weighing; you add weights to it, and so they can add the weights up depending on where the reward comes in; so it might be that you actually get a reward; in this case, you get the reward at the end of the game, and I'm spending just a little bit of time on this; this is an important thing to note, but there's different ways to add up those rewards; it might have like if you take a certain path um, the first reward is going to be weighed a little bit less than the last reward because the last reward is actually winning the game or scoring or whatever it is; so this reward system gets really complicated in some of the more advanced uh setups; in this case though, you can see right here that they give a 0.1 and a 0.5 reward just for getting picking the right value and something that's actually valid instead of picking an invalid value; so rewards again that's like key; it's huge; how do you feed the rewards back in? Then we have a board reset; that's pretty straightforward; it just goes back and resets the board to the beginning because it's going to try out all these different things; while it's learning, it's going to do it by trial and error; so you have to keep resetting it, and then of course there's the play; we want to go ahead and play; rounds equals 100; depends on what you want to do on here; you can set this different; you can obviously set that to higher level, but this is just going to go through, and you'll see in here that we have player 1 and player 2. This is this is the computer playing itself; one of the more powerful ways to learn to play a game or even learn something that isn't a game is to have two of these models that are basically trying to beat each other, and so they always; they keep finding; explore new things; this one works for this one; so this one tries new things; it beats this; we've seen this in chess; I think was the big one where they had the two players in chess with reinforcement learning; it was one of the ways they trained one of the top computer chess playing algorithms; so this is just what this is; it's going to choose an action; it's going to try something, and the more it try stuff, the more we're going to record the hash; we actually have a board hash where they self get the hash set up on here where it stores all the information, and then once you get to a win; one of them wins; it gets the reward uh, then we go back and reset and try again, and then kind of the fun part; we actually get down here is uh, we're going to play with a human; so we'll get a chance to come in here and see what that looks like when you put your own information in, and then it just comes in here and does the same thing it did above; it gives it a reward for its things or sees if it wins or ties; looks at available positions; all that fun of fun stuff; and then finally we want to show the board; so it's going to print the board out each time; really as an integration is not that exciting; what's exciting in here is one looking at this reward system; whoops; play one more up; the reward system is really the heart of this; how do you reward the different setup, and the other one is when it's playing; it's got to take an action, and so what it chooses for an action is also the heart of reinforcement learning; how do we choose that action, and those are really key to right now where reinforcement learning is in today's uh technology is uh figuring this out; how do we reward it and how do we guess the next best action? So we have our uh environment, and you can see the environment is we're gonna be or the state which is kind of like what's going on; we're going to return the state depending on what happens; we want to go ahead and create our agent; in this case our player; so each one is going to grab that until we look at a class player um, this is where a lot of the magic is really going on is what how is this player figuring out how to maneuver around the board, and then the board of course returns a state that it can look at and reward uh, so we want to take a look at this; we have uh name self state; this is class player; when you say class player, we're not talking about a human player; we're talking about um just the computer players, and this is kind of interesting; so remember I told you depending on what you're doing; there's going to be a decay gamma explore rate; these are what I'm talking about is how do we train it; as you try different moves; it gets to the end; the first move is important, but it's not as important as the last one, and so you could say that the last one has the heaviest weight, and then as you as you get there; the first one; let's see; the first move gives you a five reward; the second gives you a two reward, and the third one gives you a 10 reward because that's the final ending; you got it; the 10's gonna count more than the first step uh, and here's our; we're going to get the board information coming in, and then choose an action; this was the second part that I was talking about that was so important; so once you have your training going on; we have to do a little randomness, and you can see right here is our np.random.uniform

So, it's picking out a random number, take a random action. This is going to just pick which row and which column it is, um, and so choosing the action. This one you can see we're just doing random states, um, choice, length of positions, action position, and then it skips in there and takes a look at the board. Uh, for p in positions, it's actually storing the different boards each time you go through, so it has a record of what it did so it can properly weigh the values. And this simply just depends a hash state; what's the last date? Append it to the to our states on here. Here's our feedback; rewards. The reward comes in, and it's going to take a look at this and say, is it none? What is the reward? And here is that formula. Remember I was telling you about up here that was important because it has decay, gamma times a reward. This is where, as it goes through each step, this, and this is really important; this is this is kind of the heart of this of what I was talking about earlier. You have step one, and this might have a reward of two. You have step two; I should probably should have done abc; this has a step three, uh, step four, and so on until you get to step n, and this might have a reward of 10. Uh, so reward a 10, we're going to add that, but we're not adding, uh, let's say this one right here, let's see this reward here right before 10 was, let's say it's also 10; it just makes the math easy. So we had 10 and 10. We had 10; this is 10 and 10 in whatever it is, but it's time; it's 0.9. So instead of putting a full 10 here, we only do nine; that's uh 0.9 times 10. And so this formula, as far as the decay times the reward minus the cell state value, uh, it basically adds in; it says here's one or here's two; I'm sorry, I should have done this; abc would have been easier. So the first move goes in here and puts two in here; then we have our self set up on here. You can see how this gets pretty complicated in the math, but this is really the key: is how do we train our states? And we want the the final state, the win, to get the most points. If you win, you get most points, uh, and the first step gets the least amount of points. So you're really training this almost in reverse; you're training you're training it from the last place where you have like it says, okay, this is now I wear need to sum up my rewards, and I want to sum them up going in reverse, and I want to find the answer in reverse; kind of an interesting uh play on the mind when you're trying to figure this stuff out. And of course, we want to go ahead and reset the board down here, uh, save the policy, load policy; these are the different things that are going in between the agent and the state to figure out what's going on. Let's go ahead and load that up, and then finally, we want to go ahead and create a human player, and the human player is going to be a little different in that you choose an action, row and column. Here's your action, uh, if action is if action in positions, meaning positions that are available, uh, you return the action; if not, it just keeps asking you until you get the action that actually works. And then we're going to go ahead and append to the hash state, which we don't need to worry about because it returns the action up here, and feed forward again; this is because it's a human. At the end of the game, back propagate and update state values; this part isn't being done because it's not programming uh the model; the model is getting its own rewards. So we've gone ahead and loaded this in here, so here's all our pieces, and the first thing we want to do is set up p1, player 1, p2, player 2, and then we're going to send our players to our state, so now it has p1, p2, and it's going to play, and it's going to play 50,000 rounds. Now we can probably do a lot less than this, and it's not going to get the full results; in fact, you know what, let's go ahead and just do five just to play with it because I want to show you something here. Oops, somewhere in there I forgot to load something. There we go. I must have forgot to run this. Run. Oops, forgot a reference there for the board rows and columns, three by three; there is actually in the state it references that we just tack it on on the end; it was supposed to be at the beginning, uh, so now I've only set this up with, um, let's see where we go in here; I've only set this up to train five times, and the reason I did that is we're going to come in and actually play it, and then I'm going to change that, and we can see how it differs on there. There we go. And then you make it through a run, and we're going to go ahead and save the policy. So now we have our player one and our player two policy; uh, the way we set it up, it has two separate policies loaded up in there, and then we're gonna come in here, and we're gonna do, uh, player one is gonna be the computer, experience rate zero, load policy one, human player, human, and we're going to go ahead and play this. I remember only went through it just one round of training; in fact, minimal training, and so it puts an x there, and I'm going to go ahead and do row 0, column 1. You can see this is very basic on here, and so I put in my 0, and then I'm going to go 0, block it 0, 0, and you can see right here it let me win, uh, just like that; I was able to win, zero, two, and whoo, human wins. So I only trained it five times. We're going to run this again, and this time instead of five, let's do five thousand or fifty thousand; I think that's what the guys in the back had, and this takes a while to train it. This is where reinforcement learning really falls apart. Look how simple this game is; we're talking about a three by three set of columns, and so for me to train it on this, I could do a q table, which would take which would go much quicker; you could build a quick q table with almost all the different options on there, and you would probably get a the same result much quicker. We're just using this as an example, so when we look at reinforcement learning, you need to be very careful what you apply it to. It sounds like a good deal until you do like a large neural network where you're doing you set the neural network to a learning increment of one, so every time it goes through it learns, and then you do your actions, you pick from the learning setup, and you actually try actions on the learning setup until you get what you think is going to be the best action, so you actually feed what you think is right back through the neural network; there's a whole layer there which is really fun to play with, and then it has an output. Well, think of all those processes; I mean, that is just a huge amount of work it's going to do. Let's go ahead and skip ahead here; give it a moment; it's going to take a minute or two to go ahead and run. Now to train it, we went ahead and let it run, and it took a while; this just took, um, I got a pretty powerful processor, and it took about five minutes plus to run it, and we'll go ahead and run our player setup on here. Oops, brought in the last whoops, abroad in the last round, so give me just a moment to redo the policy save. There we go. I forgot to save the policy back in there, and then go ahead and run our player again. So we've saved the policy, then we want to go ahead and load the policy for p1 as the computer, and we can see the computer's gone in the bottom right corner. I'm going to go ahead and go 1-1, which is the center, and it's gone right up the top, and if you ever played tic-tac-toe, you know the computer has me, uh, but we'll go ahead and play it out, row zero, column two. There it is, and then it's gone here, and so I'm gonna go ahead and go row zero, one, two, no, zero one. There we go, and column zero; that's where I want it. Oh, and it says okay, you your action. There we go. Boom, uh, so you can see here we've got a didn't catch the win on this; it said tie, kind of funny that didn't catch the win on there, but if we play this a bunch of times, you'll find it's going to win more and more; the more we train it, the more the reinforcement happens. This lengthy training process is really the stopper on reinforcement learning. As this changes, reinforcement learning will be one of the more powerful packages evolving over the next decade or two; in fact, I would even go as far as to say it is the most important machine learning tool and artificial intelligence tool out there, as it learns not only a simple tic-tac-toe board, but we start learning environments, and the environment would be like in language; if you're translating a language or something from one language to the other, so much of it is lost if you don't know the context it's in, what the environments it's in, and so being able to attach environment and context and all those things together is going to require reinforcement learning to do so. Again, if you want to get a copy of the tic-tac-toe board, it's kind of fun to play with, run it; you can test it out; you can do, you know, test it for different uh values; you can switch from p1 computer where we loaded the policy one to load the policy 2 and just see how it varies; there's all kinds of things you can do on there. Welcome to Q-learning. My name is Richard Kirschner with the Simply Learn team, www.simplylearn.com. Get certified, get ahead. What's in it for you today? An introduction to reinforcement learning; what is Q-learning; some important terms; Bellman equation; steps in Q-learning; a quick recap. So we're going to go back and do a reinforcement learning recap, and you'll remember we have our agent, our action, our environment, and our reward, um, and the state, and so, uh, reinforcement learning is a branch of machine learning that trains a model to come to an optimum solution for a problem by taking decisions by itself. So we look at it, we analyze the data, make decisions, solve problems. It consists of an environment which an agent will interact with to learn to reach a goal or perform an action, and you see here we have the dog, and the action equals fetching, and the first thing you're going to notice here, and I'll talk a little bit more about this, is that our action is kind of a singularity; the action isn't a regression type model where we're looking at float numbers or we're trying to predict something or doing a reinforcement learning where we're trying to guess the value of the stock market where you have five point three three five dollars and thirty-three cents; this is very a little bit different, and you'll see why Q-learning is so powerful, but also where it falls short. The agent is also given reward if the action performed by it is bringing is closer to the goal, is leading to the goal. This is done to train the model in the right direction. We have a reward of a hopefully a doggy cookie, and action equals fetching, so gets a reward; it gets to go fetch the ball. Reinforcement learning can be divided based on whether the machine uses a model to learn or learns by itself, uh, so reinforcement learning; it's model-based, learn the model, given model, and you can think of neural networks; I think are the most common right now used for reinforcement learning; there certainly are a lot of other options out there, but you have this black box, and you go back and you send the error back up, and it trades it based on how much of an error; the error could be computed based on maybe a maximum reward subtraction or something like that, so the closer you get to the maximum reward, the better, and so we talk about a model-free; you have your policy optimization and your Q-learning, and this is kind of the cool thing about Q-learning is it can do; it's very powerful on certain things, making it very quick and easy to do a reinforcement learning. So what is Q-learning? Q-learning is reinforcement learning policy which will fill the next best action given a current state; it chooses this action at random and aims to maximize the reward, and so you can see here's our standard reinforcement learning graph; by now, if you're doing any reinforcement learning, you should be familiar with this where you have your agent, your agent takes an action, the action affects the environment, and then the environment sends back the reward or the feedback, and the state is the new state the agent's in; where is it at on the chessboard? Where is it at in the video game? If your robot's out there picking trash up off the side of the road, where is it at on the road? Consider an ad recommendation system; usually when you look up a product online, you get ads which will suggest the same product over and over again. Using Q-learning, we can make an ad recommendation system which will suggest related products to our previous purchase; the reward will be if user clicks on the suggested product, and again you can see you might have a lot of products on your web advertisement or your pages, but it's still not a float number; it's still a set number, and that's something to be aware of when you're using Q-learning, and you can see here that if you have a hundred people clicking on ads and you click on one of the ads, it might go in there and say, okay, this person clicked on this ad; what is the best set of ads based on clicking on this ad or these two ads afterwards based on where they are browsing? So let's go ahead and look at some important terms. We talk about Q-learning; we have states; the state s represents the current position of an agent in an environment; the action; the action a is the step taken by the agent when it is particular state; rewards; for every action the agent will get a positive or negative reward; and again, when we talk about states, we're usually not with when you're using a q table, you're not usually talking about float variables; you're talking about true, false, and we'll take a closer look at that in a second; and episodes; when an agent ends up in a terminating state and can't take a new action; this might be if you're playing a video game, your character stepped in and is now dead or whatever; Q-values; used to determine how good an action a taken at a particular state s is, Q(a of s); and temporal difference; a formula used to find the Q-value by using the value of the current state and action and previous state in action; and various; I mean, there's Bellman's equation, which basically is the equation that kind of covers what we just looked at in all those different terms; the Bellman equation is used to determine the values of a particular state and deduce how good it is to be in take that state; the optimist; the optimal state will give us the highest optimal value; factor influencing Q-values; the current state and action; that's your sa, so your current state and your action; uh, then you have your previous date in action, which is your s, I guess prime; I'm not sure how the how they reference that, s'a'; so this is what happened before; uh, then you have a reward for action, so you have your r reward, and you have your maximum expected future reward, and you can see there's also a learning rate put in there and a discount rate. So we're looking at these just like any other model; we don't want to have an absolute final value on here; we don't want it to; if you do absolute values instead of taking smaller steps, you don't really have that approach to the solution; you just have it jump, and then pretty soon if you jump one solution out, that's what's going to be the new solution, whichever one jumps up really high first, kind of ruining the whole idea of doing a random selection. I'll go into the random selection in just a second. Steps in Q-learning: Step one, create an initial Q-table with all values initialized to zero; again, we're looking at zero, one, so are you, you know, here's our action; we start; we're an idol; we took a wrong action; we took a correct action; and end; and then we have our actions, fetching, sitting, and running; of course, we're just using the dog example; and choose an action and perform it; update values in the table; and of course, when we're choosing an action, we're going to kind of do something random and just randomly pick one; so you start out, and you sit, and you have then then depending on that action you took, you can now update the value for sitting after you start from start to sitting; get the value of the reward and calculate the value; the value Q-value using the Bellman equation; and so now we attach a reward to sitting, and we attach all those rewards; we continue the same until the table is filled with or an episode ends; and my membership is going to come back to the random side of this, and there's a few different formulas I use for the random setup to pick it; I usually look whatever Q model I'm using, do their standard one, because someone's usually gone in and done the math for the optimal spread, but you can look at this; if I have running has a reward of 10, sitting as a reward of 7, fetching has a reward of 5, just kind of without doing like a means to using the bell curve for the means value, and like I said, there's some math you can put in there to pick, um, so that you're more like so that running has even a higher chance, but even if you're just going to do an average on this, you could do an average a random number by adding them all together, so you get 10 plus 7 plus 5 is 22; you could do 0 to 22 or 0 to 21, but 1 to 22, 1 to 5 would be fetching, uh, and so forth, you know, the last 10. So you can just look at this as what percentage are you going to go for that particular option, and then that gets your random setup in there, and then as you slowly increment these up, uh, you see that, uh, if you're idle, where's one, here we go, sitting at the end; if you're at the end or wherever you're at, sitting gets a reward of one; where's a good one on here? Oh, wrong action; running for a wrong action gets almost no reward, so that becomes very, very less likely to happen, but it still might happen; it still might have a percentage of coming up, and that's where the random programming and Q-learning comes in. The below table gives us an idea; action has been taken and how positively correct action or negatively wrong action it is going to affect the next state. So let's go ahead and dive in and pull up a little piece of code and see what this looks like [Music] in Python. In this demo, we'll use Q-learning to find the shortest path between two given points. If getting your learning started is half the battle, what if you could do that for free? Visit Skill Up by Simply Learn; click on the link in the description to know more. If you've seen my videos before, I like to do it in the anaconda Jupiter notebook setup just because it's really easy to see, and it's a nice demo, and so here's my anaconda; this one I'm actually using a Python 3.6 environment that I set up in here, and we'll go ahead and launch the Jupiter notebook on this, and once we're in our Jupiter notebook which has the kernel loaded with Python 3, we'll go ahead and create a new Python 3 folder in here, and we'll call this Q-learning, and to start this demo, let's go ahead and import our NumPy array; we'll just run that so it's imported, and like a lot of these model programs when you're building them, you spend a lot of time putting it all together, and then you end up with this really short answer at the end, uh, and we'll we'll take a look at that as

We come into it so we we go ahead and start with our location to state, uh, so we have uh, L1, L2. These are our nine locations, one to nine, and then, of course, the state is going to be 0, 1, 2, 3, 4. It's just a mapping of our location to an integer on there.

And then we have our actions. Our actions are simply moving from one location to another, so I can go to location zero, I can go to location one, two, three, four, five, six, seven, eight. Uh, so these are my actions I can choose. These are the locations of our state. And if you remember earlier, I mentioned that the limitation is that you you don't want to put in a continually growing table because you can actually create a dynamic queue table where you continually add in new values as they arise. Because if you have float values, this just becomes infinite, and then you're remembering your computer is gone or, you know, does it's not going to work at the same time. You might think, well, that kind of really limits the the Q learning setup, but there are ways to use it in conjunction with other systems.

And so you might look at, well, I do, um, I've been doing some work in stock, um, and one of the questions that comes out is to buy or sell the stock. And the stake coming in might be, um, you might take a concrete way called buckets, um, where anything that you predict is going to return more than a certain amount of money, the error for that stock that you've had in the past, you put those in buckets. And suddenly you start putting, creating these buckets, you realize you do have a limited amount of information coming in; you no longer have a float number; you now have um, bucket one, two, three, and four. And then you can take those buckets, put them through a Q learning table, and come up with the best action: which stock should I buy? It's like gambling; stock is pretty much gambling if you're doing day trading; you're not doing long-term investments. And so you can start looking at it like that. A lot of the current feeds say that the best algorithms used for day traders redoing it on your own is really to ask the question: do I want to trade the stock? Yes or no. And now you have it in a Q learning table, and now you can take it to that next level, and you can see where that can be a really powerful tool at the end of doing a basic linear regression model or something: what is the best investment? And you start getting the best reward on there. And and so if we're gonna have rewards, these rewards we just create, um, it says uh, if basically if you're uh, this should match our Q table because it's gonna be uh, you have your state and you have your action across the top if you remember from the dog. And so we have whatever state we're in going down and then the next action and what the reward is for it. And of course, if you were actually doing a something more connected, your reward would be based on the actual environment it's in.

And then we want to go ahead and create a state to location so we can map the indexes. So just like we defined our rewards, we're going to go and do state to location, and you can see here it's a dictionary set up for location, state, and location to state with items. And we also need to define what we want for learning rates. You remember we had our two different rates as far as like learning from the past and learning from the current, so we'll go ahead and set those to 0.75 and the alpha set to 0.9, and we'll see that when we do the formula. And of course, any of this code, send a note to our Simply Learn team; they'll get you a copy of this code on here. Let's go ahead and pull there we go on the next two sections since we're going to keep it short and sweet. Here we go.

So let's go ahead and create our agent. So our agent is going to have our initialization where we send it all the information. We'll define ourself gamma equals gamma; we could have just set the gamma rate down here instead of submitting it; it's kind of nice to keep them separate because you can play with these numbers. On our self alpha, then we have our location state; we'll set that in here; we have our choice of actions; we're going to go ahead and just embed the rewards right into the agent, so obviously this would be coming from somewhere else instead of from self-generated. And then a self state to location equals our state to location dictionary. And we go ahead and create a Q learning table, and I went ahead and just set the Q learning table up to zero to zero, what what what the setup is, location to state, how many of them are there, and this just creates an array of zero to zero setup on there.

And then the big part is the training. We have our rewards new equals a copy of self.rewards; ending state equals the self-location state in location, so this is whatever we end up at. Rewards new equals ending state plus ending state equals 999; just kind of goes to a dead end. And we start going through iterations, and we'll go ahead, um, let's do this uh, so this we're going to come back and we're going to call call it on here. Let me just erase that, switch it to an arrow there we go. So what we're doing is we're going to send in here to train it; we're going to say, hey, I want to iterate through this a thousand times and see what happens. Now this part would actually be instead of iterating, you might have your external environment and they're going back and forth, and you iterate through outside of here, but just for ease of use, our agent is going to come in here and iterate through this. Sometimes I'll put this iteration in here and I'll have it call the environment and say, hey, this is what I did, what's the next state? And the environment does this thing right in here as I iterate through it.

And then we want to go ahead and pick a random state to start with; that's what's going on here; you have to start somewhere. And then you have your playable actions; we're going to start with just an empty thing for playable actions, and we'll fill that up. So that's what choices I have. And so we're going to iterate through the rewards matrix to get the states directly reachable from the randomly chosen current state; assign those states to a list named playable actions. And so you can see here we have range nine; I usually use length of whatever I'm looking at, which is our locations or states as they are. We have a reward, so we want to look at the current the rewards; the new reward is our is in our chart here of rewards_new uh, current state um plus j, j being what is the next state we want to try. And so we go and do our playable actions, and we append j. And so we're doing is we're randomly trying different things in here to see what's going to generate a better reward. And then, of course, we go ahead and choose our next state. So we have our random choice playable actions. And if you remember I mentioned on this, let me just go ahead and let's just do a free form when we were talking about the next state; this right here just does a random selection instead of a random uh selection; you might do something where uh whatever the best selection is, which might be option three here, and then so you can see that it might use a bell curve and then option two over here might have a bell curve like this oops, and we start looking at these averages and these spreads, or we can just add them all together and pick the one that kind of goes in all of those. So those are some of the options we have in here; we just go with a random choice; that's usually where you start; play with it.

And then we have our reward section down here. And so we want to go ahead and find, well, in this case, the temporal difference. So you have your rewards new plus the self gamma, and this is the formula we were looking at; this is Bellman's equation here. So we have our current value, our learning rate, our discount rate involved in there, the reward system coming in for that, and we can add it all together. This is, of course, our maximum expected future setup in here, so this is all of our our Bellman's equation that we're looking at here. And then we come up in here and we update our Q table; that's all this is on this one; that's right here. We have self Q current state next date, and we add in our alpha because we don't want to we don't want to train all of it at once in case there's slight differences coming in there; we want to slowly approach the answer. And then we have our route equals the start location and next location equals start location, so we're just incrementing; we took a step forward. And then finally, remember I was telling you how we're going to do all this and just have some simple thing at the end or just generates a simple path; we're going to go ahead and and get the optimal route; we want to find the best route in here. And so we've created a definition for the optimal route down here; just scroll down for that. And we get the optimal route; we go ahead and put the information in, including the Q table, self start location in location, next location, route Q, and it says while next location is not equal to in location, so while we can still go, our start location equals self location to state start location, so we already have our best value for the start location; uh, the next state looks at the Q table and says, hey, what's uh the next one with the best value? And then the next location we go ahead and pull that in, and we just append it; that's what's going on down here. And then our start location equals the next location, and we just go through all the steps. And we'll go ahead and run this.

And now that we have our Q table, our Q agent loaded, we're going to go ahead and take our Q agent, load them up with our alpha gamma that we set up above along with the location, step, action, reward, state to location, and our goal is to plot a course between L9 and L1, and we're gonna go through a hundred a thousand iterations on here. And so when I run that, it runs pretty quick. Uh, why is this so fast? Um, if you've been running neural networks and you've been doing all these other models, you sit here and wait a long time. Well, we're a very small amount of data; these are all integers; these aren't float values; there's not a the math is not heavy on the on the processing end, and this is where Q tables are so powerful. If you have a small amount of information coming in, you very quickly get an answer off of this, even though we went through it a thousand times to train it. And you'll see here we have L9, 8, 5, 2, and 1, and that's based on our reward table we had set up on there, and this is the shortest path going between these different setups in here. And if you remember on our reward table, you can see that if you start here, you can go to here; there's places you can't go; that's how this reward table was set up, so I can only go to certain places, so kind of a little maze set up in there, and you can play with it; this is really fun set up to play with. And you can see how you can take this whole code, and you can, like I was saying earlier, you can embed it into another setup and model and predictions where you put things into buckets, and you're trying to guess the best investment, the best course of action, as long as you can take that course into action and reduce it down to a yes/no, or if you're using text, you can use a one-hot encoder, which word is next; there's all kinds of things you can do with a Q table depending on just how much information you're putting in there. So that wraps up our demo. In this demo, we've found the shortest distance between two paths based on whatever rules or state rewards we have to get from point A to point B and what available actions there are.

Hi everyone, welcome to this exciting session by SimplyLearn. Today we have a really interesting topic for you. In this video, we'll be analyzing the upcoming United States presidential election using Twitter sentiment analysis in Python. In the course of this session, we will understand in brief about the upcoming election, the candidates fighting the 2020 U.S. election, and throw light on the result of the 2016 New Year's election. We look at the election forecast by some of the popular news agencies as well as poll analysis websites such as 538. Before we begin, I want to make it clear that this video is solely for learning purposes. We at SimplyLearn do not lean towards any party in the United States or are biased towards any candidate. The analysis is based on what people have tweeted, and we will understand the mood and sentiment of the public. The main objective of this video is to help you understand how to extract tweets from Twitter handles using Python libraries, store it in a CSV file, and perform textual analysis. The results obtained are purely based on the data that we have collected. Let's begin.

The battle for the 2020 U.S. presidential election has begun, with the 59th United States presidential election scheduled to be held on the 3rd of November 2020. This will decide who will be in the White House for the next four years. All 435 seats in the United States House of Representatives, 35 of the 100 seats in the United States Senate, and the office of the President of the United States will be contested. Both the Republican Party and the Democratic Party have nominated their respective president and vice-presidential candidates. The incumbent President Donald Trump is the Republican nominee, while the 47th Vice President of the United States, Joe Biden, is the Democratic Party's nominee. The current Vice President, Mike Pence, is the Republican Party's vice-presidential nominee and is running for a second term in office. Senator Kamala Harris of California was chosen by Joe Biden as his running mate on 11th 11 August 2020.

Now let's see the results of the last U.S. presidential election in 2016. The Republican ticket of businessman Donald Trump and Indiana Governor Mike Pence had defeated the Democratic ticket of former Secretary of State Hillary Clinton and U.S. Senator from Virginia Tim Kaine. Donald Trump received a total of 304 electoral votes, while Hillary Clinton got 227 electoral votes. A significant number of central states in the United States were swept by the Republican Party. It was surprising to see that Donald Trump getting nearly 46.4 popular vote as compared to Hillary Clinton who got 48.5 votes. Donald Trump's victory came after crucial wins in the battleground streets of Florida, Ohio, Iowa, North Carolina, and Pennsylvania. Will Donald Trump be re-elected to the White House, or are we going to see a new President and Vice President of the United States? Is the big question. Let's see what our pollsters have to say and what has been their forecast for the upcoming election.

According to 538, Donald Trump has not gained ground since the final debate on the 22nd of October. At this moment, Joe Biden is favored to win the election, and he leads in both national and state polls. The Economist is analyzing polling, economic, and demographic data to predict the U.S. elections. Their model thinks Joe Biden is very likely to beat Donald Trump in the electoral college. As per their forecast, Joe Biden has a 96% chance of winning the electoral college and this 99% chance of Biden winning the most votes. The Democrats have a 75% chance of controlling the Senate, while the Republicans have around 25%. The Democrats are highly likely to keep their majority in the House of Representatives. Now, according to The Guardian, Joe Biden is leading Donald Trump in the national polls for the presidential election, but to keep in mind this does not guarantee the Democratic candidate Joe Biden's victory. Hillary Clinton also had a clear lead over Donald Trump in the polls for almost the entire 2016 campaign; she ended up losing the electoral college.

Now the BBC forecast tells that Biden is leading national presidential polls with 51%. He has hovered around 50% in recent months and has had a 10-point lead on occasions. So clearly, Joe Biden is set to be the next U.S. President. Let's wait and watch what happens on the 3rd of November. The voters of America will decide on the 3rd of November whether Donald Trump remains in the White House for another 4 years. Let's perform a Twitter sentiment analysis to understand the sentiments of people tweeting about Donald Trump and Joe Biden by tagging them in their tweets. We have our code implemented already; I'll run through each line of code and explain our analysis in detail. The objective is to check the polarity of each tweet to figure out if it's positive, negative, or neutral. We'll compare the tweets of Donald Trump and Joe Biden to see who people are favoring the most. Finally, we learn to create a word cloud. I have already extracted the tweets and stored them in two separate CSV files. It takes some time to fetch 10,000 tweets; hence it won't be possible for me to show it to you live. If you want to know how we extracted the tweets for Donald Trump and Joe Biden, then please put your email IDs in the comments section of the video; we'll share the code file and the source file with you. Now you would need to install the QP library, fetch the Twitter keys and access tokens, and then authenticate the Twitter API. I have extracted 5,000 tweets each for Donald Trump and Joe Biden, converted into a data frame, and stored it in a CSV file. So let me open and show you both the files. So here is my data set for Donald Trump. On the top, you can see trump_data_5000, which means there are 5,000 tweets here. The first column has the number of tweets; it starts from 0 and goes all the way to 4999. The next column is the text column, which has the content of the tweet. You can see they have tagged @realDonaldTrump; there are links as well; there are special characters like emoticons. And the third column is the user, which means it's the username who had put that tweet. Similarly, I have another data for Biden. You can see these are for Joe Biden, and here you can see the content; they have tagged Kamala Harris also. Similarly, we have the username. Now let me take you straight to my Jupyter Notebook where I have implemented the code.

Okay, so first and foremost, let's import all the required libraries. So we have the NumPy library for numerical computation; then we have the Pandas library for data manipulation; next we are importing two libraries, Matplotlib and Seaborn, for data visualization; then we have TextBlob and WordCloud to get the sentiment and to build the word cloud; and finally, we are importing the Plotly library for creating graphs. Let me run this cell. In the current cell, will import the data sets using read_csv function present in the Pandas library. So I have passed in my location where the data file is, so I placed it on my desktop, and the index_col parameter or column parameter refers to the columns to be used as the row labels of the data frame. Here 0 indicates that the first column will be used as row labels. Let me run it. Okay, now we have imported both the data files. Let me go ahead and print the shape of one of the data frames. So I'll write print; let's say we'll check the shape of Biden data frame, so I'll write print biden.shape; I'll hit Shift+Enter. You can see there are 5,000 rows and there are two columns; one is the text column, and the next column is the user column. Now let me go ahead and print the head of the data sets. So the head function will return the first five rows. I'll write trump.head. You can see these are the first five rows from the Trump data frame. And similarly, if you want to check for Biden, I can write biden.head. You can see it here. Now you can also use the info method. So let me show you that; you can write trump.info, and this is the result. So it tells there are 5,000 entries from 0 to 4999; there are two columns, and these are the number of rows, and similarly, you have the data types. Okay, moving ahead in the next cell.

i am displaying the 200th tweet present in the trump data frame. So, I have my data frame name mentioned, then I have my text column, and inside one more square bracket, I have put 200, which means I want to see what's there in the 200th row. If I hit enter, so you can see this was the tweet: fake fake precedent.

Next, let me show you the tweet number 1000 for Biden. So, I'll write biden and inside square brackets I'll give the text column, and the number is 1000. If I hit shift enter, so this is the tweet.

Now, the text blob function is used in natural language processing. The sentiment property returns a named tuple in the form of sentiment[polarity, subjectivity]. Let's check the sentiment of the 200th tweet made for Donald Trump. If I hit shift enter, you can see this was the polarity. Now, -0.399 means the polarity was negative, and you can see the subjectivity is around 0.933. Similarly, let's check for Biden's tweet number 1000. So, I'll write text blob, which is my function name, and inside brackets I'll give biden, then pass in my column name that is text, and I'll give the row number, which is 1000, and then I'll give the parameter or the attribute sentiment. If I hit shift enter, you can see the polarity and the subjectivity.

Now moving ahead, here I have defined a function that will add another column called polarity to our two data frames, trump and biden. Let me run this. Okay.

In the next cell, I have used the apply function that will return the polarity based on the tweets. You can see we have the apply function. This might take some time to run it. Okay, we are done. Now let me now display the top five rows of the data frame. So what I'll do is I'll click on this cell, go to insert, and I'll hit insert cell below. Let's do one more. Okay.

Now here I'll display the head of the data frame once again so that you can see the polarity column added. You can see it here. So for Donald Trump, we have the polarity. You can see the tweet and the polarity. Now, 0.0, which means these are neutral; minus is negative; and anything that's in positive value indicates it's a positive tweet or it has positive polarity. Similarly, let's see for Biden. I'll write biden.head. Okay. So here you can see the polarity for Biden. Now, this 0.5 means it's a positive tweet. Okay.

Now I want to create another column called expression where we will classify the tweets as positive, negative, or neutral based on the polarity value. So I have used the np.where function, and the condition we have here is: if polarity is greater than 0, it's a positive tweet; else if it is less than 0, it's a negative tweet; and if the polarity is equal to 0, it means it's a neutral tweet. Let me run this. Okay. So here you can see we have the column called expression, and it has classified the polarity based on negative, neutral, as well as positive. Similarly, let's do the same for the Biden data frame as well.

Okay. Before moving ahead, if you want to get access to the source code and the data sets we have used in this demo, then please put your email address in the chat section. We'll send it to you via mail. Also, please subscribe to Simply Launch channel and hit the bell icon to never miss an update from Simply Loan.

In the next cell of code, we have another function that will basically help us creating a bar plot based on the expression column from both the data frames. So this is my user-defined function name, and it has two parameters: reviews and title. You can see we have grouped the expression column and found out the count using the count function. Then we have created or converted it into a list. Similarly, below you can see here we have defined the color for the bars using RGB values. Now let me run it. Okay. So we have successfully created our user-defined function. Now let me go ahead and call the function and pass in my two parameters, reviews and title. So here we have the function name, and under reviews we have the data frame name trump, and the title we have given is analysis for trump. Let me run this. Okay. So you can see we have a horizontal bar plot on the top. You can see the title of the plot, which is analysis for trump, and as per the tweets it has classified as positive, neutral, or negative. If I hover over, you can see there were 2831 neutral tweets, 1211 positive tweets, and similarly there were 958 negative tweets for Donald Trump.

Now let me call that function once again. So I'll write exp_graph, which is my function name, then I'll pass in my parameters. This time I'll take biden, and the title I'll put is analysis for biden. If I run it, okay, you can see the plot for Biden. Now, if I hover over, there were 1290 positive tweets for Biden and 747 negative tweets. Okay. Let's scroll down.

Now let me check the number of rows that have polarity equal to 0, the number of rows that have polarity less than 0, and greater than 0 for both the data frames using the .shape attribute. So first we have for trump. Now, == 0 means we are checking the total number of neutral tweets. You can see the result. Now let's check for Biden. We have the result below. Let me run it. Okay. Now we'll drop those columns that have a polarity equal to 0 from both the data frames and display the shape. You can see we have the condition mentioned, and we have also used the drop function. So all the neutral tweets will be deleted, and I'm printing the shape of the data frame. There you go. So out of the 5000 tweets, it has deleted all the neutral tweets or dropped all the neutral tweets, and this is the final result.

In the current cell, we have defined another function called balanced_data. Now this will help us get an equal number of rows and columns and delete a specified number of rows. Let me run it. All right. Now I'll call that function to delete 169 rows from the Trump's data frame and 37 rows from Biden's data frame. This will help us subset the data frames. You can see we have now 2000 tweets from each of the data frames.

Up next, we'll create some visualizations to analyze the textual data. So first I'm using the seaborn library and creating a distribution plot for Donald Trump. If I hit shift enter, you can see this is the distribution plot. This, there are bars at the back, and there's a trend line. Okay. Similarly, let me now create a box plot. So this is a box plot. Here you can see there are some outliers. Now let me add a few cells below so that I can show that plot for Biden. So I'll write sns. I'll use the function distplot, and under this plot I'll give biden_subset and I'll pass in my column that is polarity. Let me run it. So this is the distribution plot for Joe Biden, and now let me also create the box plot. So I'll use the seaborn library and the boxplot function. I'll pass in my data frame, which is biden_subset, and I'll give my column name that is polarity. I'll run it. So here you can see the box plot for Joe Biden.

Let me tell you how you can create a cross-table using the groupby function. So before coming to this cell of code, let me insert cell above. Okay. Now, using the groupby function, you can create a cross-table. So I'll use the trump_subset data frame, and then I'll give my groupby function, and inside the groupby function I'll give my expression column and I'll count the number of expressions. If I run this, you can see this is a cross-table. Similarly, you can do it for Joe Biden as well. I'll just replace the names. I'll write biden, and the rest is the same. You can see this is the cross-table for biden_subset.

Now moving ahead, let's find the polarity percentage for negative and positive tweets. To do this, I have defined a function called pol_percent, and we have two parameters declared: subset and total. Let me run this. Okay. Now here I am calling the above-defined function and printing the polarity percentage for Biden and Trump, and I have mentioned my two parameters, subset and the total number. So these are the values.

Moving ahead, now let's create a bar graph that will show the negative and positive polarity percentage for Donald Trump and Joe Biden. Let me hit shift enter. You can see we have a really good bar graph. Now, if I hover the mouse cursor, you can see Donald Trump had 27.825 percent of positive tweets and 22.17 negative. Similarly, Joe Biden had 31.6 positive tweets and 18.4 negative tweets.

Now we want to find the public sentiment based on Trump and Biden tweets. Let me run these two cells first. Okay. Now for Donald Trump, the total positive polarity is a combination of Trump's total positive percentage and Biden's negative percentage. Similarly, for Joe Biden, the total positive polarity is a combination of Biden's total positive percentage and Trump's negative percentage. Let me run this. You can see the result here.

Next, let's plot a horizontal bar graph to visualize the public sentiment. We have our x and y labels defined. We also have set our bar colors with the help of RGB values, and we have the title as well. Now if I go up, you can see this is the public sentiment. So 26.88 is the percentage that is in favor of Joe Biden, and for Donald Trump you have 23.11.

Now coming down. Now in the next cell of code, we have a user-defined function that will basically plot a table which has the most positive tweets for Donald Trump and Joe Biden. The condition we are checking is that the polarity should be equal to 1. So if it's close to 1 or equal to 1, which means the tweet has positive polarity. Let me run it. Okay. Now let me go ahead and call the function. Now let me scroll over to the top. You can see these are the top five positive tweets for Donald Trump, and here you can see the top five positive tweets for Joe Biden.

Likewise, let's check the polarity for most negative tweets. Here the condition will be the polarity should be equal to -1, which means it's highly negative. Let me run it. Okay. Now let's go ahead and call the function negative tweets. So I'll write neg_tweets, and then inside the bracket I will give my parameters, which is trump_subset, and then I'll give my title as negative tweets for Donald Trump. Okay. Now let's copy this. We'll also check for Joe Biden. So I'll just replace this with biden, and let's say negative tweet for Joe Biden. Let's run it. Okay. If I go on top, here you can see the top negative tweets for Donald Trump, and similarly we have the top five negative tweets for Joe Biden.

All right. Finally, let's see how to create a word cloud. Now, a word cloud is a method to show which words are most frequently occurring among the given text. I have defined a function called wordcloud. You can see here, and it has two parameters: data and title. Inside the built-in function or the wordcloud function, we have mentioned the max font size, the maximum words, scale value; there's also background color, which is white. Okay, let's run it. Now this will create a word cloud. So let me go ahead and call the wordcloud function and pass in the arguments. So I'll write wordcloud, which is my user-defined function, and I'll give my parameters trump_subset, comma, I'll give my title as word cloud for Donald Trump. Let's see Donald Trump tweet replies. Okay. Similarly, let's do it for Joe Biden. Write wordcloud, biden_subset, and then I'll give the title as word cloud for Joe Biden tweet replies. Okay. Let me run it. There you go. You can see we have a nice word cloud. So on the top you can see the title, which says word cloud for Donald Trump tweet replies. So here you can see some of the words like real, Donald Trump, then there's precedent, love, right, all these. Similarly, if I scroll down, you can see the word cloud for Joe Biden tweet replies. You can see there's the handle Joe Biden, falls; there are some other words like due, taxes, belief, even, will, it's, fact, and you can see Kamala had his name here.

Okay. Now that brings us to the end of this demo on Twitter sentiment analysis using text data. You learned how to classify tweets and find the polarity. We also looked at how to create interesting visualizations with this data and understood the mood of the public. Finally, you saw how to create a word cloud that had the most frequently occurring words in the tweets.

Who is a data science engineer? Are you a data science engineer, or are you going to be looking for a different field? What exactly is a data science engineer? Well, a data science engineer is someone who has programming experience in Python and R, expert-level knowledge, ability to write proficient codes, and they—we have Python and R. I'm going to say Python or R. Once you become really proficient at one language, transferring those skills into another one is usually fairly easy. Now, R is a little different because it is an analytics platform. So going from Python to R, if you already know the data analytics in Python, moving to R is pretty easy. Now, R doesn't have all the other code that you can access with Python. There's so many things you can do with Python that are not data analytics. So that's important to keep in mind that you should probably be pretty well-rounded in Python and really have a solid foundation in R, or have a solid foundation in R and be well-rounded in another programming language where you really get the programming side of it strong. SQL and big data experience. So you have to have a strong coding skill with hands-on big data experience, and of course we're showing SQL here, most commonly used, whether you're using a Microsoft SQL Server or MySQL Server. You can also start thinking Hadoop and Spark in big data access. Hadoop file system is not a huge jump if you've already learned your SQL and you've already learned your basics and coding. Hadoop sits on top of all that and does a wonderful job creating huge clusters of data, but you really need to know your SQL because that is so common and most of the large companies now accessing their data, and an ability to visualize models and troubleshoot code of the models. This is a kind of an interesting one because these models are—can get very complicated. So you got to be able to break them down into something that you can put the pieces together and digest each of the pieces. So being able to visualize these models is very important, and then being able to drill down and troubleshoot the different models or the pieces in those models. They need to be a versatile problem solver equipped with strong analytical and quantitative skills, a self-starter with a strong sense of personal responsibility and technical orientation. This is an interesting one because when you talk about data science engineer, it's such a new field that most companies don't have it well-defined. They don't know what they're looking for. You might not even know what you're looking for. You might have an idea, and you're looking for patterns, but where do those patterns lead you? So you really need that self-starter side to jump in there and figure out where to go and be able to communicate that back to the team. And there we go. We have a strong product intuition, data analysis skills, and business presentation skills. And this was talking about is you got to bring that back to the team. So if you don't have a strong product intuition, if you don't know what's going on with the company, what they need, and where you're going with your analysis, it's going to be a dead end, and you've got to be able to present that to the shareholders, to your co-workers, which leads us to great teammate with excellent interpersonal skills. And this is kind of a strange one because you spend so much time behind your desk. So you have to be able to kind of float between you're studying all the data and being able to explain this to people in simple terms that they can understand. No one wants somebody to come up and say, yeah, the p-score and the f-score is this, that, and the other thing, and you might have two people in the room understand you. You have to be able to sit down and explain what that means and why.

So let's take a look at the data science skill set. We talk about data science engineer skill set; we're talking about database knowledge, statistics, programming tools, data wrangling—probably the least favorite and most used—machine learning, data visualization, and then the touch on big data. And so we'll start with database knowledge. The most common database is, of course, your SQL, structured query language. Uh, think rows and columns. It's an essential language for extracting a large amount of data from data sets. So knowledge of the SQL is mandatory for data science engineers, and you can see there's tools required; there's the Oracle database. I mentioned MySQL Server, Microsoft SQL Server, Teradata. There are so many different forms of SQL, and it'll just keep coming back and coming back. So if you don't have a solid basic understanding of SQL, go get it. Very important because it will come up. You know, it's—if you don't know it, it's going to bite you. And statistics, of course, we are—you know, that's what this is all about is figuring out and predicting things. So you need to know your statistics. Statistics is a subset of mathematics that deals with collecting, analyzing, and interpreting data. Therefore, data scientist needs to know statistics. So you need to understand your probabilities and what that means and what the p-score means, in the f-score, and means and mode and median, all that information, standard deviation, all of those you need to be aware of. And then we get into the programming tools, and I mentioned this earlier a little bit, but you need to master any one of these specific programming languages. Programming tools such as R, Python, SAS are essential to perform analytics in data. And again, you know, you can move in there, and you can be an expert in just R, you can be an expert in just SAS, and a little bit—you can be an expert in Python because most of these—when we're talking about data science and data wrangling—you really kind of get down to the point where you really need to have a full programming language, at least the basics. And if you're going to be working in data science, Python right now is the main one, but you know there's also Java, C++, so you need to know a solid language, at least the basics of it, so you can understand how to do basic iterations and things like that. I always found that interesting that my sister who runs a university data analytics center, that's the first question she asks for people, candidates to come and work for, is how do you iterate through data. You'd be amazed at how many people don't understand that very basic concept in programming.

So when we look at R, R is a free software environment for statistical computing and graphs. Supports most machine learning algorithms for data analytics like regression, association, clustering, and etc. Python. Python is an open-source general-purpose programming language, and there's our general purpose. I was just talking about you need to know a little bit about Python libraries like NumPy and SciPy are used in data science. SciPy, yeah, NumPy and SciPy are very central to Python, and I'd throw pandas in there also, that module, those are very central to working with Python and data analytics. And then SAS. SAS can mine, alter, manage, and retrieve data from a variety of sources. It can perform statistical analysis on the data. So where R is the open-source platform, SAS is more like the paid-for platform, and so it has some things more automated because people are paying for it, developing it, but it's also—if you really start using it as a company, you're gonna have to pay for it, where R is free, store is a free software, Python is also open source and free to use.

So we're going to talk a little bit about data wrangling, and I mentioned earlier that is both the probably people's least favorite aspects of data science and also probably one of the places you spend the most time. So we talk about data wrangling, it is a process of transforming raw data into an appropriate format to make it useful for analytics, and it involves cleaning raw data, structuring raw data, and enriching raw data. And this gets interesting because you'll get stuck on something like, in Python, you might get stuck on something that is a date time based on an integer 64 NumPy set.

You might not know what all those terms mean because you work in R or something like that, but you get the point that it's all one particular format. And then you have to figure out how to switch it so that the computer can see it correctly in whatever format you're using or whatever analytics platform you're using.

Knowledge of machine learning techniques, such as supervised machine learning, decision trees, linear regression, k-NN, etc., is useful for few job roles. So this is kind of interesting because sometimes that's the center of the job role of a data scientist, and sometimes you just have to be able to apply the programming skills to it so you can get the answer and let somebody else fine-tune your machine learning techniques. So it really depends on what it is you're working with.

You can see here we have our nearest neighbor set up where it groups things together that look alike. Linear regression: you're drawing a line through the data or curve so you can predict what the next value is going to be. And then our nice decision tree would start splitting things up and says, "You know, it's sunny out? Yes, let's go swimming. No, it's raining; we're not going to go swimming. Yes, it's sunny out; we might go swim. It's raining; we'll stay indoors. Yes, we'll walk the dog. No." So you have a decision tree that helps figure out how you get to the end result. Decision trees can be pretty powerful.

Data visualization. Data visualization is the study and creation of visual representation of data. Data visualization uses algorithms, statistical graphs, plots, information graphics, and other tools to communicate information clearly and effectively. This one you really have to master as a data scientist. This one you really have to master as a data scientist. And there certainly are so many different tools to use to visualize it. But when you're communicating, a picture is worth a thousand words. When you can put a picture up, and and people can look at it and go, "Oh, I see what's going on," that's worth a lot more than saying, "Yeah, this is good." People hear that and they just kind of go, "Okay, it's good, but why?" And you might say, "You know, okay, these numbers make it look good." No, people want something they can see. And there certainly is—Tableau is probably one of the most popular ones out there—Caleb View, Power BI, Google Data Studio. In Python, there's um, the PiKit and Seaborn. There's all kinds of different options for visualization. You need to master probably at least one or two of them so you understand how they work and how they're different.

Big data. Big data is a massive amount of data which cannot be stored and processed using traditional methods. Big data has various benefits, like access to social data can enable organizations to tune their business strategies. Big data can improve customer experience. Uh, and so we're usually, when you say big data, we're almost always talking about Hadoop and Apache Spark. It used to be Hadoop is your file system, so that's how you store all your data going across the nodes. And there certainly are other ways to store it. When we talk about Hadoop, we're usually talking about at least 10 terabytes of data when you're dealing with a Hadoop file structure. And then you can use Spark where Hadoop processes on and off the hard drive, so it's continually reading and writing to the hard drive. Spark is running in your RAM, and so when you're doing a very intensive data process, you'll run a Spark setup which will spread those processes over multiple computers. And of course, now the two uh, combine, and Spark sits on top of Hadoop, and you get your big data. There's also Talend. Tableau has its own server set up. There's Splunk, Cassandra—you'll hear the Cassandra database—Pentaho—I'm not even sure how to pronounce that one; it's not one I've worked with. Certainly there's a lot of options for big data. The two big players are, of course, Hadoop and Spark. And even your SQL servers and MySQL servers will spread across five different servers, and you can be talking about big data across five servers with those pulling them into say Apache Spark to do your high-end processing.

Then there's non-technical skills, probably the most important one in data science and any of our data analytics: intellectual curiosity; updating knowledge by reading contents and relevant books on trends in data science. This is such—there's so much going on in this field, and it's exploding now that it's hard to keep track of it all. Uh, so you really need to have that curiosity in the data science, probably even more so than a lot of other fields. But I would say in just today's world, you need intellectual curiosity because things are changing so rapidly in today's world, the way it's going with our technology. Understanding how the problem solved can impact the business. Business acumen. This is the one where a lot of people kind of skip over: you have to have a little business sense. If you don't know how it's going to impact the business, you're going to be in trouble. Why do they want to even pay you to be there unless they know what's going on and they're getting something in return? So this is your bottom line for your paycheck: what's going on with the business? How is this going to impact the business? What are you doing for them? And communication skills. Companies look for someone who can clearly and fluently translate technical findings to a non-technical team. And I've talked about this a bunch, and I I can't even iterate this enough: if you are struggling with your communication skills, work on building them. They don't have to be—you know, this isn't rocket science where you have to know all these complicated terms. What you need to do is take the complicated terms you know from data science and reduce that down to something that anybody can understand. So when you come up and you have, you know, k-NN, no one knows what k-NN is, you know, unless they're in data science. Nearest neighbor, looking for things that match together. You got to explain that to people who do not have the technical skill in the data science arena. And a data science engineer needs to work with everyone in the organization, including customers. Uh, so it's teamwork, and that's always one that throws people for a loop. But ultimately, you are analyzing your customers' information, and so sometimes getting in there and rolling up your sleeves—if you're, let's say, you're doing data analysis for the sales team, and you're backing a sales team—you might have to get in there and go start making some sales and making some phone calls and see what it's like. What is that team doing when they call 15 people in cold calls, and then they do a follow-up two weeks later or three weeks later? Which one works better, two or three? Does it work better to call and send a fax, call and send an email? You know, when should you show up in person if it's that kind of business? So working with everybody, including the customer, is very important with the data and with this kind of setup and non-technical skills for a data scientist.

Let's take a look at some of the roles data science plays in the job market. Let's drill in there just a little bit here. So when you have a data scientist, they're going to perform predictive analysis and identify trend and patterns that can help in better decision-making. Companies hiring data scientists include Apple, Adobe, Google, Microsoft. I would say that's even expanding down to smaller companies, you know, where you're talking about only 100 employees or something like that; they're starting to look at data scientists because they need them to come in and be a part of that team. And the role is understanding challenges of a system and offer best solutions. This should always be the goal of a data scientist: is to be looking at the whole system and then trying to find those patterns that are going to best enhance the company or the business or the whatever group—maybe you're working with the sales team or the marketing team. And you can see languages—pretty important to have a—these are a number of different languages, and again it depends on your own specialty. There's all kinds of different variations, but we have R, SAS, Python, Matlab, SQL, Hive, Pig, Spark. You should have at least a knowledge of all of these. Maybe a Python programmer never used R; download it, go through the basics, so at least you know what's going on in R versus Python, because R is pretty powerful for doing a very quick display. And then, of course, a lot of things you can do in Python is to be able to do queries on database, from time to time create and modify algorithms which can be used to reduce information from large databases. And usually this is one of those things you do at the beginning; you set it up, and then you kind of let it go until it breaks and you have to go back and fix it because your query isn't working. That certainly is a very common thing to to have happen, especially if you're pulling data off the web and the internet.

And then we have some more companies that are hiring data analysis, like IBM, DHL, HP. And so we're talking about data analyst. The role is responsible for a variety of tasks such as visualization, optimization, and processing large amounts of data. And so when you talk about a data analyst, we're also—we're still talking a lot of the same stuff, but you'll see we've now thrown in JavaScript and HTML in here because that's pretty common with the data analyst versus a data scientist. And you can see that SQL is still pretty solid and also C++ is pretty big. So some companies also use Java, so I'd put Java in the languages also on that for a data analyst.

And then you have a data architect: ensuring that data engineers have best tools and systems to work with. And so companies hiring a data architecture, like Visa, Logitech, Coca-Cola. They create blueprints for data management with the best security measures. I cannot highlight uh, that a data architect really spends a lot of time with security measures; that's big. How many hacks have we seen in the last year on large companies have lost data for millions of their customers, and how badly that affects them? And we're looking at languages, as you see that SQL is pretty solid across all of these under a data architect. XML—that's one that a lot of the data science jobs don't really—you don't see as much—but you're going to see more XML with a data architect because there's a lot of XML files that are that have your security measures embedded in them. And of course your Hive, Pig—those are Hadoop systems—we're doing some basic queries. So instead of doing the high-end queries, you know, how these simplified query systems built into Hadoop and Spark. So if you're using big data, you're going to want to know your Spark also.

And then we have our data engineer. The data engineer updates the existing systems with better versions of the current technologies to improve the efficiency of the databases. Think admin here; a lot of work tracking down what version you're in of the different programs. What version of Python are you using? What version of R are you using? What version of C++? So companies hiring data engineers are like Amazon, Spotify, Facebook. And they develop, construct, test, and maintain architectures such as databases and long-scale processing systems. And here we have our languages: our SQL, our Matlab, SAS, SPSS, Python, Java, Ruby, C++, Perl, Hive, Pig. You pretty much, as a data engineer at this level, when you talk about admin and updating all these databases, need to know all the different stuff that's being used in the company, and you're testing out these structures to make sure they work. And so you really are talking admin-level uh, kind of setup. So if you like spinning up an admin onto a Google Cloud service or Amazon Cloud service, now you start to get an idea what we're talking about because then you can test it out up there, and then you can bring it back and update it to the company.

Statistician: creates new methodologies for engineers to apply. And so when we look at this, this is where we're digging into the math of which models to use, what setups going to work, what kind of activation do we have on our neural network? These little tweaks are going to make a big difference. And we look for methodologies at a large level: what kind of data are we going to be looking at? How we're going to pull it together? Companies hiring a data engineer or—we're talking about statisticians—we're talking about LinkedIn, PepsiCo, Johnson & Johnson. And of course these companies probably hire a little bit of everything, but they have a lot of statisticians working for them. And so we look at the role: extract and offer valuable reports from the data clusters through statistical theories and data organization. And the languages across the board, we're still seeing SQL, R, Matlab, SAS, SPSS—data—that's a new one that I've seen in some of the other roles—Python, Perl—an older version—and that shouldn't be a surprise because a lot of these companies probably built a lot of the original packages on Perl. If you go look at Johnson & Johnson, and of course your big data, Hive, Pig, and Spark.

And then our database administrator. I mentioned the other one is an admin; they're more an admin level for the software itself. Administrator is another level. Some of the tasks involved are monitoring, operating, and maintaining databases; installation, configuration, defining schemas, training users, etc. And this is interesting: training users, because when you build a database and you define all the different tables that are embedded in that database and how they're all connected, the end users got to have that information, otherwise you're in trouble. When we have companies hiring data engineers, of course, Tableau, Twitter, Reddit. Uh, the role ensures that all the databases are available to all relevant users. Again, a little security in there: what role do they have is allowed in and not, and is performing correctly and is being kept safe. And you better be ready to spend some time in security because again that's a very big thing nowadays with the number of people who hack into these databases. And the languages are going to be covered: you have your C#, your Java, SQL, Ruby on Rails, XML—again, there's that XML because a lot of your security is embedded in XML files—um, Python—all the main players, depending on what the company's using.

Data and analytics manager. So when you see manager, that means that somebody's going to sit on top and organize the people underneath, so they're going to improve the business process as an intermediary between business and IT. Oh, you better have some really good communication skills for this one. Some of the companies hiring a data engineers are Coursera, Motorola, Slack. And of course these companies are hiring data scientists in almost all of these fields. So the role oversees the data science operations and assigns the duties to the team according to skills and expertise. So that's a big one: you better be able to communicate what's going on, help your team members come out and be able to communicate with other people. So sometimes your job is just being able to get people to talk to each other. The big thing on that is is the whole communication line going from one end to the other because you can't communicate all the information that everybody's working on; you have to pretty much come in there and and help people communicate what there's going on and also know what they're up to without being micromanagement, because that will crash a company if you micromanage everybody. But if you just let them do whatever they want and they never talk to each other, that will also destroy the company. You got to find that nice middle ground. And you really have to have an overview of everything as a data and analytics manager. One of the things, I mean, you know, it's good to have at least one or two solid bases of programming under your belt for a data analytics manager, but you do need to have a very general understanding of all the different tools being used so that you know what's going on with your different team members. Of course, SQL, R, SAS, Java, Python, Matlab—all of those are big.

Business analytics. So this is when you're talking about your BI, or your business intelligence, and your business analytics. Very important to know; this is kind of a subdivision of a lot of the other stuff we've talked about, a lot of the other roles we've looked at. So you possess specialized knowledge of their business domain and apply that knowledge and analysis specifically to the operation of the business. Uh, so you might be a specialist in banking; you might be a genetics engineer, so you know about genetics; you might just be general business, helping the marketing. So a lot of companies hiring data engineer: Oracle, Uber, Dell. And the role acts as a link between the data engineers and the management executives. So it's very similar to—we go back up here—we look at data analytics manager. A business and analytics is going to also kind of fill that role in between, but they're more specialized in business. And you better know SQL because we're talking about business specifically; most of the large companies are storing their data that's up and running right now that people are accessing to purchase something on the Home Depot website or the Uber—they're logging in—you better know your SQL; it's very central to business analytics.

So let's go ahead and take a look at the data science engineer salary trends, and you'll see that our source is Glassdoor, as we're pulling a lot of our data from. Average base pay is around $117,000 in the US, the average salary. And we look at India; we're talking about 950,000 a year. And those are based on—some of them you need to go ahead and dig deeper to find out what education level versus entry level—but those are pretty solid base once you're in the industry and once you created your career in there. When we look at the job titles, the most common job title is data scientist. Business intelligence manager is the second greatest one; that's kind of good to note; that's why they put that up there as being its own little subcategory of data science managers, a business intelligence manager, because it's such a high level of jobs. And then we looked at some of the other ones: data architect, business intelligence architect—again, your BI, your business intelligence, gets its own ranking because it's such a high level—you know, it's a business; people want their businesses to make money—so that's who's hiring, of course: data engineer, business intelligence developer, business intelligence consultant, business intelligence analyst, business analyst, and data analyst. So you can look at the data analysis being the general one that doesn't go specific to business. So if you're looking just for your career as far as getting a job, probably should have some kind of business understanding in the background. And again, you know, your companies want to make money, so they're going to hire someone who knows business—if banking business, if you specialize in banking, or retail, if you specialize in retail and marketing. And you can see here on the data science salary trends and the growth and data science job listings, it's continually going up; it's um, since 2014 to 2012, we've gone from 400 to 600; that's a pretty big increase. So you know, a huge growth in this market; it's one of the biggest growing markets right now for jobs and careers.

So let's go ahead and take a look at building a resume—always exciting, putting ourselves out, selling ourselves. And if you looked at some of our other videos dealing with resumes, you'll see a trend here: the top part is so different than what resumes were in the 90s, in the 2000s to 2010. They've evolved; they've really evolved. It used to be 2000-2010, maybe LinkedIn, maybe one other reference. Now you're going to see that we want those references. This is a sales tactic which now has come into resumes. Used to be that if you're a real estate agent—every real estate agent I knew—I used to deal with real estate software for the real estate industry back in the 90s—every real estate agent wanted their picture on their business card; they wanted them—their picture—if they could put on their contracts, they want people to see the face. So that's really a big change is to make sure that it stands out. You stand out. Here's a picture of somebody, so you're more than just a couple letters and a name. But of course you need your contact information; should always be at the top. Uh, you have your summary: what are you focusing on? Be a little careful with your summary because if you have everything in your summary and then they scroll down to experience and education and skills, they're going to stop the second you repeat yourself in your resume. That usually means to the reader, "Hey, this person doesn't have anything more to offer me; I'm done." So be a little careful how you word…

Your summary: Most companies appreciate when you come in here, and you've adjusted this summary to be both about you and how you're going to serve that company. So it's worth researching that company to find out how those connect and put that in the summary. Take some time to do that; that's actually a pretty big deal.

The references are huge also, especially in data science. When you're talking about any of the programming or data science data analytics, having a place to go where they can look it up and scroll down and see different things you're doing—whether it's LinkedIn (in this case, which is the business profile most commonly used), GitHub (where you have stuff published), Facebook (I'm always hesitant because that tends to push more towards uh, social media type jobs and other jobs)—but certainly there's people who have Facebook who do marketing and stuff like that. But these links, having these basic links here, is important. People are starting to look for that for some other uh, setup. Maybe you have a personal website; this is a good place to put that so that they now have a multitude of links that go back to you and highlight who you are.

Then the next part, or next four parts—so for the next four parts, we have a combination of experience, education, skills, certifications. And you can see they're organized. If you have, you know, a lot of people like to see what kind of degree you have; they want to know where it came from. And if you just got out of college, you're going to put education at the top and then maybe you'll put skills after that, and then your experience at the bottom. If you've been in the field for years—you know, my degree, just to give you my age, goes back to the early 90s—so I usually put education at the very bottom, and then because a lot of the stuff I'm trying to sell myself on right now is my skills, I actually put that at the top, and I'll put my education, my certifications at the bottom, my skills, and then my experience (since it's a huge part of my resume) goes next. You can organize these in whatever order you want that's going to work best for you and sell you. So remember, you're selling yourself. This is you. You don't wear an old tie-dye t-shirt with holes in it; you know, something nice because it is professional. And of course, your summary, your—and then what do you have to offer the company? And again, when I put out resumes (and I haven't done a resume in a while), you go in there, and you can take this and reorganize this. So if the company's looking for something specific, you might put the experiences specific to that company. You might even take experience; if you have like a long job history like I do, I've gone into a lot of different things; you might leave out those companies or those experiences it had nothing to do with data science because it just becomes overwhelming. Resume should only take about 30 seconds to glance over, maybe a minute tops, because after that point you've lost the person's interest. And if they want to dig deeper, they now have links, they have your website, they have LinkedIn, and they can now take this, and they come back to it, and they go, "Okay, let's look at this person a little closer." So quick overview: This is your sell sheet, selling you to the company. So always tie it to the company so that you have that: "What am I going to give this company? What are they going to get from me?"

We've covered a lot on data science engineer in general, and we've gone over a basic resume. Remember, keep it simple, short, and direct; that is so important with that resume. Today's topic of discussion is how to become a data scientist in 2021. So here are the topics we'll be discussing today: A brief introduction to data science; laying the groundwork for a data science career; a short note on who is a data scientist. Later we'll discuss the in-demand technical skills for data scientists and then the salary of data scientists, after which we'll discuss the future of data science and data scientists.

So now let's have a brief introduction to the field of data science. There is no specific definition to describe data science. If you ask 500 people to describe what data science is, you'll probably end up getting 500 different responses because data science is a continually evolving field. Well, under Wikipedia, data science is an interdisciplinary discipline using computational methods, processes, algorithms, and systems to derive information and insights from both structural and unstructured data. In simple terms, data science is the domain of study that deals with vast volumes of data using modern tools and techniques. These tools and techniques are used to find unseen patterns, derive meaningful information, and make business decisions. The data used for analysis can be from multiple sources and present in various formats. There is a saying: "A jack of all traits and a master of none." When it comes to being a data scientist, you need to be a bit like this, but perhaps a better saying would be "a jack of all trades and a master of some."

Let's now discuss about laying the groundwork for a data science career. Let's now look at some of the prerequisites for getting a data science job: Have a relevant degree. A solid educational background is required to be a data scientist. Pursue an undergraduate degree in data science or any closely related fields like statistics, computer science, or mathematics. After the undergraduate course, a master's degree in one of the relevant areas is an added advantage. Creativity and critical thinking: Creativity in data science is very much needed in this field. You will need it for creating a storyline, for innovative modeling, for developing new tools, different ways to collect your data, and lots more. Parallely, critical thinking is very useful in data science. A data scientist should be able to develop questions and at the same time question the data as well. Require technical and non-technical skills. We'll be discussing in depth the required technical skills later on in this video. Talking about the other necessary skills in this field, you will need problem-solving skills, risk analysis, and many more. Good communication skills: Having a strong command over your communication skills is very much needed in this field. The outcomes that are delivered through data science are quite complicated and hard to explain. You will have to translate your technical findings clearly and fluently to the other teams. You will need good presentation skills, storytelling skills, writing skills, and listening skills. In short, make sure you focus on building lasting relationships through your communication.

Here are some of the roles in the field of data science: Data scientist, machine learning engineer, machine learning scientist, data engineer, data analyst, statistician, business intelligence developer, infrastructure architect, enterprise architect, applications architect. Data scientist is most likely one of the most sizzling job titles among the lot. And now let's talk about who is the data scientist. In simple terms, a data scientist is the one who practices the art of data science. A data scientist analyzes business data to extract meaningful insights. In other words, a data scientist solves business problems through various methods. DJ Patil and Jeff Hammerbacker coined the term data scientist. A data scientist's work typically revolves around data, and his role will be a combination of three main subjects: computer science, statistics, and mathematics.

It's time to discuss the in-demand technical skills for data scientists. To become a data scientist, you need various skills, as mentioned earlier. Some of the significant technical skills you will need to master include programming. There are several programming languages for data scientists; learning and mastering at least one of them is essential to realize various functions of data science, but at the same time, just a single language can't solve problems in all areas. Python is the most widely used programming language in the world today when it comes to data science. The next widely used language is R. There are still some data scientists who use R instead of Python, but if you're doing applied data science these days, then you've got to learn Python because that's the most valuable programming language for data scientists. Some other languages include JavaScript, Scala, SQL, etc. Talking about Python programming language, it is open source and easy-to-use language. You will need to understand the basic syntax of this language initially. You will need to know how to write functions, loops, and modules. You should be familiar with both object-oriented and functional programming in Python. It is a general-purpose and dynamic programming language that supports multiple paradigms. One of the factors that make Python stand out from the rest is its flexibility. You can use Python for almost all the steps involved in data science.

The next skill we'll be talking about is machine learning. The concept of machine learning has been around for a while now; however, the ability to automatically and quickly apply mathematical calculations to big data is now gaining a bit of momentum. You will need to learn techniques such as supervised machine learning, decision trees, logistic regression, and natural language processing. One of the crucial reasons why you need machine learning in data science is because of the high-value predictions that can guide better decisions and smart actions in real time without human intervention. There are, in fact, a lot of advantages of machine learning in data science; it helps analyze large chunks of data, and it changes the way data extraction and interpretation works. To sum it up, machine learning and data science can work hand in hand, with machine learning being such a craze; data scientists need to learn it. This is why Simply Learn has introduced a revolutionary machine learning certification program that provides advanced-level training on the applications and algorithms it uses. We have provided the link in the description box for the same. Do not forget to check that out.

Here's to yet another essential skill in data science: Math. Mathematics is the bedrock of any contemporary discipline in science; it's one of the major requirements for data science. Some of the major math topics you will need to know are calculus, linear algebra, probability, and statistics. Some other math topics that will help you solve data science problems are discrete math, graph theory, information theory, and many more. Up next we have analytical tools. They help you extract useful, meaningful insights from data and provide you with valuable frameworks for processing big data. Data analysis is a core practice of modern-day businesses, and choosing the right analytical tools is very challenging as no tool fits every need. Anyway, some of the most used analytical tools in data science are SQL, Spark, Hadoop, Hive, and many more.

Coming to data visualization, it is an interdisciplinary field that deals with the graphic representations of data. Data visualization plays two prominent roles: communicating results clearly to a general audience and organizing a view of data that suggests a new hypothesis of the next step in a project. Some of the data visualization tools are Matplotlib, Ggplot, and D3.js. Data visualization encompasses techniques that are used to communicate data or information by encoding it as visual objects that are contained in graphics. Well, as big data keeps growing, data visualization becomes an increasingly crucial tool.

Up next we have data wrangling. This is the process of gathering, selecting, and transforming data to answer an analytical question. It will help the data scientists fast in decision-making and hence get better insights in less time. Data wrangling is also known as data cleaning or munging. At last we have business acumen. When you are a data scientist, you must have more business sense in analyzing the data. In other words, you need to know whether the insights gained from the data is of value to the business or not; hence business acumen plays a very vital role in structuring business data analytics. When we are talking about business knowledge, three things come into the picture: general business knowledge, industry-specific knowledge, and company-specific knowledge. Knowing these are very much important when you are a data scientist.

So guys, this sums up the in-demand technical skills for data scientists. Well, let's now discuss the salary of data scientists. Before we go into the topic, we would like to introduce our new platform, Skill Up, where you can find some of our best courses for free. We are offering more than 1000 hours of content for free, which can help you acquire more than 300 skills across a number of different categories. Take the next step to grow in your career by joining Skill Up by checking out the link in the description. When we are talking about the salary of data scientists, many factors come into the picture. The salary would depend on the experience he or she holds, the industry, the company size, educational background, and finally the region. According to Glassdoor, the average salary of a data scientist in India is close to 10 lakh rupees, and that in the United States is close to one lakh thirteen thousand three hundred and nine dollars. In the IT field, the driving factor behind high data science salaries is that the companies and organizations are realizing the power of big data, and they want to use it to drive smart business decisions.

Coming to our last topic of today: Future of data science and data scientists. With every passing year, data science is improving and expanding at a significantly faster pace. Data science has found its applications in almost every industry. Healthcare companies are using data science to build sophisticated medical instruments to detect and cure diseases. Video and computer games are now being created with data science, and that has taken the gaming experience to the next level. Data science has taken various other fields to a whole new level. The top three most common search terms in July 2020 are data science, machine learning, and artificial intelligence. According to the 2020 Emerging Jobs Report, LinkedIn listed data scientists as the top third job with an annual growth rate of 37 percent for four years in a row. Data scientist has been named the number one job in the US by Glassdoor. Pursuing a career in data science is a smart move, not just because it pays well and is trendy; it is because data is the pivot point on which the entire economy turns. However, you can also go through our video on the topic: "Tips for becoming a data scientist" for more information. Thank you all for watching this full course video tutorial on data science with Python. I hope you liked it. If you have any questions, then please feel free to put them in the comment section. Thanks again. Stay tuned for more from Simply Learn.